Today’s guest post is by Gary Angel, Gary co-founded Semphonic and is president and chief technology officer. You can read more articles by Gary at the Semphonic Blog.
We just finished our annual web analytics conference – the Web Analytics X Change. The conference is unusual in that it’s all small group discussions with enterprise practitioners. It’s a great format for hearing what’s really bugging people. As part of the conference, I attended a session focused on survey research and online behavioral integration. One of the most intensely debated topics was online survey sample size.
It’s a big issue because many organizations find themselves deploying almost as many different surveys as tags and they don’t want to suffer too much from “uncertainty principle syndrome” – damaging the user experience that they’re trying to measure.
How many respondents are really enough?
There are two schools of thought about sample size – one is that as long as a survey is representative, a relatively small sample size is adequate. Perhaps 300-500 respondents can work. The other point of view is that while maintaining a representative sample is essential, the more respondents you have the better.
This is a big issue because it impacts all sorts of decisions including the length of your survey, your collection mechanism, and, of course, you’re sampling rate.
So what’s the right answer?
If you aren’t integrating your survey data with web behavioral data, then a relatively small sample size might be okay (I’d emphasize the might). But if you want to combine behavioral analysis and survey data, then forget a sample 300 or 500 respondents. Those numbers simply won’t work.
Let me give you a real-world example showing why that’s true. We’re working right now with a client that samples approximately 1000 site visitors a month for their satisfaction survey. They asked us to do a study of the impact of using one of two internal search tools on their site on both overall site satisfaction and visit accomplishment.
On this site, search is used in about 10% of visits. Since the site gets more than 10 million visits a month, that still yields a heckuva lot of behavior to study – more than 1 million search visits every month. No problem there.
But our representative sample only captured about 100 respondents who’d used search.
Between the two search tools, one served about 70% of the queries. So for the second search tool, we had about 30 respondents to deal with. Getting the picture?
For our analysis, we wanted to track visit reason vs. satisfaction vs. outcomes for searchers. With some visit reasons only accounting for about 10% of visits, there were cases where we were supposed to analyze the outcomes for all of 3 visitors.
And that’s with a survey size of 1000 and a relatively simple cross-tabulation of visit intent and one fairly common behavior. Sure, we could add lots more months to the picture. But tracking behavior over extended periods of time adds all sorts of complications to the analysis. The combination of seasonality, site change and macro-economic change make this dangerous. Very few of our client sites remain constant for six months.
If doing behavioral analysis with 1000 survey respondents is challenging, imagine what it would be like with a sample size of 300. Impossible.
So what’s the right answer?
If I had my druthers, I’d recommend that high volume sites strive for a much higher sample size – something like 15K would be nice on a monthly basis.
Is that too much for your user experience to bear? Obviously, the answer depends on your site volume and your take-up and completion rates.
You can’t do much about volume, but if your survey length is impacting your take-ups or completion rates, then I’d be willing to sacrifice a whole bunch of questions to get to the increased size. The fact is that on many 30-40 question surveys, we’d only expect to use at most 5-10 of those questions in a behavioral analysis. I’d bet even money that your analysts feel that same way and that a heavy majority of questions on many long surveys hardly ever get studied at all.
At some point you may have to make a decision: do you want a whole lot of really shallow information or do you actually want to do analysis on a narrower set of data?
So when it comes to behavioral analysis combined with survey integration, the right answer is pretty obvious. A representative sample is essential, but size really does matter. Let yourself get talked into a 300 person sample, and you might as well throw all that work you did to integrate online survey data with behavioral data in the junk pile.
Very long URLs used to not really matter to people – people would surf around on the web and barely pay attention. Now, as more people share links via Facebook and twitter – the URL matters! Specifically, when you need to share a link on Twitter, a shortened URL is imperative. With both IdeaScale and QuestionPro, we’ve been working to find ways to get those URLs shortened down as much as possible.
Today’s guest post is by Kathryn Korostoff, founder of 
Today’s guest blogger is Chris Brown. Chris helps small and medium sized companies 
Rob Hoehn
Ivana Taylor
Vivek Bhaskaran
5circles
