QuestionPro Blog

[This is a guest post from Simon Chadwick, CEO of Peanut Labs, Managing Partner of Cambiar and Editor-in-Chief of Research World.]

This is a continuation of my last post, “Digital Fingerprinting and Sample Quality”…

What, then, can we do about sample quality? As usual where multivariate problems are concerned, the answer is ‘a lot of things’. There is no one magic silver bullet to solve the data quality problem, but a series of bullets that, combined, will serve to make things a whole lot better. To name just a few:

• We can come together as an industry to lay down guidelines and standards for online research and data gathering. An enormous amount of work has gone in to this, including the study by ORQC, the ACE collaborative effort between industry associations and the recently issued ISO standard. These, in addition to the ESOMAR 26 questions, lay the groundwork for real professionalism in the industry, together with learning that can be passed down through the ranks as to how to do good online research in which we can have confidence.
• As part of this, we can tighten up our panel recruitment practices. It is painful to see reputable firms subscribe to “cash for surveys” websites that register people for multiple panels, all in the service of getting “bodies”. The example below is just one of a multitude that exist out there.

• We can start to eat our own dog food. By that I mean that we can knuckle down and start designing surveys that are not only a whole lot shorter, but also more engaging to the respondent. We have known about Flash and other techniques for making surveys more interesting and involving for ages, yet how many of us truly use them on a regular basis? And, please, don’t use the excuse that we can’t do that on tracking or syndicated studies because we don’t want to risk breaking the data trends. Is it better to give clients reliable data or trendable data? And who has not heard of side-by-side trials?

All of that being said, these are longer term solutions. They require long-term adoption and education and will not solve the quality issue as it confronts us today. So what can we do that will move the needle right here and now?

Part of the answer lies in the very thing that gave rise to the problem itself. Data and sample quality issues online are the result of the technology that gave birth to this means of data collection. If we were not online, we would not have the problems with which online presents us. These include problems that are both quantifiable and a little more qualitative or “fuzzy”.

Quantifiable issues that we know about and can deal with include:

• Duplicates: a “duplicate” is someone who tries, knowingly or unknowingly, to take the same survey more than once. At its most innocent, a duplicate is where someone may be a member of more than one panel and be presented with the same survey on both. At the other end of the spectrum are people – and, indeed, survey factories – who deliberately enter surveys multiple times in order to maximize the cash value of their participation. The fact that they can do so is a function of their ability to get around most of the safeguards that are normally put in place. For example, they can delete cookies, come in under different email addresses or change their IP address.
• Geo-IP violation: simply stated, this involves a person who takes a survey as if they were in one country, but in fact are in another. So, for example, if your survey looks for people in the US, a person from China could take the survey and claim to be in the US. At its most simple (and innocent), this could be a traveling business person. At its most deviant (and more usual) it could be a survey factory in China using either people or bots to take surveys on a fraudulent basis. Simple IP checking can eliminate this unless, of course, the factory is using proxy servers
• Speedsters: these are people who take a survey and zip through it in record time. Very often, they will straightline through complex grids (“satisficing”) or deploy patterns to try and avoid being caught. They will also usually provide garbage answers to open-ended questions.

Then there are the more qualitative or “fuzzy” quality problems:

• Hyperactives: people who take a very large number of surveys in a given period of time. A now infamous comScore report suggested that 32% of surveys were being taken by 0.25% of the online panel population. Is this a bad thing or not? Do we want to put limits on this type of behavior or not?
• Cross-Panel Accounts: does it matter if someone is on 6 panels? Some data say ‘yes’, others (including ORQC) say ‘no’. Do we want to flag such people to see if their data differ from others?
• Repeat Offenders: people who have been flagged in the past as having engaged repeatedly in ’suspect’ behavior. Do we want these people in our surveys? Should they be flagged for separate analysis?

These are issues that exist in the here and now. No matter how much education and standards-setting goes on, we have to deal with them today. This is where technology comes in and puts power in the hands of the researcher to make key decisions as to what constitutes quality and what does not.

Enter digital fingerprinting. Digital fingerprinting is not a new technology. It has been used by the financial services sector for some time and, indeed, has been present in research, on a proprietary basis, for a few years. But it was when Peanut Labs introduced its OptimusTM technology in 2008 as an industry-wide solution that it really started to gain attention. Indeed, since then, multiple companies have launched their own versions of the technology, mostly on a proprietary basis (i.e. you have to use our sample or our hosting to get the benefit), and have made DF a standard offering in the quality debate.

Digital fingerprinting is not rocket science (otherwise it could not have been copied so quickly!). All it does is to take the 100-150 data points that a computer puts out when it connects via a browser to the Internet and combine those via an algorithm to produce a unique identifier for that computer that can be referenced every time it seeks to take a survey. These data points include such items as

•    your browser and its version
•    your IP address
•    your computer’s configuration
•    the software you are using and their various versions
•    the plug-ins that you have and their versions
•    other downloads that you may have on your machine.

The combination of these data points produces a machine ID that is unique. So unique, in fact, that OptimusTM can detect a machine 98.8% of the time that it comes back and tries to take a survey. Even if you delete your cookies, change your browser and change your IP address, a DF technology such as OptimusTM will nail you.

Why is this important? For starters, it means that we can detect duplicates straight away. If a machine tries to come in and take a survey more than once, DF will know. If a machine comes in and tries to pretend – even via a proxy server – that it is from a geographic location that it is not, DF will know. But there is more than that. By allowing the researcher to set certain variables, the technology will know if a machine is trying to speed through a survey, straightline through answers or provide poor quality open responses. The researcher can set the lowest amount of time that is respectable to take the survey, ask DF to look for satisficing,
inspect opens for length and other variables and look out for repeat offenders.

More importantly, DF technologies such as OptimusTM that are applied across multiple data collection sources can enable a researcher to decide whether he or she wants to block machines that have been identified as engaging in suspect behavior, or merely tag them so that they can assess the quality of data that they have provided at the back end of the survey. Peanut Labs’ Optimus™ Research Database (ORD) now has the results of 21 million respondents from across the entire spectrum of the survey industry. If one of these has been identified in the past as having engaged in suspect behavior, then they can be eliminated up front from participating in a survey, thus saving time and cost at the back end and improving the de facto quality of the survey itself.

What does ORD tell us about these 21 million respondents? Well,1.5 million were duplicates, 500,000 were Geo-IP violators and 250,000 were speedsters. The overall ’suspect’ rate was – you guessed it – 15%.

Digital Fingerprinting is not the solution to online data and sample quality problems. It is a solution, available right now, that can combine with industry initiatives, guidelines, standards and training to produce a quality product. And that is what we, research companies, data collectors and clients all want.

[This is a guest post from Simon Chadwick, CEO of Peanut Labs,
Managing Partner of Cambiar and Editor-in-Chief of Research World.]

One might be forgiven for thinking that the issue of suspect data and sample quality in online research has really only arisen in the past two years. After all, in that space of time, we have seen associations launching initiatives (including the huge study conducted by ARF’s ORQC), task forces springing up, conferences devoted to the issue and the launch of commercial and collaborative solutions – all aimed at bringing about comprehensive resolution to the problem. But, in actuality, worries about data and sample quality have been around for a lot longer – Cambiar’s first study of the online industry highlighted this as the top concern for both clients and researchers, and that was back in 2005!

Despite this activity (or because of it?), that concern persists, unabated. The fourth Cambiar study, conducted in February and co-sponsored by Peanut Labs and MROps, demonstrated that sample and data quality remain stubbornly at the top of the list of concerns.

 
As a sidebar, it is interesting that full-service market research companies evince much more serious concern about these issues than do clients, despite the recent hype about this being a client-led revolution. Additionally, it is clear that what constitutes quality differs depending where you are on the food chain. For full-service companies, it is defined as “data” or “sample” quality. For data collectors, the issue is much more about survey and questionnaire design. Who is right? Both are.

A literature review of what is out there on online data quality yields a plethora of articles, webinars and presentations. ORQC alone has amassed more than 300 articles on the issue, while studies such as that conducted by Burke in 2007 suggest that some 14% of respondents from online panels are in some way ’suspect’. Indeed, our own data suggest the same level of problems – of over 21 million respondents run through our digital fingerprinting software in the last 12 months across a wide variety of data collection sources, 15% were identified as being ’suspect’.

So what does this mean in the real world? Inefficiency, extra cost and, potentially, wrong decisions based on faulty data. If our clients are paying for the data that we provide them, but a proportion of those data are suspect, the least that can be said is that they are overpaying, since research companies routinely have to oversample in order to compensate for ‘duds’ in the data set. The research companies themselves are paying more in terms of time and salary to check data at the back end and weed out the duds. And if, heaven forbid, a client makes a decision based on faulty data, then the costs can be astronomical.

So, is this a problem? Yes it is. We can run all the studies we want to try and prove that one or other component issue in data quality ‘really doesn’t make much difference’, but the truth is that the multivariate nature of factors that make for poor data quality means that we don’t really know what the impact is on our research and how much it is skewing our results.

[This is a guest post from Gary Angel, President of Semphonic, a web analytics company based in San Francisco]

Online survey technology has made available a whole range of analysis and measurement that was really not possible before. From inexpensive primary research to a deeper understanding of your web site audience to a different perspective on web behavioral data, online surveys can contribute mightily to our knowledge.

But online survey analysis doesn’t work quite the same way for each of these tasks. When you’re doing primary research or audience profiling using online surveys, your biggest concern is probably getting a good sample. Particularly in the early days of the web, most researchers simply discounted online surveys for primary research because the online population was too different. That isn’t really true for most companies nowadays – which is certainly one of the reasons why online survey usage has skyrocketed.

But assuming your sample isn’t skewed in some fundamental sense, the analysis of online survey data for primary research and audience profiling is essentially identical to the body of techniques developed for offline research.

That isn’t true, however, for the very wide and popular range of cases where you want to apply the results of survey research to a deeper understanding of the web site and the behaviors exhibited there. To see why this is so, consider the following example:

A media site launched an online survey of visitors. They tracked overall site satisfaction and also the usage of a number of different site areas. They had recently launched a new “comment” functionality on the site that allowed users to submit comments, rate comments, and track their own status as commenters. Tracking this tool in the online survey, they found that the users who generated comments had a significantly higher satisfaction score than the site average.

From this, they concluded that the comment functionality was boosting site satisfaction and was a success.

Sadly, however, this conclusion is simply not warranted. There is no way to determine from the basic facts:

Comment users have a higher sat score than non-comment users (attitudinal)

Or even

Comment users consume more pages than non-comment users (behavioral)

if either relationship is causal. We don’t know if commenting self-selects visitors who happen to be more satisfied and consume more content or whether it actually contributes to that relationship.

People who use comment functionality may already be more engaged and have higher satisfaction than those who do not bother. If so, the apparent (and statistically valid) relationship between using comment functionality and satisfaction is non-causal – at least in the direction we are hoping for.

Comments are not driving satisfaction, they are being driven by it.

It’s as simple as this. People who are highly-engaged with your site are likely to be more satisfied with it. They may also be more likely to view or post comments. This in no way proves that they are more satisfied because they view or post comments. They may be less satisfied as a result of commenting. They may be more satisfied. There may be zero impact. You just don’t know. Looking at the satisfaction scores for each area on your site and inferring causality from them is simply a basic statistical fallacy.

This is an incredibly common source of error when doing web analytics in general and it has migrated seamlessly over into the usage of online survey data. Self-selection is, in fact, a subtle sort of sampling problem where we forget that the sub-populations we are using for an analysis are not random.

I think it’s fair to say that a simple majority of all uses of online survey data as it applies to web site performance that I see are nothing more than interpretive errors caused by self-selection.

You can defend yourself against these types of errors, but it takes significantly more work. Internally, you can try to use other variables inside the survey to hold the populations constant across a range of other factors (like intent, brand awareness, overall usage) before you look at comparative satisfaction scores.

Naturally, the quality and size of the survey also affects its analytical strength. While “less is better” (more people will fill it out), a good survey will ask the same questions in different ways, in order to judge the quality of responses.  “Were you able to find what you’re looking for on this website?” can be paired with “Was the navigation or search on this website effective?”

Widely disparate answers to these two questions suggest that survey respondents are not really paying attention to what they’re answering, and can then be filtered out.  This, of course, is all standard surveying technique.

A different technique is to use behavioral data integration to analyze the population of relatively similar respondents (as discussed in a previous post) if you hold constant for number of visits, engagement milestones and total activity you can often get a good comparative population. Finally, you can use sampling techniques directed to tracking satisfaction of users before and after trying a tool or area (like commenting).

Each of these methods is designed to give you a valid population with which to compare the group who did the activity you’re interested in. Of course, each of these is more work than just doing a cross-tabulation between two survey variables. But what works okay for profiling your basic web audience is more than likely to be fundamentally deceptive when applied to a range of analysis that can involve self-selecting behaviors on the web.

As I mentioned above, behavioral analysis, analyzed using “engagement” as part of the analytical process, is just as prone to self-selection as online survey data. Combining the two is often the best way to build a much more comparative population set than either can achieve on their own.  Survey data, when combined with behavioral data, can enlighten marketing and editorial teams about not just what visitors are doing, but also what they’re thinking.

Indeed, finding ways to develop better control groups is one of the larger, if somewhat hidden, advantages of combining web behavioral and online survey data.

More Info:

• Semphonic, a web analytics consultancy: http://www.semphonic.com/
• QuestionPro Pop-up Survey Configuration:
  http://www.questionpro.com/help/64-window.html

[Gary Angel is President of Semphonic (http://www.semphonic.com), the leading independent web analytics consultancy in the United States. Headquartered in the San Francisco Bay Area, with offices in Washington, D.C. and Boston, Semphonic works with all of the major web analytics tools including Omniture, WebTrends, Unica, Google Analytics and Coremetrics. Semphonic clients include companies like the American Express, Barclays, the BBC, Charles Schwab, Genentech, Intuit, Kohler, the National Cancer Institute, National Geographic, Nokia, and Turner Broadcasting.]