Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our User Agreement and Privacy Policy.

Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our Privacy Policy and User Agreement for details.

1.
For B2B Market Research Professionals November 20, 2009 The Online Panel Quality Wars: All Market Researchers Benefit — Especially B2B Tech by Brad Bortner with Ellen Daley and Chétina Muteba Executi v e S ummary At the same time that online quantitative research has taken off, dubiousness abounds about the representation of the online panel sample. Buyers ask: Are panels representative of the markets researchers use them for? What can we do about the moral hazard (greatly increased for B2B tech) that leads bad actors to take surveys inappropriately? A wide variety of approaches have been percolating in the market ranging from “it’s not really a problem” to various comprehensive solutions. Buyers are finally pushing back. Major buyers — such as Microsoft and Proctor & Gamble — are making very specific panel quality demands for their research vendors, and many of the major full-service research vendors and panel providers are taking note. The end result? Three years of equally competing solutions with no clear direction for buyers. This means market research professionals must be very specific about demanding high quality panel for all of your research needs, or risk having your findings undercut when your internal customers ask: “Is this analysis really representative of our market?” A Dodgy Sample is the unwelcome guest at the online market reseaRch party Online, panel-based research has quickly gone from the poor stepchild of market research to the dominant mode for quantitative research in the US and is now undergoing rapid growth in Europe.1 Why the rapid growth? It’s faster, cheaper, and delivers innovative new capabilities.2 The fly in the ointment of all of this growth is suspicion about how representative online panels really are. This is not just a matter of maintaining a representative panel; it is a matter of ensuring that respondents are who they say they are, answer accurately, and do not take a survey more than once — under multiple aliases. While the Internet greatly simplified the job of recruiting and managing panels, it also opened the door to potential problems based on the moral hazard, the so-called “professional survey taker” problem. What is going on? · Bad actors, in general, are incented to behave badly. Online surveys generally pay per survey. Those respondents who are financially motivated may attempt to take many surveys to make as much as possible. Some accomplish this goal by signing up for many different panels, perhaps under different identities. Others may disguise themselves to appear more attractive to a survey, or sign up for the same panel under multiple personas. The end result can be one person taking the same survey multiple times (especially when panels are mixed), creating invalid responses by people pretending to be what they are not. There are even cases of bad actors creating “survey bots” to harvest fees by responding to large numbers of surveys.3 The end result is corrupted insights. Headquarters Forrester Research, Inc., 400 Technology Square, Cambridge, MA 02139 USA Tel: +1 617.613.6000 • Fax: +1 617.613.5000 • www.forrester.com