Why So Much Market Research Sucks

My first brush with bogus market research came as a young product manager in a large metals company. Eager to transform ourselves from pure commodity competition to a value-added premium status, we invested in product quality and improved distribution. Over time, these small changes in products that were functionally interchangeable with those of our competitors garnered us "preferred" status with quite a few customers. When they had to choose from one of a half-dozen suppliers able to deliver a similar product, we began to earn more than our fair share of the business.

A key source of frustration to our CEO was that even though we were gaining market share, our prices remained the same as the competition's. Shouldn't we be getting a wee bit more for our product?

Market Research to the Rescue

The argument over whether or not to keep matching the competition in price continued until our ad agency came back with the results of a survey of our customers and potential customers. "Price" came in dead last, after factors like quality, performance, delivery speed, and customer service. The data surprised some of the old hands in the business who had been matching the competition for decades, but provided the confirmation the CEO needed to push through a price increase of a few percent.

The results of the price change were dramatic, though not in the way anyone hoped. Orders dried up. Unless we were the only source that had the product in stock, someone else got the business. We quickly retreated, and, over time, sales rebounded.

When Good Data Fails

In retrospect, it's easy to see why the survey data proved to be misleading, even though the data was collected in a sound way and the respondents weren't intentionally providing false information. "Price" was indeed a negligible factor in the typical decision because everyone charged the same price! The customers didn't ignore price - they had printed lists from each vendor, and would confirm the price when checking on availability. But, given that the prices were identical, the final decision came down to who had the product in stock, how quickly they could deliver it, how well the products from that vendor were performing in the factory, and so on. Price was a non-issue.

The Deeper Problem with Surveys

Proper survey design might have avoided the disaster I just described - a few questions about price differences between vendors, how often the customer paid a premium price for a product available from multiple sources, etc., would likely have raised some red flags.

But, there's a much bigger problem with a lot of market research conducted by survey, focus group, and other traditional methods: people are often incapable of articulating why they do things or how they would behave in the future. Our behavior as humans is influenced by many, many factors, most of which aren't conscious or rational.

"The Interpreter" in Your Brain

Even when do things for reasons we aren't aware of or can't articulate, we are quick to explain why we did them in rational, logical terms. Indeed, neuroscientist Michael Gazzaniga says we all have something called "the interpreter" that incorporates our actions into a logical and consistent framework.

The same split-brain research that exposed shocking differences between the two hemispheres also revealed that the human left hemisphere has the interpreter. The left brain interpreter's job is to interpret our behavior and responses, whether cognitive or emotional, to environmental challenges. The interpreter constantly establishes a running narrative of our actions, emotions, thoughts, and dreams. It is the glue that keeps our story unified and creates our sense of being a coherent, rational agent. It brings to our bag of individual instincts the illusion that we are something other than what we are. [Emphasis added. From a lecture by Michael S. Gazzaniga, Consciousness and the Brain.]

The split-brain research mentioned by Gazzaniga included showing a subject a different image in each eye, meaning the right and left halves of the brain were presented with different stimuli, with neither being aware of the other. The subject's right brain then correctly pointed at an object related to the image it saw. The subject's left brain, at a loss to explain the connection, fabricated a plausible but totally false explanation based on the different image it saw. (See How “The Interpreter” Screws Up Market Research for more on this topic.)

Even if your customers have brains with normal wiring (or at least with connected hemispheres!), their "interpreter" is alive and well explaining emotional decisions in rational terms. ("My new convertible? It was a great deal, and gets a lot better mileage than the minivan I looked at first. I looked up resale prices, and convertibles hold their value better. Plus, this is a new style, which means that the look won't change for years.")

In recent years, there have been numerous books published that show how our behavior is governed by non-conscious drivers. Check out Dan Ariely's Predictably Irrational, Daniel Kahneman's Thinking Fast and Slow, David Eagleman's Incognito, Thaler & Sunstein's Nudge, to name just a few. If you think all, or even most, of your actions are based on reasoned, rational decision-making, read one or two of these books to get over that inaccurate notion.

Get The Real Story

If you want to get the real story on the behavior of your customers, readers, etc., don't rely on self-reported data. While such data can be fine for simple facts, like, "Did you eat breakfast today?" it will rarely answer questions like, "Why do you prefer Grey Goose vodka?"

The solution is to measure your customers' actual behavior. Don't ask people if a 20% discount would make them buy a product right away - test it! Web-based businesses can perform such tests quickly and easily, leaving very little to chance.

Neuromarketing studies, which measure brain activity and other biological indicators, are another way to gauge true emotional reactions instead of relying on how people say they feel. EEG caps and biometric belts are the most common tools used, though other techniques, ranging from reading facial expressions to measuring tiny differences in reaction time, are also used. At the moment, the cost of such studies means that they are mostly used by larger companies with significant advertising and marketing budgets.

Is Market Research Dead?

RIP Market Research?

Since I spend much of my time writing and speaking about neuromarketing, I'm often asked if traditional market research is useless, or if it will be replaced by "neuro" studies of some type. The quick answer is, "of course not." For capturing factual information, and even teasing out feelings, traditional market research tools like questionnaires, focus groups, etc., can provide valuable insights at a modest cost. It's only when we get to the "why" and "would you" questions that we move onto shakier ground. The more a decision is rooted in emotion, the less useful survey data will be.

In short, keep asking your customers questions. But, when you prepare to do so, ask yourself, "Is this a question that will help me understand their behavior?" (The "importance of price" question above fails this test.) Then ask, "Can my customers really answer this question accurately, and will they do so honestly?"

Have you got a market research horror story? Or an example of a surprising insight gleaned from asking customers questions? Share your story in a comment!

I write about science-based business strategies. One thing customer experience and corporate culture have in common is that they both involve humans. My books Friction (McGraw Hill, April, 2019) and Brainfluence (Wiley) both translate important behavioral science into strat...