Quantitative Market Research: The Use and Misuse of Market Research Statistics

“I want to see the entire marketing team in Conference Room A at 10 AM this morning. No exceptions.”

As voice mail from the Executive Vice President Scott McDermott cascaded through the key teams, each VPs’ forwarding message raised the stakes and anxiety for each recipient in the food chain.

There were no stragglers to this meeting, their first with the Big Name Business Leader Media Star, freshly recruited from outside the industry. Across the assembled teams, managers quietly speculated and shared rumors on this latest fire. As the EVP commanded the room, his body language broadcast worry and anger:

“I’ve just received confidential intelligence on our industry’s Direct Mail performance for latest quarter. [Competitor A] did a mailing that got an exceptional response rate. Far higher than has ever been reported!!! On the other hand, this department is barely beating the industry average—you’re 3% compared to 1% to 2%. Too much time is being spent sitting around congratulating yourselves, when quarter after quarter you’ve only won each race by a nose.”

“Well, this time you’re not. [Competitor A] is crushing us. And I’m being asked why we’re doing so poorly in comparison. Hell, some of you came from over there. Why can’t you figure out how to get push the envelope???!!! I don’t want to hear excuses; I want results. Get moving; get busy. I want to see our numbers climbing and I want it fast. I’ve instructed each of my direct reports to make this top priority. Now get me some answers!”

And he left.

The room burst into multiple conversations.

The meeting’s tone transformed from Executive Suite to a gym, with each speaker hoping to demonstrate insight and commitment by trying to “out-macho” all precedingcomments.

Close to me sat a frustrated product manager and list development manager.

“Doesn’t matter,” answered his colleague. “Scott is obviously pissed about this and BS flows downhill. We’ve got to come up with something.”

“But what idiots! No one, simply no one ever gets such an outrageous response rate. It’s just BS! And they believe it and expect us to do better. Can’t be done.”

Turning to me, the Product Manager asked, “Well, Dr. Research, if it’s not a breakthrough, what is going on? It’s your damn report.”

“First, it’s not our report. Some other department is buying a syndicated report from a research company. So not only is the company paying twice for the same report, but someone is cherry picking the data. I can’t speculate on their motivations, good or bad, but I investigated when I first read the report – because that number was so totally off-the-wall.”

“So what gives?”

“The research company claims they are just reporting what they found, which, of course, is bogus. In this case, the published table combined results from combined multiple reports. Whoever sent this up the line might not have explained that, or they may have deleted all the footnoted caveats regarding unique sampling size issues, which make the competitor’s number totally unreliable. ”

“Because they didn’t want any technical issues detracting from the point they wanted to make with the EVP?”

“Or it could be that some junior staffer here may have stripped the number out of the report and had no idea what it means or what trouble it might cause, especially taken out of context.”

“Who would create a fire storm so they can sell their solution to the new fire chief?”

“Well, there already is a fire drill. Just look around. But to get anywhere, our conversation needs to move away from a Blame-Game blind alley and focus on the real issues.”

“Which are….?”

“Doing strategic market research that produces customer insights and competitive advantage and improving the information flow process. I’ve raised that issue before. Marketing must assign a single source to acquire, vet and disseminate all syndicated research reports.”

“Senior leaders don’t have time to spend learning technical research issues. They need to be focused on pertinent, high quality results for decision-making. And if they’re not, as today, chaos reigns. No one wins when competitive intelligence becomes the wild west.”

“Well it’s all total BS!,” the list manager whined. “What kind of a fool would believe a number like that and what kind of a charlatan operation would report that as valid?!”

“Okay,” the product manager said. “So obviously something else must be going on here. I doubt Scott believes it either. Then why the fire drill? Maybe the anger and frustration could have been handled better, but clearly Scott’s rear is getting chewed from higher up and something’s obviously got to change. And it appears that that is our problem. How do we respond in ways that satisfy the thirst for blood and also up our game. Maybe we can do better than we’re doing………………”

“Well,” I picked up, “we’ve been talking about doing a segmentation study of our potential customers so that we can align the offers more tightly with actual perceived needs.”

“We’ve been over that ground,” the product manager interjected. “It’s much cheaper to send out the mail to do the tests than to take six months to do segmentation.”

“Yes,” I agreed, “Segmentation research is long and laborious…..but in a highly competitive environment, it can give us a real edge. Consider this: if we identify the main drivers that cause response—what offers appeal most to which potential customers, we can up the response rates by better targeting. And we’ll have a much better understanding of how to position the offers.”

“The products were so hot for a while and we were so far ahead of our competitors that we didn’t really need to understand them more than we did. But now, competition is closing the gap; maybe we need to think in that direction.”