So, I find my way to their boardroom with its long table surrounded by similarly pimpled kids. The huddle of marketers at one end eagerly scribble notes as we discuss game features.

I don’t remember what I said. I think I made a point or two that seemed to be what a hard-core gamer would say. Good enough to get my pizza and video game, anyway. But, probably useless for the video game producer.

Later in my marketing career, that tables were turned. I created focus groups to evaluate advertising messages for our ad agency. We traveled across the country to pull in small groups of target audiences and stood behind one-way windows saying things like “Aha” and “That’s interesting…”

I wondered how much we should act on that input from a couple dozen people. If one person makes a comment about our print ad, and no one in the room disagrees, does that mean they all feel the same way? And, would our entire target audience of hundreds of thousands agree?

Now I also wonder if they were as qualified to be there as I had been in my video-gaming days. Where they just there for the pizza?

The Hidden Risks of Qualitative Research

There are several reasons why using qualitative feedback alone can lead to misleading findings Tweet this:

Self-Selection Bias

Self-selection bias is a significant problem when users volunteer to be in a study. This is the error I introduced as a young focus group participant. You’ll also never be able study people who don’t want to participate in studies. Look out for real customer motivations for giving feedback.

Preset Goals Creating an Artificial Scenario

In usability testing, the researcher sets predefined goals for the user to attempt to accomplish and then monitors their success or failure and points of difficulty. Probably the most fundamental limitation of this type of user testing is that the scenarios are artificial. The task you choose for the user may not be the task a typical user would choose. It’s also not likely to be relevant to the particular user you’re testing. In other words, you’re asking a person to imagine and act as if they were the type of person who wanted to accomplish the task you want them to. That’s asking a lot!

Mis-matched motivations?

Limited Imagination

Qualitative feedback can tell you about possible miscommunications, interface problems and technical errors, but they usually don’t generate ideas about the layouts, content, and value proposition that would be more persuasive. Steve Jobs was once quoted in BusinessWeek saying, “It’s really hard to design products by focus groups. A lot of times, people don’t know what they want until you show it to them Tweet this:

All of these errors and biases limit that value of qualitative feedback.

Is all Qualitative Testing Useless?

Focus groups, qualitative surveys, and usability testing can be useful for gathering feedback from your visitors. They can lead to valuable hypotheses to test. Traditional usability testing, for example, is valuable in exploring a variety of scenarios quickly, gaining immediate interactive feedback, and developing hypotheses that could not otherwise have been predicted.

But when used in isolation, these aren’t good tools for website decision-making. This is not conversion optimization.

The potential insights generated need to be validated through controlled testing. Blindly following qualitative findings without verifying using controlled A/B/n tests can lead to dangerous mistakes.

Conversion Optimization Requires Quantitative Testing

I still come across people regularly who say they’re doing website testing. When I probe into their methods, often they have run user testing or have customer panels who have opted-in to be surveyed. I’ll repeat: this is not conversion optimization! Tweet this:

By all means, incorporate qualitative methods into your marketing system. Then, use the input from them to generate better hypotheses for split testing.

The scientific method of marketing starts with formulating questions to ask of your visitors. If you can use these qualitative studies to help you develop better hypotheses, that’s great! That’s the role they should play.

Some of the most dangerous traps marketers face are jumping to conclusions without data, acting too early with limited data and misinterpreting data. A reliable scientific testing system avoids those traps and helps you make proven marketing improvements.

Great article Chris! I have discussions about exactly this topic with online usability companies and our clients on almost a daily basis. I always tell them we will need to test any work that comes out of a usability study (which often only has a sample size of around 5 people…..).

A lot of companies here in New Zealand still only incorporate usability as part of a website build project allocating significant budgets to it and do not see CRO sitting in that same field (yet). CRO tends to come in after a site has been rebuild. How's that in the US? Most likely there is still a lot more education to be done here.

Thanks for the encouragement and your own ongoing commitment to educate the global market. I'll keep you posted on how things are developing here. We are the leading agency in NZ and soon to expand into Australia as well.

Great post!
Qualitative research can be done effectively, but the focus should be entirely different than with quantitative approach, and it might not be suitable for baisc website usability testing (although, as you said, you might get priceless insights).

Loved this article! I agree that qualitative and quantitative research do not stand alone, but each need the other to build the most accurate big picture and direction for the future. I'm keeping this article on hand to share with my team!

I totaly agree with this article and strongly recommend to read Clotaire's Rapaille Book : Culture Codes, dealing a lot with side effects described in the article.
The interesting point is more how he get through these biais not using quantitative solution ( which can be properly applied only on the web but hardly for brick & mortar business) but rather a different approach of focus groups.

[...] Traditionally, marketers have relied solely on their own thinking to come up with the right features to create and emphasize. Sure, they may get input from small groups of customers with surveys and focus groups. But, there are many reasons those focus groups aren’t as great as they seem. [...]

[...] Traditionally, marketers have relied solely on their own thinking to come up with the right features to create and emphasize. Sure, they may get input from small groups of customers with surveys and focus groups. But, there are many reasons those focus groups aren’t as great as they seem. [...]