How to Fix the 5 Most Common Mistakes with Focus Groups

Addressing fundamental problems with focus groups.

Article No :671 | May 13, 2011 | by Jay Eskenazi

The purpose of this article is to raise consciousness about fundamental problems with traditional focus groups used for product development. Even experienced researchers often ignore, dismiss, or are sometimes even unaware of these issues and how they may be impacting the quality of their research.

1. There is a gap between what focus group participants say and what they do.

Traditional focus groups do not provide an accurate read on customer behavior.

If you are relying on focus groups to get a direct read on customer behavior or purchase intent, you are most likely using the wrong method. Moderators who tell you they can provide you with “insights” on these topics may be well intentioned, but are likely overconfident of their abilities or lacking serious understanding of the wealth of research that's been done in this area.

As Harvard Business School professor Gerald Zaltman wrote in his influential book How Customers Think, “The correlation between stated intent and actual behavior is usually low and negative.” Eighty percent of new products or services fail within six months of their launch, often after they have been vetted through focus groups. In Hollywood, for example, TV pilots and motion pictures are often screened via focus groups for audience reactions. Countless products “pass” in focus groups, but fail when they reach the market, and the gap between what participants say vs. what they do is likely the reason.

Another problem with focus groups is they often ask people for their quick, topical reaction to products they have not seen, used, or experienced. The problem is that everyone has an opinion. And the more detail they give you about their opinion, the more likely you (the focus group observer) are to think you are hearing the customer’s voice.

In general, the more abstracted an expressed opinion is from reality (i.e., the context of actual usage of the actual service or product), the more likely it is that the research participant’s opinion will be incorrect, inaccurate, and misleading.

What does this mean exactly?

If you show people a product, they will give you an opinion or reaction. If you have them use or interact with the product, their feedback will be based on their actual experience, which is far superior to their initial, hypothetical thoughts.

For example, if you ask a participant what they think of Android devices, they will offer an opinion. If they do not have an Android device, or have never used one, their opinion will be based on their attitudes toward Google or Android, what they have read, and what their friends have told them. But it will not be based on their actual experience.

If you have them use the device instead and then ask for their feedback, now you’re dealing with their actual experience. And feedback based on actual experience occurs at a much deeper level of cognitive involvement, meaning the participants will give you better, more accurate feedback.

This is a critical distinction. The deeper the level of involvement with the product, the better the insights customers will be able to offer about their usage of the product.

Traditional focus groups miss the mark when they focus on hypothetical choices rather than actual experiences. As Zaltman says, “Contrary to conventional wisdom, [focus groups] are not effective when developing and evaluating new product ideas, testing ads, or evaluating brand images.”

The solution is to create a research plan that focuses on actual user experience rather than hypothetical questions or scenarios. This will look like a naturalistic usability test, or a hybrid usability test that might involve having users use the product individually before returning to the group. In that manner, they will have the individual experience of using the product and can have a group discussion about the experience.

2. Small local samples cannot be generalized.

Focus groups tend to work with small numbers of people. The most common arrangement is to conduct 2-3 groups with 6-8 participants per group in several cities over the course of a week or so.

There are three factors here that affect your research:

Small numbers – a small sample size is often useful when you want to probe people’s attitudes in greater depth and try to get into their minds. Qualitative research can help flesh out themes that can be used for follow-up in quantitative methods such as surveys.

Sample – the sample used in this type of research is not representative of the larger population. Will the findings be generalizable from what a few people expressed as their opinions in Los Angeles? Are you willing to gamble with the direction of your product and/or company on the basis of what a few people said during a focus group? Obviously there is great danger here.

Generalizability – there is a problem generalizing small sample sizes because they are not representative of the underlying population. This is a basic concept in inferential statistics and probability.

Do not use focus groups or research with small samples to assess purchase intent. This type of finding is not generalizable from a small sample.

If your research questions pertain to user experience, have the participants use the product and then probe the experience. A common mistake is to provide a guided walkthrough of the product. As mentioned above, you will obtain far deeper insights from actual product usage vs. hypothetical experience. People are able to give you better input when they are not relying on their memory and they are also more cognitively involved in the experience so you will obtain better insights.

3. The group leader affects what is, or is not, said during the group, and the group leader is not the moderator!

Have you ever heard the term alter leader? I’ve met very few people who are aware of the concept of the alter leader and how it impacts observed behavior and comments in focus groups.

Sharon Livingstone, a well-known market researcher, has written extensively about this type of group member. The concept is that seating position in a traditional group, and the group dynamics surrounding that, affect what occurs in the group. This notion is rooted in group psychotherapy and I have observed these phenomena in my previous work as a clinical psychologist, before I started working in the UX field.

The alter leader can impact focus groups in the following ways:

Impact on positive and productive focus group - In a focus group, or traditional market research, the natural leader or alter leader helps the moderator. He often physically sits in the chair opposite to the moderator. He is often the first to speak and may be enthusiastic about the topic, thus influencing what others say (or what they don't).

Impact on a nonproductive or negatively focused group - The alter leader is vocal, but he is negative or resistant, and fights for control with the moderator. For example, he may object to the questions the moderator is trying to ask, or may answer questions that were not asked. In essence, this person's behavior is driven by his need to be argumentative and competitive, and he is in competition with the moderator to control the group.

The alter leader may be the first one to speak negatively about the topic at hand and question the tone, focus, or direction of what is being evaluated. He might say, “Well that might be good for other people, but it's too basic for me” And viola, suddenly you have a group that wants to be seen as more experienced, advanced, or knowledgeable than they actually are about the topic at hand. Or the alter leader might comment about the privacy concerns of the topic, and suddenly he has touched a hot button that will cause the group members to voice concerns as part of the group that they probably would not bring up if they were being interviewed individually.

I was involved in focus groups many years ago for a prominent online authentication system. At the time, it was the largest system of its kind, used by tens of millions of people on a daily basis. During a round of focus groups in Los Angeles, participants were asked for their input on a certain topic. One dominant group member raised the issue of security. Suddenly, security emerged as a concern for the group and the tone was that people were “very concerned” about security. Contrast this finding with reality that tens of millions of users were actually using that exact system on a daily basis that the focus group was discussing hypothetically.

Were the group’s concerns reasonable or legitimate? Certainly. But these concerns appeared larger than life in that 90-minute session (this relates to the vividness effect discussed below). Undoubtedly some of the same people in the group who expressed these security concerns during the focus group went home and actually used that authentication system the same day without even giving it a moment’s thought!

The best solution to completely eliminate the effect of group dynamics is to conduct the groups online. Online research eliminates the positive and negative impact of the alter leader entirely. Depending on the specific research platform that is used, participants can be in a group but have a one-to-one dialog with the moderator that is uncontaminated by group dynamics. For example, participants can be required to type their response before they see others’ responses, so they are not influenced by others as they would be in a real-life focus group. By having participants offer their own opinions before they hear or see other people's opinions, you ensure that they are:

being more thoughtful in their responses by having a chance to think them through

not being influenced by others, especially extroverted and dominant group members

prevent group dynamics from biasing the results and conclusions of the group

Even the most skilled and experienced focus group moderator cannot achieve this outcome in all groups. It is simply impossible to control these variables in traditional, in-person focus groups.

4. Introverts lose their voice during focus groups.

If you've observed a focus group recently, try to reflect on members in the group you observed. Here are a few questions to stimulate your memory:

How many people were in the group?

Do a few of the group members stand out in your mind's eye?

Do you remember what all of the people said, or just some of them?

As you think about what you observed and what you concluded, were your conclusions skewed toward the verbalizations of the more extroverted and dominant group members?

Can you remember any of the more introverted group members?

Do you recall what the introverts said during the group?

If you're like most people I've tried this exercise with, you'll find it nearly impossible to recall much of what the introverts said! This phenomenon is another hidden dynamic that occurs in nearly every focus group.

Distinguishing introverts and extroverts

Researchers have found that introversion/extraversion is one of five fundamental personality variables that have been shown to be consistent across many studies, across many cultures and countries, and across an individual’s lifespan. In fact, if you’ve ever taken a personality test such as the Myers-Briggs or the NEO-PI, this is one of the core dimensions.

There’s a fairly easy way to determine whether someone is an introvert or an extrovert: just ask! But instead of asking someone if he’s an introvert or an extrovert, ask if he gets his energy from being around other people, or if he likes to be alone.

An extrovert will tell you he likes interacting with people and that it “charges his batteries,” while an introvert will say people can be overwhelming and draining at times and he likes to be alone to recharge his batteries. People who are more in the middle will offer responses that show more shades of gray and indicate that sometimes they like to socialize and sometimes they like to be alone.

In group settings, extroverts love the attention and energy of the group. They will be “charged up” by it and it will bring out that energy.

They will often be talking quickly and more loudly, and will take up a greater percentage of the “air time” in the group, which subtly or not-so-subtly affects the group agenda. If the focus group is 60 minutes, you can be assured that the extroverts will dominate most of the hour.

Why is this personality factor so important for in-person focus groups?

It matters because you’re not hearing the voice of the introverts, who are likely half of your customer base since the personality dimension tends to be normally distributed.

In an in-person focus group, you are unlikely to obtain insights into the perspectives of your introverted participants to the same degree as you do with the extroverts. They often have equally interesting and compelling things to say, but it takes them longer to articulate those thoughts. In reality, your customers are a mix of introverts and extroverts, but if you are doing traditional focus groups, the group interactions and findings and conclusions are biased by the effect the extroverts have on the group. Extroverts prevent the introverts from having a say, and that affects the group discussion and your research conclusions.

This personality dimension can become more polarized depending on who we are with

Here's a simple rule-of-thumb: our personalities become more polarized and extreme based on who is in our environment. Introverts tend to be quiet by nature. When they are surrounded by extroverts, they typically become even more introverted.

And since extroverts love the energy of others and being in groups, they tend to become more energized by the focus group. This has the effect of making the extroverts appear even more extroverted and loud. This, in turn, makes the introverts even more passive and introverted.

In summary, introverted group members tend to be heard less during in person focus groups while extroverts tend to be heard more. This personality variable impacts what occurs during the focus group in an uncontrollable way, and that means what you’re hearing is more biased toward what the introverted participants have to say.

And this personality dimension also ties in with the next item, the vividness effect.

5. The Vividness Effect

The vividness effect is a well-researched phenomenon in cognitive psychology research that explains how vivid events (those that are highly graphic or dramatic) affect an individual’s perception and recall of a situation, event, or memory.

Because of the way our brains process cognitive and emotional information, we are more likely to be impacted by, and remember, highly emotionally charged information. Vivid pictures, movies, or things we’ve observed tend to stick out in our memory more than the mundane ones do. And since vivid events are more easily remembered, they may receive a disproportionate share of our memory, focus, and attention, and this tends to make our recall and experience biased.

Specifically, this means that the vividness effect often impacts our own recall of situations and events, and this effects they way we assess current or future situations.

The vividness effect impacts focus group observations and conclusions

How does this apply to consumer research and focus groups? The vividness effect impacts both the focus group participants and the observers.

When research is observed in person, the vividness effect causes observers to automatically be drawn toward sensational, vivid, and extreme examples. If a particular person in the focus group is animated and dominates the discussion (common with dominant, extroverted group members), what he says is more likely to dominate the conversation and be remembered.

After the focus group, when having a team debrief about what was observed, the observers are more likely to remember the more extreme and vivid comments.

Imagine a focus group exploring people’s attitudes about an airline’s customer service.

One of the participants, Helen, was an extroverted woman who was the dominant group member. Helen talked at length about the problem she experienced with sitting on the tarmac for four hours during the flight delays during the summer of 2007. She told the whole group in a very colorful, dramatic manner about how the flight attendants would not even serve her food as she sat for the four hours! The plane sat on the tarmac in the Florida heat, and began to smell like a high school locker room. She was drinking coffee before the flight and she had to go to the bathroom so badly that she nearly peed her pants. And the flight attendants, according to Helen, were “Nazi-like” and spoke over the intercom that no one would be allowed to get up out of their seats.

Do you have a good picture of Helen and how she feels about the airline? What is your conclusion—did the airline do a good job with customer service?

Actually, the answer is that we do not know since Helen is a made up example to illustrate the vividness effect at work and the impact it can have both on the group and those observing the group. This example is so vivid that it will likely obscure the ability to hear what and recall what the others are saying. If the other seven group members all had positive experiences with the airline, you are still likely to remember Helen’s comments because they will stand out in your brain due to the vividness effect. And despite knowing this is a fictitious example, it will still stand out in your mind—that is how powerful the vividness effect is! Quite clearly, the vividness effect biases what occurs during the group and also what observers observe and recall after the group, and that impacts the validity of the research.

About the Author(s)

Jay Eskenazi (@jayeskenazi) is a seasoned user researcher and manager of user research teams, and has personally conducted or supervised 500+ usability studies in the U.S. and internationally since 1999. His research career spans 20 years, with the last 12+ years focused on user interface issues. Prior to co-founding Customer Experience Labs, a boutique usability firm, he worked as a user researcher on Microsoft’s e commerce properties and then created and served as Director of the User Experience Research team at Expedia for 7+ years.

Jay earned his B.A. in Psychology from University of California at Santa Cruz, an M.A. in Communication (Social Psychology) from University of California at Santa Barbara, and an M.A. in Psychology and a Ph.D. in Clinical Psychology from Alliant International University in San Diego.

Comments

D.

December 15, 2013

I can't thank you enough for putting together this list! Having participated in a few focus groups myself, I saw all of these observations embodied in several focus groups that I had participated in in the past. The moderator for one of the focus groups I was in was so bad, not only did he not allow ANY divergence from the required questions, he would consistently challenge people as if his own personal opinions on the product were being scrutinized. I just remember it being one giant argument with no control what so ever. I don't know how the company got any useful information out of it. If I was the market research company, I would have fired the moderator after the first 5 minutes. This experience sort of put a sour taste in my mouth when it came to participating in focus groups. This was until I participated in a focus group conducted by a company named SQM (http://goo.gl/oqCYth). They're moderator was top notch. She allowed opinions to flow freely while maintaining a form of order that definitely contributed to the group. Now I know what to look out for next time I participate in a focus group. Hopefully a few market research companies will read your tips and improve their own efforts as a result!

Taking this one step further, I think a lot of the issues the author is outlining pertain to stakeholder interviewing and contextual inquiry as well. For example: I get MUCH better results from a one-on-one stakeholder interview than a dyad, triad or group meeting. The mask comes off more easily. Unless group dynamics are a data point that you're looking for, you have to be very careful about interviewing or observing groups in a business setting. The differences in power, authority or position among the members can really get in the way.

Many of the drawbacks described in this article can be overcome with skillful moderation & careful recruiting. There are many ways of dealing effectively with dominating respondents without shutting the group down. Seasoned moderators know how to work with group dynamics so everyone contributes fully - even introverts.

If you are going to explore attitudes towards a particular product, issue, or category then you need to find people with the right background who can talk intelligently about your topic area.

Finally, as with any other research method, it's important for the researcher to know the strengths and limitations of a particular method, as well as when you need to use a combination of approaches. Focus groups are great when you want to hear the conversation of a group of strangers (who share key characteristics) around a particular topic. They are also great for brainstorming, exploring the factors that weigh into decisionmaking.

They are also good to do at the beginning of a research project to figure out the right questions to ask or to learn the language people use. Also, there is something about face-to-face interaction that is more powerful, interesting, and fun to watch.

I support Andy Bruch's comment; every methodology has biases and limitations. The key is to understand the possible ways your data can be unreliable or not valid. You can then add controls to the testing (for example a good moderator will have excellent techniques for controlling air-time and getting the introverts to talk and the extroverts to listen) and you can also understand the limitations of your data.

Focus groups are great for understanding issues that you are unaware of and can, properly run, be a great way of overcoming the shortfalls of one-on-one interviews and the practical problems with contextual enquiry and user tests. The insights in this article are great, but the conclusions are a bit myopic maybe? It almost reads as 'focus groups are tragically flawed' whereas the title suggests there were going to be some good tips for moderators.

I'm especially surprised by the statement that the "personality variable impacts what occurs during the focus group in an uncontrollable way". For an unskilled moderator maybe, but there are loads of techniques to control this - stand behind the extrovert, even rest a hand on their shoulder. I've known people who will literally put both hands on the extrovert's shoulders and press down quite hard! Extroverts/introverts are *completely* controllable. The vividness effect is controlled quite easily by listening back to your recording and taking notes from that. I've discovered insights I missed at the time on viewing back the group recording.

In my experience, focus groups are a great way of discovering ideas and attitudes, a great way to test out ideas that arise from individuals by passing the idea/attitude around the group, and a great way of discovering differences between different segments.

They are also the best way of thinking you have some good data but not realising that your execution of the method was completely flawed; this article is fantastic for showing what you need to deal with. But it can all be dealt with by a proper study design.

Though a generally good analysis on some of the intakes with focus groups, I must say I disagree with number 4. It is not necessary that introverts are quiet and therefore won't provide input. Really the main difference between introverts and extroverts when it comes to socialization is that introverts tend to be reserved IF they feel that the conversation is menial or unimportant. Introvert isn't a synonym for quiet or shy by any means, and many introverts are very socially outgoing. In the setting of a focus group, you can't assume that the introverts won't talk and the extroverts will do all the communication, as it mit be sn intellectually stimulating conversation for the introverts, anodyne you may not even know the diffence between the two. I think it's important to notice that introverts add a valuable input to the focus group process, and it's important to include them.

Excellent article, I very much enjoyed reading Mr Eskenazi's description of the potential pitfalls associated with focus group research. ALL research methods - focus groups, usability studies, online/unmoderated studies - have strengths and shortcomings; the key is to understand how to account for them in your research. I work daily with clients who wrestle with identifying the right approach to understanding customer experience scenarios, and having read this I feel more than a little bit better equipped to help them understand this method and account for its potential pitfalls.

Fabulous insight from Mr. Eskenazi. I couldn't agree more with his analysis. Definitely agree that online, bulletin board, and/or one-on-one studies produce more productive results. Furthermore, yes, let someone try out a new product and return to give his/her thoughts about if after using/experimenting with it. That's experience. Remember when you were in high school or college and you were asked if you've ever experienced sexual relations? Saying you’ve had sex and actually experiencing sex are not the same thing.
Finally, the vividness effect is a wonderful observation/point. However, personally, I have tough time with the "introvert vs. extrovert" topic. I have a friend that is definitely not social when it comes to his work. Stick him in front of a computer for 8 hours and he is happy as a clam, but don't ask him to sell you anything or do a presentation. Conversely, he is a social butterfly outside of work, hosting events and leading numerous get-togethers and outings. Therefore, I've always had an issue with Myers-Briggs and related tests. However, with that said, yes, the alter leader and/or extroverts can definitely overwhelm a focus group, thus potentially impacting data.