Wednesday, October 8, 2008

83% of people agree

I’m going to do a 'before and after' on a sentence I pulled from a qualitative research report I was asked to read recently.

Here’s the before sentence: Of the 4 groups, 83% of people said they liked the design.

Now some of you reading will, at this point, know exactly what I’m talking about: You can go and play. For those who don’t please keep reading.

There are many things wrong with the above sentence being part of a qualitative research report. Here are 3 points to start:

1. It’s not robust

Percentages, in a qualitative context, are pretty much meaningless. While at first blush, a grand 83% looks pretty good, what does it really mean? It means that, assuming 4 groups of 8 participants, 26.6 research participants, screened to fit a particular profile, and willing to attend a particular research group, said they liked the design. That’s a very small, skewed sample: hardly robust and hardly worth reporting. But that’s only the beginning…

2. It’s not controlled

To get a good, clean read on any particular issue in a quantitative survey, the way the questions are ordered and the way they are asked is key. For all intents and purposes, and as much as possible, the survey should be administered in a controlled environment. Even rotating the order of questions is controlled. This ensures reliability (being able to replicate the findings) and therefore, some confidence in the results.

In contrast, to get a good read in a qualitative study (we don’t necessarily go for clean in qual), we need to dance around a bit. Cover the floor. A good qualitative facilitator will bounce around, jump ahead, reverse, turn corners, step to the side…you might even see a grand jeté.

The point here is that the context within which the question is asked, ie the discussion group, will vary wildly for each group. In effect, it will be confounded by all sorts of, well, confounding variables; not least, the discussion itself. The fact that 83% of people said they liked the design means absolutely zip without understanding the discussion that came before.

3. It's open to misinterpretation

The third and most worrying point is the potential for misinterpretation. The most obvious here is making the assumption that 83% of people, per se, liked the design. And then using this ‘finding’ to make Big, Important and Expensive decisions, like changing the design.

Here’s the after sentence: Positive feedback for the design was based on factors X, Y and Z.

This reminded me of a case study I once read where people were put in a pitch black room, with a pin point of light. Afterwards they were asked if the dot of light was moving or stationary. If they were immediately separated & questioned, they all said that the light was stationary. When they were allowed to discuss the question in a group environment (with one "mole" confidently declaring that he "knew" the light was moving), they all eventually agreed that the light was moving.It's hard to control results sometimes.

eeek - qual and %'s dont go, surely that's the first rule you learn?? kirsten...those research pieces on group conformity are really interesting (some more here http://www.age-of-the-sage.org/psychology/social/asch_conformity.html)....i feel that a good quallie can help to counteract those effects though by encouraging debate and differences of opinion so that group members dont feel so much pressure to conform in their opinions. this can be the case even with really sensitive research matter. what do you think?

It is great to observe somebody put enough craze towards a matter. I'm thankful that I stumbled upon this process. I am lucky I used the time to read on past the 1st paragraph. You have got a whole lot to say, very much to offer. I really hope guys understand this and look into your page.

Nice Article topics. This is really excellent and fantastic. I hope it’ll be helpful for our readers. We are - Muse Event Planning and Production, LLC is a full service event planning and production company that can accommodate many types of events. From intimate dinner parties to events/weddings for over three hundred, we can handle every last detail. We work as your seattle wedding planner . We have an extensive list of award winning vendors and venues to build a team for our client’s events that will make any event picture perfect and stress free. We are well known as tacoma wedding planner or tacoma event planner . Our goal is to ensure that our client’s event is a reflection of them and their vision. Using creativity, attention to detail and experience in management and bookkeeping, we vow to create a one-of-a-kind event within their budget. We were rewarded as number one wedding planner seattle last year. We demand, we are the best wedding planner washington state . Thanks for your cooperation.