What’s wrong with surveys?

Written by Martin Nyløkke Gronemann

As companies revamp, what are people doing? If you go by all the surveys that keep telling us that consumers want to adopt a more environmentally friendly lifestyle, you’d think that people are pretty progressive. Yet the market seems addicted to surveying consumers about being green, which says something about the validity of the surveys themselves.

Companies take their carbon footprints seriously, redesign products with cradle-to-cradle compostable “green” packaging, and recycle a lot. These corporate efforts are commendable, but the focus on the supply side says a lot about what’s going on in the market. Companies obsess over tracking how green people are, which suggests a deep insecurity among businesses about whether people are really committed to being green–and big questions about how to get consumers more committed.

Why do we keep asking consumers the same questions? Edelman’s 2010 Good Purpose survey says consumers rank the environment as their #1 cause.The 2009 Cone Consumer Environmental Survey revealed that 35% of Americans have higher interest in the environment today than they did one year ago. Another 2009 survey conducted by Yahoo! revealed that 77% of American consumers describe themselves as green.

Is green consumerism really a growth market? Should companies adapt their value propositions and products accordingly? Will investments in green products automatically lead to increased return on investments and an improved brand value?

The gap between survey and reality

Something’s not right. For example, less than 1% of Danish households have installed solar power (ReD Associates, 2008), even though Danes are generally an environmentally friendly bunch. And despite all the green love in the States, the sales of SUVs in the U.S. have increased more than 50% since last year. When the price of gas skyrocketed, sales dropped. When the price plunged, sales rose. Consumers seem pretty simple.

So what’s going on? Why do surveys overemphasize our willingness to adopt a more environmentally friendly lifestyle when in reality we don’t follow through?

Is it just about bad surveys?

Many surveys aren’t well-constructed. Either the questions are poorly phrased or the conclusions are skewed because partisan stakeholders have commissioned those findings.

It makes you think when, in his Greenbiz.com blog, Joel Makower shows that Rockwell Automation, a manufacturer of equipment to make factories more efficient, found that Americans wanted “more energy-efficient production” and that Green Seal, a purveyor of eco-labels, concluded that one in three consumers say they don’t know how to tell if green product claims are true.

Conflict of interest aside, there’s a more profound reason why surveys aren’t a roadmap for how consumers really behave.

Our built-in biases make it impossible for surveys to depict reality

Surveys by nature present respondents with a limited number of options: “Do you perceive yourself as being green?” [Yes, No, or Don't Know.]

The problem is that surveys are distorted by what scientists call cognitive biases—our tendency to make systematic errors in decision-making because of a slew of factors that aren’t evidence-based. These biases influence the way we think and act—they are distortions in the brain that are difficult to eliminate and which lead to perceptual inaccurate judgment or illogical interpretation.

Whatever happens when people say yes or no on a survey isn’t mirroring the complex motivations and balances that take place in our brains while we make decisions. So when you ask someone point blank, “Are you green,” they’re not thinking about the trade-offs they’re making. Most likely they’ll think sure, I’m green. But if you ask them would you rather be green or get extra vacation time you might get a different answer. In real life, consumers have to constantly prioritize. Questions around environmentally friendly lifestyle cannot be separated from all other decisions consumers have to make.

How do theories explain the ways we make decisions?

Social scientists have numerous theories around these cognitive biases we possess. But as a general rule, these biases mean that people don’t always think rationally. One example of what people typically experience when taking surveys is a very influential cognitive bias called “framing”—it means the approach or issue is simply too narrow. So people will focus disproportionally on what they’re asked, which means the issue takes on more relevance than it otherwise may have in that person’s life—so the way a person responds is inaccurate because the respondent will have made a decision he or she wouldn’t ordinarily have made.

A second example of a cognitive bias is the way people place a higher value on objects they already own. If in a survey you mention something that has a role in someone’s life, like a particular refrigerator or a style of couch, they experience what’s known in behavioral economics as the “endowment effect”—they’ll respond more positively to questions about that object. This too skews the survey.

Good marketing tries to compensate for that. So in the project in Denmark we suggested that our client substitute the phrase “natural heating” for “green.” The idea was to express that the principle existed in nature already to underscore the fact that being sustainable is not a trend but that it is something more eternal.

Should you chuck your surveys?

None of this means surveys are useless. You can design better surveys or use surveys to validate the deeper information we discover while doing fieldwork (that’s what we do). This kind of methodology is increasingly being used in businesses today—companies are realizing that surveys aren’t telling them everything they need to know. So we come in and take to the streets to see how people behave. Then we fold surveys on top of that so the core data comes from observations in the field. (This is a way of getting more information against which we balance people’s cognitive biases.) The Q&As get pulled in along with the fieldwork data. The drawback is that you get a lot of data from the fieldwork that may or may not be relevant but the benefit is that you get a more holistic view of how people really feel.

With the Danish energy company, for example, we realized that people weren’t going to run out and buy their green products because people prefer immediate benefits versus long-term benefits, even if the long-term benefits are better. So we compensated for this by giving our client’s customers a solution that would let them make money from day one instead of down the road, once the green savings really started adding up.

So as you think about investing in a more environmentally friendly product or about refining your services, it’s useful to step back and think about the kind of data with which you’re working. If everything hinges on a survey or two, you may be missing the more psychological (and more accurate) stuff that goes on below the surface. If instead you’re confident that you’ve taken a deep dive into consumers’ real-world issues and you’ve also quantified and validated those insights, you’re already in a better position to make informed decisions.

Subscribe to this?

ReD is commited to maintaining a long conversation with it’s readers. Subscribe to get email notifications when a story is added to Sense making in business