Does This Phone Survey Flatter Me?

The evidence continues to mount that phone surveys should be a method of last resort. Pew Research Center has released results of an experiment conducted in 2014 to test mode differences in how surveys are administered. The research question was whether people answer survey questions differently when responding to a real person on a telephone vs. how they respond to identical questions on a web survey.

The answer is yes, they respond differently. The research found that mode differences are common and measurable, even if not very large. Most importantly, there is a systematic “social desirability” bias in how people answer questions on the phone. People want to look good when confronted with a real person. As such, in this experimental survey they were more likely to:

Give higher satisfaction ratings on their family life

Give higher satisfaction ratings on their social life

Rate their communities as being excellent places to live

Report talking with their neighbors

Rate their health as being excellent

Acknowledge discrimination among minorities

… and they were less likely to:

Express extreme negative views of politicians

Report being unable to afford food

Report needing recent medical care

Report not attending religious services

The study concludes:

The fact that telephone respondents consistently exhibit more socially desirable reporting is consistent with a large body of preexisting literature on the topic. For most of these and other differences described here, there is no way to determine whether the telephone or the Web responses are more accurate, though previous research examining questions for which the true value is known have found that self-administered surveys generally elicit more accurate information than interviewer-administered surveys.

Now add to this list of reasons the problem of social desirability bias. Survey respondents want to offer the most flattering portrait of themselves they can manage. But as a researcher probing for the best insights I can find, I want the whole truth no matter how ugly.