Random blog posts about research in political communication, how people learn or don't learn from the media, why it all matters -- plus other stuff that interests me. It's my blog, after all. I can do what I want.

Monday, April 17, 2017

Face-To-Face vs Online Surveys

The general idea is controversial or sensitive survey questions get very different results if asked to respondents face-to-face or on the phone versus a more impersonal approach, such as online. Let's look to see if that's the case using fresh ANES 2016 election data and whether a survey was online versus face-to-face.

Is Obama a Muslim? Percent who say Yes:

F2F: 31.1%

Online: 30.3%

Voted for Trump

F2F: 41.3%

Online: 39.2%

OK, let's look at these first two above. Clearly the Obama Muslim question has no mode effect, in other words asking it face-to-face versus online makes no difference given it's 31.1 versus 30.3 percent thinking he's Muslim. On voting for Trump there's no real difference either. We're talking a couple of percentage points, nothing significant or at least not substantive. OK, let's try another.

Should transgender people have to use their birth sex bathroom?

F2F: 50.2%

Online: 52.6%

Slight difference above, but again it's slim, too slim to have any real meaning. How about something more mundane.

Do you attend religious services?

F2F: 64.5%

Online: 58.2%

There's something going on in the question above, one that fits theory. Usually a socially desirable response (attending church, etc.) gets more "yes" responses in a phone or face-to-face survey, and that's the case here by several percentage points, enough that I'd argue survey mode matters. Here's another that's kinda interesting below:

Favor a wall on Mexican border

F2F: 29.5%

Online: 33.4%

That's a decent spread above, enough that I'd argue we have a small mode effect. Respondents were a little more willing to favor building a wall in the online group versus the face-to-face group.

I can do this all day, but I'm running out of time. As we can see above, there are mode effects and they do matter, but sometimes they don't matter at all.