Two Polls That Highlight the Challenges of Polling

On Monday, SurveyUSA did something unusual for a pollster: It released two polls of Colorado voters on the same day.

Despite the similarity in the overall result — both showed the Republican, Cory Gardner, ahead by a slight margin — the differences in how the polls were conducted reveal the challenges and choices facing pollsters.

The survey conducted for The Denver Post was like most SurveyUSA polls. It randomly sampled (with random-digit-dialing, also known as R.D.D.) Colorado adults on landline telephones using automated interactive voice response technology, and it had an online panel to reach voters who do not own a landline. The survey for High Point University called people from a list of registered voters, known as registration-based sampling or R.B.S. Those voters with a cellphone but no landline were polled in live phone interviews instead of automated calls.

There’s a longstanding debate among pollsters and those who depend on their data about which technique is better. The two Colorado polls come as close to directly comparing the two methods as it gets: two polls of the same state, by the same pollster, conducted at nearly the same time.

Most campaign pollsters use registered-voter sampling, in part because the voter file has additional data on people’s demographics, party registration, location and vote history that can be used for weighting the sample and result. Most news media polls, on the other hand, use the random-dialing sampling method. Such sampling can claim the theoretical benefit of being closer to a true random sample, since registered-voter samples can miss voters who did not give their telephone numbers on their voter registration forms. Capitalizing on the benefits of registered-voter sampling can also require pollsters to make assumptions about the composition of the electorate, which makes news media organizations uncomfortable.

The polls also stirred a different debate, this one about Hispanic voters. The SurveyUSA poll showed Mr. Gardner faring exceptionally well among Hispanic voters, causing many to question whether the survey had adequately sampled a challenging demographic group that’s essential to Democratic fortunes.

I asked Jay Leve, SurveyUSA’s founder, about all of these issues. Here is an edited version of the interview.

Image

Cory Gardner, a Republican running for senator in Colorado, with a supporter at a meeting with Latino Republicans in the state.CreditBrennan Linsley/Associated Press

Q: You usually use R.D.D. sampling for statewide polls, presumably because you think it’s preferable. Why did you do an R.B.S. sample with High Point?

A: We gave choices to H.P.U. in spring ’14 at their request: we’re comfortable with R.D.D. or R.B.S., contacting cellphone voters from Internet or calling them with live interviewers. When High Point looked at what they wanted, they said: R.B.S. sample with live operators dialing cellphones.

Q: Did you have any concerns about using R.B.S. sampling? About a decade ago, you polled a Cleveland mayoral race with both R.B.S. and R.D.D. sampling, and suggested that there were coverage issues with R.B.S. sampling. (Not everyone gives their telephone numbers when they register to vote). Did you have questions about the quality of the voter file?

A: Other concerns have arisen since 2005, with nonresponse and trouble reaching people with live telephones, so right now we’re happy with using a voter list sample. We did not hold our breath in 2014 about R.B.S. coverage issues.

Two other things help. The state of voter list samples has improved, and many voters choose to give their cellphone to the registrar or Secretary of State when they register. So you end up with a company — Aristotle, in our case — that will sell you a voter list sample with cells and landlines, and that makes one-stop sample shopping easier.

Q: Why is the Hispanic share of the electorate so much larger in the R.D.D. (16 percent of likely voters) sample than the R.B.S. sample (6 percent of likely voters)?

A: We need to weight our respondents “back” to something. In R.B.S., we’re weighting it back to the voter file. The problem is that in Colorado, like many states, the voter’s race does not appear on the voter list file, so as a result, the R.B.S. study was not weighted to race at all — it was weighted to gender and age, but not race. And that explains why it is whiter. In R.D.D., we weight to census targets for gender, age, and race, and end up with larger pool of Hispanics, for better or for worse.

Q: Did you weight by party registration? (The Colorado voter file includes information on the party registration of every registered voter). Do the party affiliation figures reflect party registration or self-identified partisanship?

A: We didn’t weight for party registration. The party affiliation figures are for self-identified party-ID, not party registration. I don’t have the party breakdown of the sample at my fingertips.

Q: Over the last few weeks, Fox News released a number of R.B.S. surveys that were pretty good for Republicans. Do you think there’s any reason to think that R.D.D. provides better answers for Democrats this cycle?

A: I’m not sure. We rarely get a chance to A/B test this.

What I will observe is that there was a fascinating difference between landline and cellphone respondents. This was the first time I’ve seen that. Our landline respondents between the two polls were effectively the same. The cell respondents by live operators were 12 points less Democratic than the ones interviewed on the Internet, and that’s significant. And I can’t give you, yet, a full considered response to why that would occur. But I will observe that it accounts for much of the difference between these polls. I’ll have to look at it further; it’s the first time I’ve been confronted with such a striking difference with polls that were otherwise as similar to each other as these two are.

Q: Another difference was age. Seniors represented a far larger proportion of the electorate in the R.B.S. poll than the R.D.D. poll.

A: Here’s the way I would say it: In the R.B.S. sample, 77 percent were age 50 or older; in the R.D.D. sample, 53 percent were.

Now, what could account for that? If you say as a stipulation when drawing the voter-list sample, as High Point University did, that you only want to look at registered voters who voted in both 2010 and in 2012 — you can imagine how that squeezes the young end of the age balloon. I don’t think there was any deliberate attempt by High Point to make the sample older, and I think High Point would be surprised to see that it did, but an older respondent pool is what results if you only look at past voters.

Q: So the sampling frame was registered voters who participated in both 2010 and 2012? That’s a tight screen. And you then applied a likely-voter screen as well?

A: Yes, there was a question that asked are you “certain,” “probably” or “50-50” to vote, or “will not” vote. For High Point, we allowed registered voters who said they would “definitely” or “probably” vote to be likely voters. For media polls, that’s not our custom. In our experience, those who tell you in an R.D.D. vote “definitely” are in fact the voters — “probably” are in fact the posers. The screen was more relaxed on the R.B.S. side because we had preselected the likely voters at the time the sample was drawn.

Just 85 percent of the self-identified Democrats said they would “certainly” or “probably” vote. Ninety-five percent of Republicans said they would “certainly” or “probably” vote. Fifteen percent of Democrats said they were 50-50 — four times as many as Republicans. And you can see how that moves the numbers: The registered voters who participated in 2010 or 2012 — or were newly registered since 2012 — identified as Democrats by a 35 to 33 margin. Among the likely voters, it’s Republicans by 34 to 32.

Q: I noticed that the polls, particularly the R.D.D. poll, offered some pretty bleak figures for the Democrats among Hispanic voters. As you know, there’s a long discussion about whether polls are doing a good job of sampling Hispanic voters, or at least the most Democratic-leaning Hispanic voters. There’s also a theory that Democrats have tended to outperform the pre-election polls in states like Colorado for this reason. Is there anything you’re looking at that gives you confidence in your sample of Hispanic voters?

A: I don’t know that I’ve ever seen a good sample of Hispanics. It’s a real challenge. It’s a tough population to poll.

There’s a decision at the outset: Are you in a position to offer your clients bilingual interviews, and are they able to pay for polling both in Spanish and in English? When we give our clients the bilingual option, many decline for cost reasons. When they accept it, like Nevada in 2012, where all of our polling was bilingual, I don’t know that we got one micron closer to the Latino population than had we polled in English only. Our data was criticized at the time for being too Republican, for being at odds with common sense. I get that criticism; I understand it. And the Hispanic data that you’re looking at in Colorado, that shows a Republican ahead among Hispanics, is also at odds with common sense. So I can’t defend it except that we give people the opportunity to self-identify as Hispanic, and we record it.

We have been accused in the past as having blacks who are not “black enough.” I get that criticism. Our black respondents, instead of being 90-10 Democratic, are sometimes 67-33. Do I think it turns out that way on way on Election Day? No, I think we’re too Republican on black voters, just as we are sometimes too Republican on Hispanic voters. This is not unique to SurveyUSA.

Are there people who specialize in Latino polling who conduct elaborate studies and then in turn prove, to their satisfaction and probably mine, that the Latino population is overwhelmingly Democratic? Yes. Is there something that we can do better? I’m sure that there is. At the moment, though, it is what it is. It’s what the respondents tell us when we give them a chance to identify as Hispanic and we ask them for whom they’ll vote.

Q: If you think that your surveys systematically understate Democratic margins among nonwhite voters, how should poll consumers think about your results? Do you expect that your polls tend to understate Democrats, as many polls have in recent cycles? Or do you think that there’s some countervailing, pro-Democratic bias among white voters that happens to cancel it out?

JL: Poll consumers should look at SurveyUSA’s entire body of work — which goes back 20 years and is the largest of any state-level pollster. Seven-hundred-and-twenty-two of our recent statewide election polls were analyzed by FiveThirtyEight. They found across our entire body of work a Republican bias of 0.2 percentage points. That means our polls would have been more accurate, on average, had you taken away a tenth of one percentage point from the Republican candidate and given that tenth of one percentage point to the Democrat. I don’t think the polls we publish need to be unskewed any more than that.