Site Search Navigation

Site Navigation

Site Mobile Navigation

July 18: New Polls, but Little Change in Horse Race

By Nate Silver July 19, 2012 9:53 amJuly 19, 2012 9:53 am

Polls from major news organizations, including The New York Times, have a lot going for them. They make a real effort to take a true random sample of Americans — including the substantial number of Americans who rely exclusively on cellphones. They usually have strong standards of disclosure and transparency. They include long and detailed survey batteries, in addition to reporting the horse-race numbers. In an era of constrained news media budgets, they aren’t cutting a lot of corners. All of that counts for a lot, in my view.

At the same time, all polls are subject to sampling error – the variance that inevitably creeps in because you are surveying just a subset of the population rather than everyone in America.

And all polling firms need to make decisions about the right way to identify registered or likely voters, the right way to conduct demographic weighting, the right way to phrase questions, the right way to characterize undecided voters and so on. These are difficult decisions, and they introduce a lot of uncertainty into polling.

They can also introduce “house effects,” which are persistent tendencies to show more favorable results for one party or one candidate than the consensus of polling firms. To be sure, these house effects can also be introduced – intentionally or not — by poor pollsters who do take shortcuts, who have partisan agendas or who just don’t know what they’re doing. But even relatively strong polling firms can have them.

And no polling firm – however strong – can rid itself of sampling variance, although it can reduce it by taking a larger sample of voters.

The point is simply this: even very good polls can have more random noise than people usually realize.

On Wednesday night, there was a lot of cognitive dissonance in my Twitter feed when The New York Times and Fox News came out with polls at about the same time. The New York Times poll, conducted in conjunction with CBS News, showed Mitt Romney with a nominal one-point lead over President Obama. The Fox News poll had Mr. Obama four points ahead instead.

It’s easy to drive yourself to exhaustion by trying to determine which of these polls is “right.” Some analysts are always trying to unlock the Da Vinci Code by looking at the demographic subsamples in polls – a noble idea in theory, but a problematic one in practice (indeed, I think it is often counterproductive) because the demographic subsamples in polls are subject to considerably more statistical noise than the top-line results.

To be sure, I do not take the view that all polls are created equal. Nor does our forecast model, which weights certain polls more heavily based on their methodological standards and their past accuracy.

It’s my view that you’ll almost always be better served by taking some kind of an average of the polls, by looking at more polls rather than fewer (including looking at state polls) and by looking at the trend lines within individual polls, which allow for more apples-to-apples comparisons.

All of this is a long-winded way of saying that none of the polls out on Wednesday night ought to have changed your impression of the presidential race much. The New York Times/CBS News poll showed a fairly good top-line number for Mr. Romney. But The Times/CBS News has generally shown more favorable results for Mr. Romney (and less favorable ones for Mr. Obama) than the polling average.

The Fox News poll qualifies as slightly good news for Mr. Obama, since its polls have been slightly Republican-leaning so far this cycle – but it was really nothing that couldn’t be explained by sampling variance. And Fox News’s most recent poll, in June, had shown Mr. Obama five points ahead, little different from their new survey.

Over all, the FiveThirtyEight model has shown an extremely flat race for the past month or so. There is little in the way of persuasive evidence that the race is moving much in one direction or another.

If you want to indulge in a more elaborate hypothesis, I might offer this one. It looks as though there may have been some recent deterioration, by a percentage point or so, in Mr. Obama’s approval and favorability ratings – perhaps following the fairly poor economic news of late. However, the head-to-head numbers between Mr. Obama and Mr. Romney have been essentially unchanged.

So it’s a least plausible that Mr. Obama is being weighted down by the economic news, that Mr. Romney is being weighted down by the negative coverage surrounding Bain Capital, and that these effects are roughly canceling each other out.

It’s also plausible that all of this is just statistical noise. Furthermore, it’s plausible that, within a week or two, we will seem to see some clearer sign of a trend toward one or another of the candidates.

There were some interesting data points out on Wednesday apart from the national polls. For instance, a Public Policy Polling survey in New Mexico showed President Obama five points ahead of Mitt Romney.

Despite his holding the lead in the poll, this is an unfavorable number for Mr. Obama. Public Policy Polling has shown good numbers for Mr. Obama this year, both in New Mexico and in other states. Its previous New Mexico poll, in fact, had Mr. Obama up by 14 points instead.

Is the poll enough to start thinking of New Mexico as a swing state, something we’ve generally advised against so far this cycle?

In my view, probably not. Mr. Romney’s chances of New Mexico did improve materially on the poll, to 11 percent from 5 percent, according to our forecast model. But Mr. Obama had a pretty big cushion there: the demographics of New Mexico are quite favorable to him, and there’s little evidence that Mr. Obama is having trouble with Latino voters in other states.

Plus, this is the first poll of the cycle to show Mr. Obama with anything other than a double-digit lead in New Mexico. I hope that this survey will inspire other polling firms to take the state’s temperature — New Mexico has been sparsely polled. But in general, the underlying currents of voter opinion in each state aren’t shifting as quickly as the polls sometimes seem to imply.

Mr. Obama did get some good news on Wednesday in the form of an increase in housing starts. The news from the housing sector has been one of the few bright spots in the economy lately; any sort of economic upside case for the second half of the year probably depends on those numbers continuing to rebound. Our model’s economic index improved slightly on Wednesday because of the gain in the stock market, which reacted to the housing data and to testimony by the Federal Reserve chairman, Ben S. Bernanke, on Capitol Hill.

Still, for all of this new information, the forecast was not much changed. The model gives Mr. Obama a 67.9 percent chance of winning the Electoral College, versus a 66.8 percent chance before.

What's Next

About

This is an archive of posts from FiveThirtyEight, a blog founded by Nate Silver in 2008 and devoted to rigorous, data-driven analysis of politics, polling, public affairs, sports, economics, science and culture, from August 2010 through July 2013.