FiveThirtyEight |
The Uncanny Accuracy of Polling Averages*, Part III: This Time, It’s Different?

Site Search Navigation

Site Navigation

Site Mobile Navigation

The Uncanny Accuracy of Polling Averages*, Part III: This Time, It’s Different?

By Nate Silver October 3, 2010 9:40 amOctober 3, 2010 9:40 am

This is the third installment of a five-part series on the accuracy of polls and polling averages.

In Part I, I explored why our intuition may mislead us when it comes to forecasting the outcomes of elections — for a variety of reasons, we may tend to assume that there is more uncertainty in the forecast than there really is. In Part II, I demonstrated, by contrast, that a simple average of polls has performed very well over the past six election cycles in determining the winner of the contest. For example, Senate and gubernatorial candidates who have trailed by 6 to 9 points in the polling average with a month to go until the election have won their races only about 10 percent of the time in recent years.

However, just because an indicator has been reliable in the past does not necessarily mean it will be so in the future. For instance, it happens to be the case that when the A.F.C. team wins the Super Bowl, the stock market has tended to perform poorly in the next year, and vice versa. But most of us would grasp that there is little if any predictive power in the relationship. If the N.F.C. team’s kicker misses a field goal wide right in the closing seconds of the Super Bowl, handing the A.F.C. team a victory, would you pull your retirement savings out of the stock market when otherwise you would have doubled down? I’d certainly hope not.

The case that we’re studying here — the relationship between polls and election outcomes — isn’t quite like that. We probably all agree that the polls have at least some predictive power. But are there reasons to think the polls will be less accurate this year than they have been in the recent past?

This is what we’ll address in the final three parts of the series. Originally, in fact, I was planning to take up all of the objections in one concluding article. But because there are several different hypothesis to consider, and because they each deserve a full hearing, we’ll be breaking the article up.

So today, in Part III, we’ll be addressing whether there are things about this political cycle that make it harder to forecast. Part IV will address the question of whether polling itself is deteriorating in quality, irrespective of the political environment. Part V will take up a couple of more technical issues, like those having to do with overfitting and fat tails.

Part III

An objection I’ll sometimes encounter is that this is an unusual political cycle, and that makes it harder to predict. There are two versions of this argument, one that is pretty daft and the other of which is potentially more on-target.

The first form of the argument (see for example this treatment by the Weekly Standard‘s Jay Cost) is that voters have a lot to wrestle with this year — an economy in shambles, a widening national debt, two unpopular political parties, a changing media landscape — and it’s difficult to predict how exactly they might react. “These models over-simplify what are really very complex processes,” Mr. Cost writes.

I agree that voter behavior is inherently fairly complex — and it may be especially so this year. Fundamentally, the dilemma that voters are facing this year is that they’re very unhappy with the direction of the country — and therefore the Democrats who are in charge of it. But, also, they have a low opinion of the major alternative to Democratic governance, which are Republicans. This set of circumstances is relatively rare in recent American history; in 1994, by comparison, both Democrats and Republicans were more popular than they are now.

Suppose I had told you in January 2009, when Barack Obama took office, that — 20 months hence — people would have become no more optimistic about the direction of the country, but also, that favorability ratings for the Republican Party would still be near their historic lows. Then I’d asked you to predict for me what would happen at the midterms.

Any number of answers would have seemed plausible. You might have said …

“Well, Nate, the Republican Party has been in a state of long-term decline. If they haven’t been able to improve their image even as the public has grown skeptical of Obama, that speaks to magnitude of their problems. Maybe Democrats will lose a few seats in Congress, but I’ll bet it won’t be many — voters aren’t done punishing Republicans.”

Or you might have said …

“You know, Nate, elections are a referendum on the incumbent party, and Democrats dominate the government now. Once you told me that people would still be unhappy about where the country was headed, that was really all I needed to know. Democrats are headed for some pretty big losses. I don’t care how unpopular Republicans are — what will matter to voters is that they’re not Democrats.”

It turns out that the latter prediction appears as though it will be much closer to the mark. It appears as though most voters who are both unhappy with President Obama and who are skeptical of Republicans are going to vote Republican nevertheless.

How do we know that? Because that’s what the polls tell us, and they do so pretty definitively. On measures that ask voters to compare the two parties — and elections are exercises in comparison, after all — Republicans are generally getting the better of it, both at a local and national level.

The factors that influence this voter sentiment may well be fairly complex. But for forecasting purposes, that doesn’t really matter, provided that the polls — and the other statistical indicators that we look at — are able to do a reasonable job of absorbing that complexity. We aren’t trying to predict the outcome of the election from first principles, but instead from polls.

Now, it could be true that when voter sentiment is relatively more conflicted — or when it has undergone relatively more change — the polls will do a less reliable job of representing it. This is the stronger form of this argument. If voters are having a difficult time reconciling how they feel about the country and the government, then polls might be more sensitive to things like subtle changes in question wording. Or, even if the polls were doing a relatively good job of capturing the public’s mood at any given moment, it might be more volatile than usual, which would also imply greater uncertainty in the forecast.

I’m not unsympathetic to these arguments. But I don’t know that they have much evidentiary support.

According to an analysis by Daniel Hopkins, for instance, some of the worst years for the Senate and gubernatorial polls — like 1990 and 2000, when their average error was high, and in 1998, where they overestimated Republican support — were relatively normal (even “boring”) political cycles. By contrast, the polling was not obviously worse in “wave” cycles. The polling did slightly underestimate the full magnitude of the Republican wave in 1994, but only by a point or two. And the polling was quite strong in the Democratic wave year of 2006.

Polling also had a decent year in 1992, when voter dissatisfaction with the country was high, and you had the unusual contingency of a strong third-party candidate in Ross Perot making a credible bid for the White House. That’s a year that you’d think might be tough on pollsters — and, indeed, the polling in the Presidential race was extremely volatile over the summer months. But by Election Day, it had settled down, and the pollsters had one of their better years.

In 2008, you had something else that had never happened before — an African-American man on the Presidential ballot — and you had a financial market collapse, and all sorts of people who had never voted before entering the electorate, and everything else. But the pollsters did quite well, especially for the Presidential race.

Indeed, it is probably dangerous for us to deem ahead of time what constitutes a “normal” political cycle and what constitutes an unusual one. I’m sure that if you take an election that seems very commonplace in retrospect — say, the Presidential election of 1996 — commentators at that time had all sorts of reasons for believing it to be unusual. (The polls did pretty badly that year, by the way, significantly overestimating Bill Clinton’s margin of victory over Bob Dole.)

But even if we could do that — even if we could proclaim with confidence that 2010 is a unique year, not just in the ways that all elections are unique, but an especially peculiar one — it is not clear that there is any relationship between this and the accuracy of the polls. There hasn’t been any such relationship in the past.

By the way: there are some ways in which wave elections are easier for pollsters. In general, the higher the turnout in the election, the easier it is to poll: that way, pollsters have to guess at just one thing (whom the voter will vote for?) rather than two (will he or she vote at all?). Some wave years, like 2006, are associated with relatively high turnout. Others aren’t: the Republican wave year of 1994, for instance, was brought about more by especially low Democratic participation than high Republican turnout. But this year looks like more like 2006 than 1994: there is not a lot of evidence that Democrats are especially disengaged, and their turnout may be fairly average for a midterm year — instead, Republican voters seem unusually motivated, and that is what is motivating their strength in the polling.

What's Next

About

This is an archive of posts from FiveThirtyEight, a blog founded by Nate Silver in 2008 and devoted to rigorous, data-driven analysis of politics, polling, public affairs, sports, economics, science and culture, from August 2010 through July 2013.