Late Breaks and Polling Biases

November 2nd, 2014, 12:00pm by Sam Wang

At least one journalist is chattering about whether there’s a late break in polls for Republicans…based on one data point, which is probably statistical noise. Some people are hopeless. Then again, several polls today have pushed the Meta-Margin almost as far as it’s been toward Republicans this campaign season.

He is missing a far more important point: Final election results can vary across-the-board from midterm polls – in the same direction. This last-minute polling bias is typically 2-3 percentage points – five times larger than the bias in presidential years. The direction of the bias is unpredictable. (Read this for a review of the subject.) This is why I care about the exact margins for front-runners McConnell (R-KY) and Shaheen (D-NH). Their states are early-reporting. From them we can make a rough estimate of nationwide polling bias, as follows.

Even compared with the last week of polling, results can differ substantially by several percentage points. Even at the last minute, the bias (or bonus, if you look at it from the point of view of final results) can be rather large. Candidates could still win if they trailed by a margin of less than 3 percentage points in the week before the election. Here are the details for 2010 and 2012:

(click the image to see data going back to 2004)

Here, negative numbers (in red) indicate a GOP final win or polling lead, positive numbers (in blue) indicate a Democratic final win or polling lead. In 2010, the error was enough to flip the result to the opposing candidate in Colorado and Nevada. In 2012, no such errors occurred. The last column is the “bonus”: how much either party overperformed polls on Election Day.

Democrats, do not be fooled by this sea of blue. The error can go in either direction, as I wrote on Friday. It is not correlated with which party had a wave that year. It might possibly have to do with late shifts in voter opinion.

Statistically, we know that this is a correlated error. Here is why. The median year-by-year error, which measures systematic error (i.e. nationwide error) is much larger than the standard error of the mean, which is much less than 1 percentage point. This proves that the overall polling bias bounces around substantially from election to election. Furthermore, midterms vs. presidential years differ (using a one-tailed t-test, p=0.03). In short – midterms are weird, and there may well be an unpredictable overall error this year. There are six Senate races whose medians are within two percentage points. Republicans could win all six – and Democrats could win all six. Based on past midterm polling, both of these outcomes are within the range of possibility.

Here is the nationwide bonus compared with polls in Senate races that were ultimately decided by 10 percentage points or less (for data going back to 1990 read this).

Let’s call this last-minute bonus “Delta.” It is the opposite of the bias I am writing about. I am proposing to estimate Delta as early as possible on Election Night. In Kentucky and New Hampshire, which end voting very early, I hope to get some indication of what Delta will be.

For example, if McConnell wins by only 4 percentage points, he is underperforming polls by 2 points – and so might Republican candidates in other states. If he wins by 9 points, Republican candidates in other states might also do better than expected. Conversely, if Shaheen only wins by 1 point, this would indicate trouble for other Democrats. And if Shaheen wins by 5 points or more, that is good for Democrats nationwide.

This calculation of Delta might give an early-evening indication for four close races that should eventually be resolved by the end of the night: Iowa, Colorado, Kansas, and North Carolina. It will be of less help for Alaska, which is slow to report, and Georgia, which might go to a January runoff.

36 Comments so far ↓

Wonderful tip, Sam, regarding watching McConnell’s and Shaheen’s opening numbers on election night. There could also be another clue as early as 7:00 p.m., when all polls in Kentucky close and counting in most of the state would already have an hour head-start. If at that point, Wolf Blitzer on CNN uses the phrase ” too close to call ” it would likely rule out a Republican wave – especially for a GOP state like Kentucky, where a GOP call would be far more likely to be called from the outset – and that might even hint at a possible Democratic tilt for the night, regardless as to how the Kentucky race specifically turns out.

You might also look at the VA Senate race. It’s not that R has a chance, but it could it closes relatively early, will be called relatively quickly and should help with your “Delta Project” (i.e., does Warner over/under perform?)

What techniques do political pollsters have to account for sample fatigue? Alaska, with its small population, has had a ridiculous number of polls, and at this point most of the people I know actively screen calls to avoid them.

I would imagine, although my imaginings are based on circumstantial evidence, that the people who take polls at this point are the people with the strongest opinions, with the rest self selecting out.

Is there any conventional wisdom as to how that should be weighted in the final result?

I’ve wondered the same thing. I seem to be on a lot of Colorado likely Hispanic voter phone lists (or there are just a ton of polls using the same lists), to the point where I stopped answering the phone for any number I didn’t recognize. I’m not sure what bias this would cause or how one could take this into account, but it may reduce the correlation between polls (i.e. the people who respond to earlier polls are disproportionately likely to get many calls, and thus disproportionately unlikely to respond to later polls, so in effect successive polls are filtering out people they contact from the pool of probable respondents).

Naively, one would not expect much of an effect, because most polls have a tiny sample size compared to the overall population and should not be repeatedly contacting the same few people. But I suspect that a lot of polls, especially those run by partisan groups, are working from the same relatively short lists of likely voters in key demographics. I have difficulty believing that polling organizations collectively have the staff to call every Hispanic likely voter in Colorado the number of times I (and my family members) have been called (along with all the other demographics of interest in each state). You’d need a full-time dedicated staff of thousands, so it must instead be the case that these lists share either some pre-selection process, or contact information shared between political organizations about their donors.

Of course, this would typically increase correlation between polls, since people on these lists may share some traits, and are more likely to respond to more than one poll. So poll fatigue would probably just partially cancel out that correlation.

None of this really clarifies what direction you would have a systematic bias toward (if either). To speculate completely, I would guess that for close elections, those supporting the incumbent would be less excited about the race, and thus more easily fatigued by polls.

Sam,
The connection seems tentative from the data. The wave moved toward Ds by 3.7 and 2.7. This gave us 4 potential switches, but the Ds only capitalized on 2 (Nev, Co but not Ill in 2010) and not Nev in 2012. The range of change (0.8 to 7.7) across states makes it harder to relate changes in one state to predicted in other states.

Having said that, a 50% swing in the 4 states Rs lead by less than 3% (AK, IA, CO, GA) would result in a 50-50 Senate.

First, the median year-by-year error, which measures systematic error (i.e. nationwide bias) is much larger than the standard error of the mean in any given year. Therefore we know that polls as a group have a bias that differs from zero. So I regard it as proven that polling errors bounce around substantially from election to election. There is no aggregation method I know of that addresses this error.

Second, compare midterms vs. presidential years. That difference is significant by a one-tailed t-test, p=0.03. So midterms are weird.

Therefore it is basically settled that there should be an unpredictable national bias this year. The only thing left would be to figure out a way to guess its direction and size. I have not identified a way to do that. I don’t think early voting will work.

Today I met Chris Matthews backstage at NBC. He told me he thought Democrats would bomb because MSNBC’s ratings were through the floor. I pass this on without further comment.

For some reason I am seeing a lot of commenters here who do not understand what I am saying. If there is anything I can say differently, please enlighten me.

Sam,
I agree that there is an unpredictable national bias this year and I appreciate the details you add to the discussion. I also agree that all 6 “at risk” seats could go either way.
I look at the 2010 and 2012 tables. There was 1 “at risk” R seat and 5 “at risk” D seats in 2012. Even though there was a 2.7% D Bonus, no seats switched due to the Bonus.
I assume that the 2.7% Bonus was calculated as the average difference between Poll & Final for all the individual Senate races.
In 2010, there were 3 “At Risk” seats with a average Bonus of 3.7% D. 2 of the “at risk” seats switched, 1 did not.

Looking at the Kentucky & New Hampshire races may indicate the direction of the Bonus, but not the average value or how a Bonus in either state would relate to the value in Alaska.
(2012 3.7% average bonus but only a D Bonus of 0.8 in Nevada race is not enough).

I think I understand your points and tried (maybe incorrectly) to extend the point.

I’m not sure I know a better way to say it, Dr. Wang. But in specific response to JayBoy2K;s first observation, it is irrelevant (statistically) whether only two of the four switches were capitalized on (and it was really only 3 potential switches, since IL was not within the 3 point margin Dr. Wang describes). The data doesn’t indicate that a race within 3 points as opposed to between 3 and 10 has a significantly different amount of shift from the polling median to the final result.

As for the range of changes across states, that’s what we have statistical calculations for. It may *look* wide, but the statistics showing that the nationwide bias is larger than the standard error of the mean take that into account and still come up with a significant difference.

What it comes down to is that the statistics tell us some very specific things – that there is often a meaningful bias and that midterms have a larger bias the presidential years.

The main caveat I have about these conclusions is that our sample sizes are so small. One reason poll aggregation works well is because to some extent it’s like having a much larger (and probably more normally distributed) sample. However, when determining the statistical bias as we are here, each race is essentially one data point. So our data set is pretty small, even if you go back to all the close races since 1992. That one-tailed t-test does take sample size into account of course. In any case, it occurs to me that if there is any way to determine the direction and size of the bias, it probably lies in expanding the analysis of the bias to include the individual polls, not just the median prediction for each race. However, I’ll freely admit that I can’t picture how one would do that; it just seems that the only way to determine the bias is with more data, and the individual polls are the main additional data we have that isn’t at least one step removed from exactly what we’re looking at.

One fascinating – and somewhat timely – tidbit, Sam, in your graph of 2004. In the Kentucky senate race of that year, the polls of the day gave the Republican incumbent a 9 % lead. He eventually won by only 1.32 %. The polls therefore understated Democratic support in that state by over 7 %. In your power rankings, McConnell has currently a 7 % lead. It’s a fascinating coincidence.

I think looking at the 2004 Kentucky Senate race to try and rationalize the polls being wrong in 2014 is a reach. In 2004, the race was not thought to be close, so there wasn’t a lot of polling done (At least from what I see on RCP). Also Jim Bunning ran a terrible campaign, did a debate via video sync and used a teleprompter during it, made a few strange statements, all of which lead many to think he was beginning to suffer from dementia.

In 2010, which isn’t included in the chart, if you go off of the RCP average, Paul was +11.0 and won by +11.6, so the polls underestimated Republican support by 0.6.

Given that PECs forecast has moved toward the GOP by 15% in the past few days, it seems to my eyes that there’s more than an “outlier poll” at work. There’s been a switcheroo in the tone of your posts, Sam, from urging people to trust the polls to urging caution over them.

Also, how on earth can the snapshot — which seems to show the GOP with an 80% chance not 65% — be rather different from the forecast? Shouldn’t they be the same at this stage?

I mean mostly in comparison to prior years. Your methods have changed a lot, toward accounting for much more uncertainty, and so has the tone of your posts. What would your 2010 or 2012 models, or say the first version you ran this year, say about the chances of the GOP winning the senate now?

Thanks for the excellent cautionary note. Yes, I too caught the extraordinary weight being given to the Seltzer Iowa Poll showing Ernst up by 7, one that came out the same day and using the same time frame and a larger LV model, Yougov, the Yougov poll giving Braley a 1% lead.

I respect your thoughtful analysis, Professor Wang. The truth is, there is no comforting port in the storm for Dems in the senate at any site right now. I am sadly resigned to the likelihood of the GOP taking over the senate with all the ugly shenanigans that will entail. However, if there is a systemic 2% bias in favor of the GOP (and this has seemed to be the case lately), couple with a better GOTV effort by Dems, I could be pleasantly surprised. My gut bet, looking at all the polling? This will not be decided election night, and will come down to run-offs in Georgia and Louisiana. I do believe that at least one of Iowa, Colorado and Alaska will be a “surprise” Dem win, but also will not be shocked if Hagan and/or Shaheen loses in a mild upset.

There is no way to underestimate the TV ego at work. The fact that Mr. Mathews seems to have arrived at his summary insight based on his woeful ratings is amusing. It must be the excuse he is giving his soon-to-be ex bosses. None the less, it is indicative of this teeter-totter year that he has at least a 20% to 80% chance of being right.

Hmm…is it possible that some of the midterm bias/bonus effect is the result of partisan polls representing a larger percentage of the total poll population? I wonder what the bias would look like if we removed all polls from the pollsters at either end of the spectrum…cut out something like 15% on either end. This starts to feel like “secret sauce” territory, but I have to wonder how much those consistently off pollsters might be contributing to this effect. Earlier in this election, in response to a 538 piece, I ran numbers on Rasmussen’s 2010/2012 mid-October polls vs results. They ran R+4 to R+10 in every competitive race. Could this sort of factor be a contributor, and would eliminating unreliable pollsters help reduce the margin of bias?

I think the thing that may give shocking results is higher than expected turnout for a Non-Presidential election. Just since the Primary election in May, my home state of KY saw an increase of 41,000 new voters. The turnout in some states is higher than expected in the “swing” states according to the report I just saw… Some 600,000 voters who have never voted before. The polling is just an educated (I hope) guess on the make up of the electorate. If this unknown variable changes on election day, all these races could be shockers. Hagan may lose NC. Braley may win easily in Iowa. It’s hard to say until votes are counted. And, I still think it’s still possible McConnell could lose. Of course, he could win by 10, too. After Cantor, who knows how voters feel. We are living in odd times politically. Mitch’s poll average is still less than 50%, which is not good for an incumbent. You base your predictions off polling, but it’s far from an exact science. Some are angry; some are hurting in their lives. It can make them vote differently than expected. The bias may well be more about incumbent than Republican or Democrat.

Dew – you make a fascinating point about the 2004 Kentucky senate race. Yes indeed, the Republican ran a terrible campaign. Many Republicans were worried about that. But the fact still remains that the polling – in spite of all of that – underestimated Democrat support by over 7 %. The difference between a polling average of a 9 % lead and a razor thin 1.32 % win is immense. The fact remains – the polls were spectacularly out, and there can be no explanation for that outside of faulty polling methodology.

It is now the day before the election and the meta-margin is R 1.0 %. On October 22, Harry Enten of 538 offered the thought that the polling errors of the past have ” likely ” been corrected. If the polls are astonishingly accurate this time, and Republicans have a true meta-margin advantage of 1 %, they will win, and Mr. Enten will be right. But if the Democrats are underestimated by an average of 3.1 % in all close senate races – as they were in 2010 – they will win. Of course, they actually don’t need as much as that to shift the results decisively. If that happens, not only will it be a great night for Democrats, but the polling firms will – yet again – be going back to the drawing board in their continual struggle to create a ” likely voter screen ” that manages to keep pace with a continuous and rapidly changing demographic reality in the country, which always appears to be a step or two ahead.

You are a brilliant predictor, Sam, and I have followed your web site for a while now. Please don’t take this the wrong way, but I am hoping like crazy, this is the year that all your predictions are wrong—and the Full Moon drives as many Outraged Democrats to the polls, as there will be Republicans………See you on the other side……..:)