Yes, Polls Underestimate Trump

I’ve heard some claims from fellow Trump supporters that our guy’s support is underrated by the polls. This is a likely possibility and not even necessarily the result of bias, because all polls sample the people who the pollsters believe are most likely to vote; if a candidate comes in and pulls new voters with them, they’ll be underrated. I cross referenced Trump’s support in the Republican primaries this year with the polls to see whether they really did underestimate his support.

Scroll down to the first sentence with big letters if you don’t care to read about my methods.

To look into it I did a statistical meta-analysis of RealClearPolitics Republican Presidential primary polls at the state by state level for the 2016 Republican primaries. My null hypothesis was that the polls were equal to or greater than the actual Trump support, and my alternative hypothesis was that Trump support was significantly greater than the polling averages predicted.

The analysis covered 94 polls in 25 states. The rest either had no data, small sample sizes, were outliers, or were months old at the time of the primary election in their respective state. Caucus states were categorically excluded.

I found that the polls underestimate Trump support by 3.2% on average.

Mind you, this is even when they do everything right; recent polls with large sample sizes, averaged with other such polls, should give a pretty accurate result. But they generally fall below the mark.

The only important caveat is that my p value was 0.26. This means that there is, in theory, a 26% chance that the effect I observed is actually due to chance and not a reflection of flawed polling methods.

In practice, I’m willing to bet that the odds of this being a coincidence are less than that. Importantly, my standard deviation for predicted Trump support was massively, artificiallyinflated by the fact that I went state by state instead of looking at national polls, since primaries are done on a state by state basis. The fact that different states have different opinions about The Donald means that my standard deviation was higher than it would be for Trump support in general, and that means that my p value will also be higher.

A similar profile characterized Obama’s run in 2008: he was an “outsider” candidate (according to his marketing, anyway), it was assumed that he couldn’t win, but a bunch of people came out of the woodwork and he won with more votes than anyone before him.

Ryan Faulk found that Trump overperformed the primary polls by 3.3% when he did a similar analysis. He also discusses the possibility that the pollsters aren’t reaching Trump supporters proportionally.

The other explanation for this phenomenon is the Shy Trump hypothesis, for which the Morning Consult found some evidence; according to their findings, college educated Republicans are much more likely to admit to supporting Trump over the internet than over the phone.

Regardless of why it happens, however, the takeaway is that there is at least a 74% chance that Trump will do better in the general election than predicted. That chance is higher when you consider my mutant standard deviation.

For my next trick, I’ll see whether polls affiliated with or done by organizations implicated in the DNC leaks peg Clinton support significantly higher than other polls do. Stay tuned.