Our accuracy compared with national poll averages

July 31st, 2008, 12:08pm by Sam Wang

The question arises: how good is the Meta-Analysis in comparison with a more conventional measure such as an average of multiple national polls? Answer: Today, the Meta-Analysis is over four times as accurate as an average of recent national polls. Furthermore, it gives an estimate in the units that matter – electoral votes. Here’s why.

Averaging is an excellent way to see past variation in individual data points. A standard measure of how well you know the true average is the standard error of the mean (SEM), which can be thought of as a multiple-poll version of the famous Margin of Error. Today, the SEM of the last 6 polls is 2.3%, a fairly typical value.

But the Meta-Analysis uses dozens of polls at any given moment from states in contention, and over 100 polls in all… Look at today’s Meta-Analysis, which gives an EV estimate of Obama 329, McCain 209, an EV margin of 120 EV. The Popular Meta-Margin is 3.36%, which gives a linear conversion of 120/3.36=36 EV per percentage point. (Remarkably, linearity is not a bad assumption, as seen in past election data.) Our 95% confidence band is 70 EV wide. Typically, such a confidence band is about 4 times the SEM, making our equivalent SEM 18 EV. This converts to about 0.5% – less than one-fourth the SEM of the Pollster.com average.

An important caveat is individual states are polled less frequently than the nation as a whole. Therefore the Meta-Analysis responds more slowly to changes in opinion. But since it is so much more accurate, it’s still likely to be a better way to detect swings. I can’t prove that – not yet, anyway…