Saturday, April 7, 2012

The best and worst of the polling companies in the 2012 GOP primaries

As I (and many others) have noted, the 95% confidence interval is not doing what it is supposed to do in the polling results from the 2012 GOP primaries. When we add and subtract the "margin of error" from the given proportion, we are supposed to be 95% confident actual number in the true poll on election day will be within that interval from the opinion poll. One proviso is the line "if the election were held when the poll was taken", a statement that is almost never technically true, but should be close enough when the polls are relatively fresh, say within a week of election day.

Example: Let's say some candidate is at 47% in a poll with a "margin of error" of +/-3%. This means that 95% of the time, we should expect that candidate to get between 44% and 50% of the vote. The thing is, maybe the None of the Above count in the opinion poll is 8%, while on election day None of the Above (third party candidates, left blank, etc.) only accounts for say 1.5% of the vote. To be fair, we should scale the 47% to 47%/(100-8)%*(100-1.5)% = 50.3%. The new interval would be from 47.3% to 53.3%.

Using these methods, one polling organization with multiple polls this year stands head and shoulders above the rest. Suffolk University in Boston polled in New Hampshire when there were six candidates, and in Florida and Ohio when there were four candidates. Of their 14 numbers, they caught 13 in the margin of error, only missing Ron Paul's big outing in New Hampshire. At 92.9% accuracy, no one else comes close except for some companies that are perfect but have only one poll in one state to put them in contention.

Then we have the work horse companies, the ones that produce lots of polls in lots of states. Instead of 14 results, Public Policy Polling has 65 results with 47 caught inside the margin of error for 72.3%. (Remember, it should be 95%, so this is nothing to write home about, even if it is the second best around of companies taking more than one poll.) PPP did not do a good job of prediction in New Hampshire or Iowa, but since the field has been narrowed to four candidates, they have been perfect 7 times of 13 and better than the average poll 10 times of 13.

And then there's the other end of the spectrum. Polling companies come and go, and I did not recognize the name from when I was reading polls in 2008. The archive on their website only goes back to 2010. They have taken seven polls fresh enough to be considered in this sample and they have never had a perfect prediction. Even worse, they have never been better than average poll in any state and their "95% confidence intervals" catch about 32.1% of the actual numbers.

Polling companies come and go. If people looking for polling companies can actually do any comparison shopping - like visiting this blog, for instance - this outfit should not survive long.