Is polling science, art or witchcraft?

posted at 3:31 pm on September 30, 2012 by Jazz Shaw

One of the stories I’ve been following here at good ole’ Hot Gas this month has been Allahpundit’s ponderings on precisely what value – if any – the polls have in gauging the temperature of the electorate. (Here and here this week.) I’ve not only read AP’s questions and coverage, but many of your responses on the subject, and I’ve got to be honest here… I was still completely confused on a couple of fundamental points.

One of the first has to do with a subject which the current Oval Office occupant likes to trumpet… arithmetic. (Or, as Joe Biden might put it, a simple three letter word: MATH.) I mean, given all of the time that we spend obsessing over the polls here – as well as on every cable news channel – you’d think there was some actual, er… science behind it, wouldn’t you? Polling has been going on for longer than I’ve been alive and there are major elections every two years. Surely by this time somebody could have looked at the results of the various polling agencies and compared them to the final vote totals in the myriad races and determined who was hitting on all cylinders, right?

Apparently not. But some of AP’s questions have raised the specter of criteria we could use to figure out if anyone is putting their thumb on the scale and – more to the point – when. For the polls which publish their cross tabs, is there any metric where we could see if some of them are running an incredible D+ 1 bazillion in September and early October, but then suddenly push the margins down toward reality during the final two weeks so their “final” predictions would be more in line with reality? It seems to me like that would be a useful piece of information which could have been compiled by now. But has it?

Over the last couple of days I’ve been trying to track down some answers. This effort included speaking with polling analysts and pollsters themselves. The answers I received ran the gamut from things which sort of made sense when I heard them to contradictory responses which left me scratching my head. The last interview I did, and one of the most enlightening, was with Brad Coker of Mason-Dixon Polling and Research. The following are a few of my conclusions about the questions above when it comes to the science of polling.

First, there are varying claims about how pollsters arrive at the total number of people from each party affiliation (plus independents) who are to be surveyed in any given poll. One analyst said that it’s barely even a concern; they just pick a target number of interviews and let the chips fall where they may. (Within reason.) Brad wasn’t exactly that “hands off” about it. He said that they know from experience and previous results how many people of each affiliation are out there – more or less, because it does shift – but it’s one of the less reliable demographics. If he gets a number that’s totally out of reality in either direction and he has the time, he’ll try to get some more interviews to even it out. But the general consensus was that pollsters don’t start out shooting for “x” number of Democrats, “y” number of Republicans and “z” independents. Take that as you will.

Second, everyone seems to agree that poll results shift as you get closer to this election. But as opposed to some nefarious plot to influence the vote, the virtually unanimous response from these industry insiders is that polling is extremely tenuous a year in advance of an election. Identifying “likely voters” at that point is impossible to pin down for a variety of reasons, including people who move, people who only come of age to vote shortly before the election, people who die, and folks who simply aren’t paying attention that early and may not have any clue if they’ll be voting or not. In the final weeks before the election you can structure a much better likely voter model, and this will tend to shift the numbers when it happens.

Next, the subject of “weighting” was addressed by a couple of people. This, in my opinion, is where we really get into the “man behind the curtain” mystery ride. The people I spoke with were pretty much in agreement that weighting is done primarily as a matter of experience in the field and it happens on an ad hoc basis. For example, young people are harder to get a full interview with than seniors, so they typically get less responses from them. In response, they will “weight” the results to increase the influence of younger voters and decrease that of seniors if the number of responses is too far off from the usual turnout numbers. The precise numbers for that weighting don’t come from any handbook or specific formula… you just have to know how to do it.

The only thing everyone seemed to agree on was that no legitimate pollsters are influenced by the media, by partisan bias or money. (The term “legitimate” in this case is meant to exclude campaign push polls and marketing calls.) They’re just producing data based on research and they all seem to feel that their results have panned out pretty closely to actual results over the years, with the notable exceptions of 1980 and 2000, and to a lesser degree, 1996.

“There is terror in numbers,” writes Darrell Huff in How to Lie with Statistics. And nowhere does this terror translate to blind acceptance of authority more than in the slippery world of averages, correlations, graphs, and trends. Huff sought to break through “the daze that follows the collision of statistics with the human mind” with this slim volume, first published in 1954. The book remains relevant as a wake-up call for people unaccustomed to examining the endless flow of numbers pouring from Wall Street, Madison Avenue, and everywhere else someone has an axe to grind, a point to prove, or a product to sell. “The secret language of statistics, so appealing in a fact-minded culture, is employed to sensationalize, inflate, confuse, and oversimplify,” warns Huff.

You keep trumpeting that editorial. I don’t know why. Pointing out that a poll with a D +9 sample isn’t predictive of an election outcome has nothing to do with believing polls are historically anything. It wouldn’t matter if polls have always been perfect predictors — the poll at hand with the +9 sample is obviously not.

And your editorialist writes things like this, anyway:

The criticisms are largely unsound, especially when couched in terms like “oversampling,” which implies that pollsters are deliberately rigging their samples.

This is something he has in his own head, this definition of oversampling. A D +9 poll has oversampled democrats by four or five points, assuming the poll was supposed to be of any use whatever. And his qualifiers, like “largely” here, make his editorial about as useful as the recent, oversampled polls.

Nothing else in that editorial is even remotely interesting to me, so I’ll drop it.

But what’s your point? Do you want us to stop doing math? Because seeing the bias in most of these recent polls isn’t difficult. It isn’t subjective. You should be able to see it too.

Polls have become like rating of a TV show. They have nothing to do with reality and change as often the wind. One week it get millions and then the next week does not, with no real reason why. It is not like something happened on the show that influences people not to watch or too watch. Something made the people in Ohio suddenly go for Obama by 10% more then just a few days ago. Or in Pennsylvania when when poll has Romney by 2 and Obama by 8 on the same day.

If they poll people on how much they like chocolate they will get swings all over depending on who they ask. No poll with a sample of 800 people will be valid for Mars to change the taste of the chocolate and neither should a poll influence how someone feels about the race.

That is what is all about, influence. When the media shout of a new poll that says Mitt is down by 10 and only yesterday he was down by 1 with nothing happening in the 24 hours there has to be some sort of self fulfilling prophecy in play and they don’t care if that poll ask only 10 people at a soup line. It is all about the narrative and then talk about that for the next 24 hours until the next poll has it Mitt by 1 then nothing on that one.

They’re just producing data based on research and they all seem to feel that their results have panned out pretty closely to actual results over the years, with the notable exceptions of 1980 and 2000, and to a lesser degree, 1996.

That’s a lie. Near election day, they will change their partisan or demographic split to be closer to what the actual results are. But weeks or months before the election, they skew it intentionally toward who they want to win. They did it against Hillary and she kept winning state after state. They did it in 2010 where most everyone was wrong. They did it in 2008 and overestimated Obama’s win. They did it against McCain. And they’re doing it now.

If it was a one time thing where they got it wrong, that’d be fine. But partisan split doesn’t just shift within a matter of weeks before the election. The fact that polls change in the days before the election shows that the polls are propaganda before that date.

If it was a one time thing where they got it wrong, that’d be fine. But partisan split doesn’t just shift within a matter of weeks before the election. The fact that polls change in the days before the election shows that the polls are propaganda before that date.

MrX on September 30, 2012 at 4:03 PM

Sort of like how the unemployment numbers each Thursday are “unexpected” and the previous week’s number is almost always (all but 2-3 weeks in the last year) REVISED UPWARD.

You can credit stuff like that to mistakes if the mistakes don’t consistently help one particular side…

One of the stories I’ve been following here at good ole’ Hot Gas this month has been Allahpundit’s ponderings on precisely what value – if any – the polls have in gauging the temperature of the electorate. (Here and here this week.)

Not that these polls shouldn’t be taken with a grain of salt due to their oversampling of Dems, but on the other hand we wouldn’t be questioning them, but rather taking them as Gospel if they showed Romney to be way ahead.

All I know is that the actual registration of Democrats and Republicans is far closer to what actually happens on Election Night. It makes sense though. People who are newly registered are more likely to vote than people who have been registered for 20 years.

That is what unskewedpolls.com is all about. It takes the registration and applies it to a poll. It shows that Romney is beating the brains out of Obama.

Credulity, perhaps, but not credibility. They make their money two ways. First, they produce “Internal” polls that are detailed and accurate. What they produce as an Internal poll is orders of magnitude different from what is produced for public consumption. The breakdowns are infinitesimally detailed and accurate. Contingencies and alternative scenarios are discussed. The entities [campaigns and news organizations] who buy them want truth, but they do not allow that truth out to the public. That costs a bunch of money, and they pay it.

Second, pollsters produce polls for publication. They use the Golden Rule. “He who has the Gold, makes the Rules.” They know what the paying client [once again, campaigns and news organizations] wants as a result; and they will bias the results in any way necessary to give the client what they want. So long as both the public and private markets are getting what they are paying for, “credibility” means diddley-squat. In fact, the polling companies get “credibility” through their association with the media gatekeepers. As long as for the last poll before the election they are close enough to have an excuse, it is all good for them.

The reader/listener/watcher is not the Market. He/She is the product being manipulated into [or in the current election, out of] the voting booth. If you are in a floating craps game [a passable description of the American political system as it has evolved and stands now] and look around and cannot figure out who the ‘mark’ is …… it’s you.

I thought I’d explain why this article is so absurd. This will not please libfreeordie as seen by his spamming of Nate Silver, someone who knows nothing about polls and only picks the ones he wants and was equally wrong during 2008 until the last two weeks.

In that article, there is a link to another story from 2004 where they complain about the DRI split shifting and producing a difference of +8 towards Republicans. What they don’t tell you and what they are not considering is that the final split was BETWEEN those two polls. The average of which came close to the final results within a couple points.

Read that paragraph again, the actual DRI split of the polls went on both sides of the eventual actual results. That is NOT happening this time. There are NO polls even remotely skewed toward Republicans even though Rasmussen and Gallup show a much different picture and so does 2010.

Also, if it were only 4-8 points, that’s chump change compared to what’s going on now. We’re talking 8-13 point skew toward Dems. That’s an impossibility.

FYI, one of the people I talked to *was* Nate Silver, but we elected not to use that for various reasons. But we heard from both sides. They’re a pretty close-mouthed group by and large, but it’s clear that most of them include a variety of fudge factors into their final numbers based on “experience and historical data” but I couldn’t make much of that, since they don’t share the details of their predictive models. Really, I was left with more questions than answers by the time I was done.

Look, when they first poll scientifically how many messiah supporters of 08 intend to even vote, then you have a more accurate sample upon which you place your figures. Obama, the hope and change messiah that will calm the seas is not the failed arrogant hater of the constitution, religious conscience,and life in or just out of the womb, who is running today.

In the meantime -these folks know their polling data based on 08 turn out is worthless as a scientific predictor, so it then becomes a convenient propaganda tool.

can we stop these poll denial stuff. its getting beyond ridiculous! yes, the polls show obama winning and its especially serious in swing states. romney has to do something to turn around the ship or defeat will come and hot gas will murder a lot more chocolate bunnies.
romney! please save the bunnies! you are our only hope!

I recently read Forrest McDonald’s “The American Presidency: An Intellectual History.” What he writes on p. 470 about polls is interesting:

The American people are, according to the polls, notoriously fickle, and they have short attention spans; they hold few opinions for very long and are concerned with few issues throughout an extended period. They also contradict themselves, as in 1959, eight years after the adoption of the Twenty-second Amendment–which limited presidents to two terms–when almost two-thirds of those polled said that they approved the amendment, yet 58 percent of the same people said they wanted to elect Eisenhower for a third term.

I think the polls are a bit less than totally accurate, but there is no reason to throw them all out.Romney should be ahead by a pretty good margin but he isn’t no matter how much we wish it. At best he is tied or slightly behind and that is worrisome…

Witchcraft or not, I don’t have a good feeling about where we stand at the moment. Romney’s run a lousy campaign, and the electorate has become stupid enough that they actually seem to be buying the President’s snake oil (not to mention all the goodies he’s parceled out in Ohio and elsewhere). Combined that with early voting (which I can’t see helping us any at the moment), and I have the same (bad) feeling that I had at this time in 2008 with McCain. I hope I’m wrong (fingers crossed) but my instinct is telling me I’m not.

Wasn’t Larry Sabato on one of the Sunday talk shows, basically saying that he is under the impression that some of the polls that have come out lately are a little fishy and that the D turnout in 2012 will be nowhere near what it was in 2008????

Also, at the height of hope and change the Dem turnout was 39%. So, how is it possible that polls showing a Dem turnout of 48% could be considered accurate???

can we stop these poll denial stuff. its getting beyond ridiculous! yes, the polls show obama winning and its especially serious in swing states. romney has to do something to turn around the ship or defeat will come and hot gas will murder a lot more chocolate bunnies.
romney! please save the bunnies! you are our only hope!

nathor on September 30, 2012 at 4:23 PM

So you believe that when the polls said Hillary was going to lose a state in 2008 by 20 points and then won it by 10+, that this was denial? She did this over and over. That in 2010, the polls were wrong. That they got it wrong with Walker’s recall was denial? That they do this over and over is denial? That skewing by 13 points is denial?

Don’t you get it? The pollsters will NEVER show overall polls going for Romney. It simply will not happen. What then? Should Republicans be perpetually depressed? That’s their plan. Your argument makes this a self-fulfilling prophecy and turns Republican against Republicans. It’s a beautiful scheme by the Dems. They get Republicans to get Romney to change course.

Statistics is like everything in science.
The general public doesn’t understand the mechanics of it & so they both get used for evil & rarely for the good.
And since people in the world mostly fall into the stupid category, reason does not play a part in any of this.

Read that paragraph again, the actual DRI split of the polls went on both sides of the eventual actual results. That is NOT happening this time. There are NO polls even remotely skewed toward Republicans even though Rasmussen and Gallup show a much different picture and so does 2010.

Also, if it were only 4-8 points, that’s chump change compared to what’s going on now. We’re talking 8-13 point skew toward Dems. That’s an impossibility.

MrX on September 30, 2012 at 4:16 PM

You mainly use 2004 and 2010 years Republicans did well in your argument. You seem to assume this year will be similar.

I think the polls are a bit less than totally accurate, but there is no reason to throw them all out.Romney should be ahead by a pretty good margin but he isn’t no matter how much we wish it. At best he is tied or slightly behind and that is worrisome…

can we stop these poll denial stuff. its getting beyond ridiculous! yes, the polls show obama winning and its especially serious in swing states. romney has to do something to turn around the ship or defeat will come and hot gas will murder a lot more chocolate bunnies.
romney! please save the bunnies! you are our only hope!

nathor on September 30, 2012 at 4:23 PM

Romney and Obama are tied, with independents breaking for Romney. Romney is winning.

OT/

side story, I want to congratulate the romney team for avoiding foot in mouth moments for the last few days! good job!

nathor on September 30, 2012 at 4:27 PM

Romney’s campaign has been largely gaffe-free to this point.

For your own good, please stop making things up and join us in the real world. Otherwise, election day is going to present you with an existential crisis, and why have that?

You bring up a good point that unfortunately goes against your premise. The fact that so many people look at RCP means that pollsters gear their results toward advertising on that site. They use their poll numbers as advertising. This is something that was talked about in years past.

i know this link has been going around a couple of days…but i think this pretty much sums it up

The Romney campaign is not flailing and it is not guilty of missteps; what it is, is under constant attack by a massive leftist bloc composed of think-tanks, campaign operatives and the media who then use their own attacks to manufacture a narrative of incompetence. Each attack is then called a “gaffe” and used as evidence that the campaign is flailing.

This isn’t a new technique, what is the new is the complete and shameless integration of the media into the spin corps and attack poodle ranks of a political campaign so that there is hardly any difference between an anchorman, a reporter and a campaign spokesman. What is new is the level of intense coordination that allows one campaign total airtime and allows the other campaign a chance to pay for ads and be attacked all the rest of the time.

This has nothing to do with Romney, just as it had nothing to do with McCain; the same exact treatment would have been meted out to any human being who chose to run against Obama. The media’s treatment of Romney is as impersonally vicious as the behavior of students when faced with a new substitute teacher. It isn’t about Romney, it isn’t even about Obama anymore, it’s about power.

Romney and Obama in Dead Heat. So Why Does the Media Think Obama is Winning?
(Sep. 25, 2012)
****************

If you get your news from the mainstream media, you probably think that President Obama is clobbering former Governor Romney in the polls. Why is the media calling this race for Obama? Are journalists relying on polls that oversample Democrats? Find out on this Trifecta.
VIDEO:(9:03)
===========

For your own good, please stop making things up and join us in the real world. Otherwise, election day is going to present you with an existential crisis, and why have that?

Axe on September 30, 2012 at 4:32 PM

no gaffes? lol! I bet you 10000$ there were lots of gaffes! sorry, I my existencial crises was months ago when romney was sure to win and I realized how bad candidate he was and how his pandering of the right wing damaged his political credibility. now I just in bitter schadenfreude mode.

The polls are definitely lying. Gary Johnson is actually winning over both Mitt Romney and Barack Obama by a double-digit margin.

Hey, if Rombots can be utterly delusional and keep telling themselves that their guy is actually winning, people actually supporting the only real conservative in the race should be able to get in on that action too.

You may have noticed that I’m reading about the polls.
Can you discern the difference between watching the polls and worrying about them ??
Your insult hasn’t found a resting place.
I’ve also sworn off worrying about insulting posters.

Uuurgh! I’d like to contribute to this discussion, have worked many, many years in the market research industry and have an advanced degree in the subject. However I have to leave and won’t be able to come back until later tonight. I’ll check back later and see if anyone’s still around.

I’d say ‘alchemy’ because there’s definitely a fair amount of chemistry involved in propaganda. And by that I mean, the polls themselves – regardless of source or methodology – aren’t really the problem.

The real problem is the frequency in which the MSM bandwagons are loaded up and paraded about with those results. That much overload on the information highway tends to skew the view and piles the poop so much that all the electorate wants is another drink!

Bottom line: the only poll that truly counts is the one on election day.

I my existencial crises was months ago when romney was sure to win and I realized how bad candidate he was and how his pandering of the right wing damaged his political credibility. now I just in bitter schadenfreude mode.

It’s pretty simple, actually. Conservatives that are voting for Mitt, aren’t. They’re voting against Obama. Once they realize that Mitt won’t win, they won’t bother voting for him. So a 6 point deficit today becomes 10-20 in about three to four weeks when Indies break against Mitt and R’s start to lose heart…after the three debates that Mitt is all but guaranteed to go squish, come off as a insensitive rich guy (falling perfectly into the lib meme) and most importantly, showing the rest of world what guys like me have been saying all along – Mitt’s not a conservative and therefore has no true core values – other than those needed today to get elected.

Don’t forget to buy your popcorn before the first horror show on 10/3.

You bring up a good point that unfortunately goes against your premise. The fact that so many people look at RCP means that pollsters gear their results toward advertising on that site. They use their poll numbers as advertising. This is something that was talked about in years past.

MrX on September 30, 2012 at 4:33 PM

isent RCP a republican leaning site? I dont thing they would allow themselves to be easily manipulated.

There has always been criticism of polls, from both sides. Yet in terms of poll averages they did a decent job in 2008, 2010 and will do so in 2012.

lester on September 30, 2012 at 4:34 PM

This is completely untrue. Go back and see for yourself if polls were accurate this far out? We’re not days before the election. What if the pollsters update their polls to show a trend towards Mitt just before the election? What happens to your narrative? Every single year, the polls are knowingly skewed. This is not a fantasy or wishful thinking. There is proof of it happening. Look at the Walker recall. Look at 2008 two weeks out. Look at 2010. Look at what they did to Hillary during the primaries. What they did against Hillary was insane. They will do this against Mitt. It’s going to get a WHOLE lot worse before election day.

isent RCP a republican leaning site? I dont thing they would allow themselves to be easily manipulated.

nathor on September 30, 2012 at 4:48 PM

What in the world are you talking about? Manipulated how? The polls come in and they average it. All pollsters have to do is put out numbers that make them more popular with the people who buy their services.

no gaffes? lol! I bet you 10000$ there were lots of gaffes! sorry, I my existencial crises was months ago when romney was sure to win and I realized how bad candidate he was and how his pandering of the right wing damaged his political credibility. now I just in bitter schadenfreude mode.

nathor on September 30, 2012 at 4:40 PM

The Romney campaign has been largely gaffe-free to this point.

You’re dishonest.

My life is so much calmer now that I’ve sworn off worrying about polls.
Ahhhhh.

pambi on September 30, 2012 at 4:28 PM

I’m coming with you. Even with a chalk board out and arrows and smiley faces and everything — people keep saying the same things. I was wrong. It has nothing to do with needing a clarification. They just don’t care.

Get used to it, because come Nov. 7, there’s going to be a lot of finger-pointing towards phony, unprincipled conservatives and Establishment types who insisted that Mitt was the only solution.

newtopia on September 30, 2012 at 4:55 PM

come Nov 7th? There already is the finger pointing-ad nausea check in here any time of day. We have our resident posters that tell us 24/7.
I would even agree but he is the nom. Not by my doing.
Even if he wins the finger pointing and his every move will continue.

Get used to it, because come Nov. 7, there’s going to be a lot of finger-pointing towards phony, unprincipled conservatives and Establishment types who insisted that Mitt was the only solution.

newtopia on September 30, 2012 at 4:55 PM

All that voted for McCain = 47%.
We lose none.
All that stayed home mad at Bush or McCain.
We pick up?
All that voted for the first black president that don’t show up in 2012.
They lose?
All that voted for obambi but have waken up?
We gain?

That would be me. Though, nothing will stop me from voting-not even a cardiac arrest. Thx for your future analysis.
Sigh* like I havent read it here from others 100000 times before in the last month.

bazil9 on September 30, 2012 at 4:51 PM

hate is a powerful motivator, but a very poor persuader! and I really think this one of the main problems with conservatives today. I dont hate obama, he is for me just another lying politician with a gift for rhetoric.
and you have to be sure of a set of ideas if you want to have any chance of persuading independent voters, because hate alone will not suffice.

he has to offer something, because obama hatred alone will not suffice.

nathor on September 30, 2012 at 4:58 PM

Why do y’all refuse to listen to what he’s offering ??
It’s out there, and there’s plenty more to come. .
Perhaps you have no tolerance for anything he’s laid out because it’s not PRECISELY what you want him to be saying??? Too bad.
To say he’s offered /is offering nothing yet is purely untrue.

hate is a powerful motivator, but a very poor persuader! and I really think this one of the main problems with conservatives today. I dont hate obama, he is for me just another lying politician with a gift for rhetoric.
and you have to be sure of a set of ideas if you want to have any chance of persuading independent voters, because hate alone will not suffice.

One does not have to walk in hatred to vote AGAINST what the incumbent’s policies will do to the country if given another term. Dislike Mitt ? Fine. Dislike O’s plans even more than the alternative ?? That does not equal being motivated by hate, just doing our part to stem the tide of destruction anticipated.