Demystifying the Science and Art of Political Polling - By Mark Blumenthal

September 29, 2004

MoveOn vs. Gallup

I have to admit I didn’t see the MoveOn.org newspaper ad attacking Gallup until very late last night. And when I did I was somewhat taken aback by its ferocity, but given the subject matter of this blog, a response is obviously in order.

Please remember as you read what follows that I too have questions about Gallup’s likely voter model. I doubt that their selection procedure – which includes probes of knowledge of polling place location, interest in the campaign and reports of past voting – is as appropriate in July and August as it may a week before the election. Gallup’s methods are also worthy of scrutiny given the disproportionate attention their surveys receive due to their prominence on CNN and in USAToday.

It is also obvious that Gallup’s likely voter results in August and early September had Bush farther ahead than other polls of likely voters conducted in the same period, although MoveOn picked the week when that difference was greatest. Gallup’s survey of likely voters conducted September 9-14 did show a 14 -point Bush lead on the three-way vote (54%-40%), and there were seven survey organizations that reported over a roughly comparable period (IBD/TIPP, CBS/New York Times, Pew, Harris, Democracy Corps, New Democratic Network/Penn Schoen Berland and ICR) that average to a 48%-43% Bush lead among likely voters on the three way vote. Note that two of these (Democracy Corps and NDN) were conducted by Democratic polling firms.

So yes, it is appropriate to question Gallup’s likely voter model, and likely voter models generally, but the tone and substance of the MoveOn advertisement just goes too far. If Ruy Teixeira dances on the line between spin and demagoguery in his daily calls for weighting by party, this attack by MoveOn leaps across it.

Whatever doubts I have about Gallup’s model, I don’t believe for a minute that they are intentionally "Gallup-ing to the Right,” as MoveOn loudly charges. They say Gallup "refuses to fix a longstanding problem with their likely voter methodology” and imply that weighting by party is the fix, never mind that most of the "other publicly available national likely voter polls” they tout to counter Gallup do no such thing. And then they slime everyone involved by implying that the company slants its surveys to suit the whims of George Gallup, Jr, an evangelical Christian no longer involved in Gallup’s political polling operation. According to the MoveOn ad, Gallup, Jr. called his polling work a "kind of a ministry” whose "most profound purpose…is to see how people are responding to God.”

Call me a partisan, but I always thought that this sort of guilt-by-association smear of someone based on an exercise of a constitutional right – no matter how disagreeable – was something that Liberals fought against.

Also, consider this observation by Richard C. Rockwell, a professor of Sociology at the University of Connecticut (and former director of the Roper Center) who posted the following on an listserv of survey researchers (quoted with permission):

The Gallup Organization has historically been among the most forthcoming of all polling organizations about their methods and about any problems that might arise from those particular methods. This goes back to the 1940s, when Gallup (i.e., George [Sr.]) was among the founders of AAPOR. Moreover, the Gallup Organization makes its data available for public inspection through the archives of the Roper Center for Public Opinion Research at the University of Connecticut - the raw data, not just the tabular reports. Anyone can check out these data for any evidence of error or bias. You can even re-weight the data as you wish. The Gallup archives go back to the 1930s. Given the public availability of their data on a site not owned or controlled by the Gallup Organization, it would be extraordinarily difficult for Gallup to mess with the data for political or any other reasons.

I may not always agree with the decisions of the methodologists at Gallup, but I have no doubt they are professionals who exercise their best objective judgment in an atmosphere of intellectual freedom. We should respect that.

Finally, let me take off my survey research hat for a moment and put on my Democratic Party hat. I have admired MoveOn’s efforts, but I have to ask, is it now so flush with cash that it can afford to buy a full page ad in the New York Times a few weeks before "the most important election of our lifetimes” attacking a polling company? Do swing or less-than-likely voters really care? Wouldn’t it be better to spend that money, say, making a case against George Bush or just turning out the vote?

Advice from one Democrat to another: let’s keep our eyes on the prize.

Comments

MoveOn is a political organization. Gallup's political polls are, of course, fodder for political commentary and political news.

Given Gallup's relationship with CNN, it's polls are picked up and spread by CNN as close to the gospel truth, with little regard for it's methodology or it's relationship with other polls.

That's not really Gallup's fault, but the media's fault. (CNN for trumpeting "their" poll, for instance).

Sure, in a perfect world, MoveOn could take out a full-page add attacking CNN for blantantly ignoring the huge disparity between Gallup and the bulk of the polling world, and that would be the end of it.

Neither politics nor people work that way, and I applaud MoveOn for being pragmatic enough to take the fight where it's needed.

Distasteful or not, MoveOn is playing to win. As are their counterparts on the right.

I would gently suggest that you are missing the point. MoveOn is devoted to getting Democrats elected. They've made the decision to do this by trying to influence both voters and the media. This is simply a way to influence the media narrative around polling. By buying a large flashy and vitrolic ad in the Times, they will force news outlets to at least mention the fact that there might be problems. To be honest, this is more than half the battle - if the narrative can be changed to be at least neutral towards Kerry, then the MoveOn ad has done its work. Is the ad over the top? Sure. Does it have elements of smear by association? yup. Does that bother me? yes, much as it bothers you.

Alas, in our current "news" climate, this is what works. We'll never see an actual conversation about the relative merits of various likely voters models, sampling, weighting, etc. by you or others like you. Even if we did see such a thing, no news organization would be able to distill it into the requisite 5 second explanation.

MoveOn has been moving in this direction during this election cycle - less content and more attack. This is why I no longer give money to them. But, I do appreciate the results...

Posted by: Scott Pauls | Sep 29, 2004 2:09:45 PM

I would also look to the response by both CNN and USA Today - in neither case anything approaching a thoughtful look at the issue. Whether we like it or not (and I don't), this election is caught in a perception trap.

When Jeff Greenfield (for example) can blandly state an out-and-out untruth (that Fox News broke the Bush DUI story in 2000), and not be held accountable; when CNN and USA Today can back Gallup without addressing the issues raised (whether the critics are right or not, there are issues); when there are massive differences among seemingly competent pollsters (go to http://anonymous.coward.free.fr/polls/pollbias.html and then all the way to the bottom of the page for a graph); when one might argue that CBS was shut down by forgery to avoid an airing of both the Niger story and - perhaps more important - any more prison pictures; then MoveOn bringing attention to this issue is not only right, but necessary. Most mass media outlets are avoiding any issues other than soap opera plotlines - we deserve better than that.

Posted by: Buzz Potamkin | Sep 29, 2004 3:11:19 PM

As an amateur sociologist, I can sit back and sort of scratch my head and wonder why Gallup's results are so out of line with other pollsters, and acknowledge that different organizations have different sampling methodologies and they're all run by competent people who are sincerely interested in the truth, and who knows, maybe Gallup has the best technique and all its competitors are off the mark, etc., etc.

As a Democrat, I am terrified that a steady flow of "Bush ahead by double digits" headlines will dissuade the soft Kerry supporters from voting (volunteering, donating), and the poll will become a self-fulfilling prophecy.

MoveOn is just working the refs[*], and as long as they do so without actually lying, I'm not going to quibble about their tone. The Republicans have spent twenty years learning to make editors cower before accusations of "liberal media bias," and it's about time Democrats learned to use the same techniques.

The ad only works if MoveOn's 'brand' is more trusted than Gallup's. In other words, the ad only works with Kerry's base......which is okay as far as it goes.

Seth's comment about the impact of polling is right on - dispirited supporters are not going to bang on doors the day before the election. So reaction is absolutely required - and I'm not sure what else they could do to attempt to discredit Gallup. But there are other polls out there that are equally devestating: Time Bush +6 and CBS Bush +9. Since they are probably doing the same weighting as Gallup I think MoveOn would have been wise to attack all three at once for identical reasons.

Posted by: Matthew Ryan | Sep 29, 2004 4:55:19 PM

What's funny about this is that it was conservatives last time who insisted that Gallup's polls were off, particularly the polls from October.

All the complaining in the world didn't make a difference to press reports of the polls. So, fair being fair, they wouldn't make any difference this time.

For those who recall, those on the right were insisting that people should pay attention only to the Rasmussen polls--those were straight down the middle, these partisans said, and always showed a Bush lead. (I think that's how one was supposed to know that they were straight.)

So much has changed, but not so much.

Posted by: Thomas | Sep 29, 2004 8:12:22 PM

What's funny about this is that it was conservatives last time who insisted that Gallup's polls were off, particularly the polls from October.

All the complaining in the world didn't make a difference to press reports of the polls. So, fair being fair, they wouldn't make any difference this time.

For those who recall, those on the right were insisting that people should pay attention only to the Rasmussen polls--those were straight down the middle, these partisans said, and always showed a Bush lead. (I think that's how one was supposed to know that they were straight.)

So much has changed, but not so much.

Posted by: Thomas | Sep 29, 2004 8:12:54 PM

Buzz Potamkin --

Why should either CNN or USA Today be expected to respond thoughtfully to a demagogic, sliming, guilt-by-association attack ad? If there are serious, reasonable points to be made that CNN and USA Today should answer, it's MoveOn's responsibility to bring them up in a serious, responsible manner.

If, however, MoveOn is just trying to rally the base by throwing out irresponsible attacks, CNN and USA Today have no obligation to treat their claims with any more respect than they have to answer the wild rantings of the John Birch Society, whether or not there's a point buried deep within the slime.

Posted by: WL | Sep 29, 2004 8:32:12 PM

Call me a partisan, but I always thought that this sort of guilt-by-association smear of someone based on an exercise of a constitutional right – no matter how disagreeable – was something that Liberals fought against.

There seem to be a lot of Democrats who aren't what used to be called Liberals.

Posted by: rosignol | Sep 29, 2004 8:40:49 PM

Hasn't Gallup basically come back into line with the other polling outfits?

They now have Bush up by 8, and so does Pew and CBS(!). The average over at RealClearPolitics is Bush by 5.9.

The Kerry campaign badly needs to fight off the perception that he is out of the race, not so much because of any bandwagon effect (which is a myth in my estimation) but because the media drumbeat on stories about how his campaign is failing and how it failed would drown out any hope he would have of getting his own message out.

The fact that Moveon exists and can make the attack rather than Kerry makes it all the better.

The fact that it involves the type of smear you said you thought liberals fought against might cause you to open your eyes a bit. Your side is not as pristine as you would like, if it ever was. People like you, on one side, and like me on the other, have to speak out loudly and forcefully against our own side when they do crap like this. It is the only way we can pressure the mud out of the system.

I keep on waiting for Move On to Move On. Until they quit the mud slinging, I think that the opposite of their commercials are true.

Posted by: David R. Block | Sep 29, 2004 10:16:46 PM

The Gallop organization will not doubt survive the "fit of stupidity" on the part of MoveOn.org. No Doubt, MoveOn.org will too.

Posted by: J_Crater | Sep 29, 2004 10:33:38 PM

I find it hard to put any faith in any of the polls at this time. There are predictions that this election will have record turnout from the deceased. Since the poll takers refuse to hold a seance, they just can't predict how the vote will go.

Posted by: burt | Sep 29, 2004 10:38:06 PM

Who gives a rat's ass what that bunch of homos think? Move on up an ass.

Posted by: Fentress | Sep 29, 2004 10:42:04 PM

I think MoveOn.org should concentrate on
Sen. Kerry's most important issue.

What he should wear to President Bush'es
re-inauguration late January 2005.

Anything else is really just a waste
of MoveOn.org's contributer's money.

Posted by: pragmatist | Sep 29, 2004 11:14:56 PM

Why in the world would Moveon.org spend that kind of money advertising in the NYT where they are complaining to the faithful? Wouldn't it have made more sense to split up that money and buy ads in several different major newspapers in SWING states such as Florida and Ohio. So what if it was for national exposure because at the national level the message is somewhat diluted and wasted on committed states and voters. Oh well, I guess Moveon.org can spend George Soros' money any way it sees fit. Democrats just crack me up.

Posted by: Harry in Atlanta | Sep 30, 2004 12:19:46 AM

As far as money well spent, by placing the ad in the NYT, more people are talking about it. I don't know if it would have gotten that type of buzz in a large number of regional papers. I think that Gallup does have some valid points on party affiliation. I am not registered with any one party, but I'll be rooting for a Republican victory, as will my wife (also unregistered). My parents, also unregistered, are voting for Kerry. In this election, I'm leaning more Republican, and would probably identify as such to a pollster, even though I'm not registered.
Perhaps the question should be changed to "Which political party are you currently registered with?"

The interesting thing is this...If MoveOn was a site anyone visited they need not have taken out an ad at the New York Times but need merely have posted at their site instead. So they have confessed their own irrelevance in the most public way possible. Sucks to be them. ^_~

Posted by: The Towering Barbarian | Sep 30, 2004 3:02:26 AM

I actually have some polling experience from the 2000 election with the head of CNN's polling (who is directly involved in both the Gallup/CNN/USA Today and Harris/TIME/CNN polls). If there were bias introduced by a survey he was affiliated with, I suspect it would go to the left, and not toward Bush.

However, when I worked with him during the 2000 election, he made every attempt to be an honest public opinion researcher, and I would be shocked if he or any poll he was associated with intentionally introduced any bias. For that reason, I believe that MoveOn.org is almost certainly wrong.

I suspect that what probably happened with that poll is that it was just outside the confidence interval. When polls give a margin of error, the margin of error is with 95% confidence (generally). That means that one out of every twenty polls is outside the margin of error, if the general population were to take the same survey. When one considers this, it is not at all surprising that some polls come out with results that are out of line of other polls. Given the number of polls that have been done on the national Presidential election in 2004, it is surprising that more outlier results aren't noticed.

Weighting for party ID doesn't solve this problem in the least. It is a result of the underlying problems of RDD polling.

Ultimately, this is just another example in the laundry list of examples of political and media commentators acting on polling data without understanding polling methodology and statistics.

"Professional pollster Mark Blumenthal started Mystery Pollster to provide better interpretation of polling results and methodology... offers much needed help to Political Wire readers" - Political Wire