Demystifying the Science and Art of Political Polling - By Mark Blumenthal

February 28, 2006

The Zogby Poll of Troops in Iraq

[3/1 - Note to regular readers: I have added additional thoughts at the end of this post].

MP has received email from several readers asking about a just released survey of troops currently serving in Iraq conducted by Zogby International and noted in Nicholas Kristof's column in the New York Times this morning.

According to the release, Zogby conducted the survey in collaboration with the Center for Peace and Global Studies at Le Moyne College. Those who click through to the Zogby summary will find the following methodology statement:

The survey included 944 military respondents interviewed at several undisclosed locations throughout Iraq. The names of the specific locations and specific personnel who conducted the survey are being withheld for security purposes. Surveys were conducted face-to-face using random sampling techniques. The margin of error for the survey, conducted Jan. 18 through Feb. 14, 2006, is 3.3 percentage points.

I wrote about another survey of troops in Iraq conducted last year by the Military Times newspapers (here and here) and know that polling active duty troops is no easy task. So I sent an email to John Zogby this morning asking if he could describe the survey a bit more in general terms if not in specifics.

Zogby returned my call this afternoon. While not exactly a fan of this site (to put it mildly) he was courteous enough to provide a more in-depth explanation of what his company did and why he is unwilling to disclose more publicly. Unfortunately, the ground rules for our conversation prevent me from sharing much of what he told me. But I did come away convinced that Zogby has good reason to withhold the details of how he was able to interview U.S. troops the way he did. More disclosure could put the interviewers' lives at risk.

Here is what I can say:

The Center for Peace and Global Studies paid Zogby to conduct the study but otherwise played no role in conducting interviews or gathering the data. [Correction: In our conversation Zogby indicated that an "anti-war" sponsor paid for the survey but played no role in conducting interviews or gathering the data. I wrongly assumed he meant the Center for Peace and Global Security, whose faculty according to a story in yesterday's Syracuse Post Standard, did help "develop and word the poll's questions." In fact, the funder was "a wealthy war opponent who [Zogby] would not name." Thanks to reader Bob for pointing out the Post Standard story. Apologies for the error. See this update for more details].

According to the procedure Zogby described, respondents were intercepted randomly (e.g. they were not self selected) at multiple locations throughout Iraq (e.g. not just in the so-called "Green
Zone") and interviewed using a paper questionnaire that they filled out with the assistance of an interviewer.

Zogby was willing to share the specific geographic locations where they collected data on the condition I not repeat them. I passed on the offer as my knowledge of Iraq and military operations there is cursory at best, but I have no doubt his offer was genuine.

Zogby provided Nick Kristof and others reporting on the poll full details about his methodology on an "off-the-record" basis.

So in short, I can tell you that Zogby found a creative solution to the difficult problem of polling troops in Iraq, but I promised to say no more than that. I asked Zogby what advice he would offer data consumers who find this all puzzling. In this case, he said, "you have to trust me."

PS: I neglected a hat tip to the reader who blogs at Fickle Minded and first emailed me about the Zogby poll.

PPS (3/1): Readers should not interpret my commentary above as either an endorsement or criticism of Zogby's methodology. However, in response to some of the very reasonable questions posed by readers in comments and email, I want to clarify a few points.

First, to be clear, the Center for Peace and Global Studies is in effect a "partisan" sponsor in that, according to Zogby, they opposes the war in Iraq. [Correction: As noted above, the survey had a partisan sponsor that opposes the war in Iraq. However, that sponsor was not the Center for Peace and Global Studies, which collaborated on the study but did not fund it. Apparently, the Center, despite its name, is a non-partisan academic institution that takes no formal position on the Iraq war. The confusion on this point was mine. Again, apologies for the error].

Second, while Zogby says his interviewers selected respondents randomly at various locations, he makes no claim of random selection with respect to the locations involved. I apologize for being so vague, but the most I can say is that the method Zogby used to gain access to those locations constrained his ability to make random selections.

So to evaluate this survey, one important question is whether the troop populations accessible at the locations Zogby selected are representative of all troops in Iraq. This issue is analogous to the question of whether precincts selected for an exit poll are representative of all precincts in a given state (although exit poll precinct selection is usually random). Zogby believes the locations involved provide a reasonably representative sampling and, as noted above, he offered to share the names of specific locations on an off-the-record basis. I declined largely because I lack the knowledge and resources to make an independent assessment. So for me, this question remains open.

Third, even if consumers of this data knew all that I know about how Zogby's interviewers "walked up to troops" (as commenter Karen puts it), they would still have questions about the impact of such an interaction might have on the kinds of troops most likely to agree to participate in the survey. Consider the exit poll example again. Even though exit pollsters have disclosed the procedures they use to train interviewers and select respondents, we still debate the effect of those procedures on the kinds of voters that choose to participate. Disclosure in that case cannot resolve all questions, but it at least enables an informed debate. Unfortunately, such discussion and debate is impossible in this case.

The survey procedures Zogby described to me involved compromises analogous to those used in surveys of Katrina victims conducted by the Washington Post/Kaiser/Harvard and CNN/USAToday/Gallup. A survey of troops in Iraq would be impossible without some sort of methodological compromise along these lines, and Zogby's approach may be the best available. Yet my "bottom line" on this survey remains uncertain. I hope that those with more military expertise will assess how well it represents the troops in Iraq in terms of the age, rank, branch of service and any other similar characteristics (and no, I did not ask Zogby whether he weighted or adjusted his results).

The issue of disclosure is more difficult. Note that for the Katrina surveys, news organizations clearly characterized the results in terms of the limitations of the survey design. For example, the Washington Postreported results from "evacuees living in shelters." USAToday reported on the opinions of "residents who sought help from the Red Cross." Both organizations spelled out the methods used to contact respondents so inquisitive consumers could reach their own conclusions about the value of the data. Yes, Zogby has good reason for withholding the details in this case, but the secrecy limits the ability of consumers to evaluate the data and of news organizations to report it.

The toughest question here is whether it is appropriate for news organizations to report on a survey with partisan sponsorship that requires readers and viewers to place an unusual degree of trust in unspecified methods and procedures. I will admit I do not have a good answer for that one -- different news organizations will apply different standards -- but big "grains of salt" are certainly in order.

PPPS: In addition to Democracy Project (which has track-back links below), see also the posts by The Officer's Club and Murdoc Online for a different perspectives on the Zogby poll.

Also a reader emails with a very good question that I probably should have put to Zogby:

What is the problem with releasing the demographics on the poll? Were it truly
representative, it would have a breakdown of service branches proportional to
the military in Iraq. Same with the breakdown for gender, age, etc. I can't
imagine any security risk in releasing the demographics, or motive for keeping
them hidden other than they would expose the lack of validity of the survey.

Neither the poll release nor (I'm told) the full report available for sale provides statistics on the demographic composition of the sample.

Comments

To believe Zogby is to be manipulated. The guy is a political hack. Do you really believe he just walked up to troops all over Iraq? What about the troops unwilling to talk to the pollster? This secretiveness is so juvenile and insulting that the poll should be put with the rest of the propaganda. I remember well Zogby telling us all those senate seats the dems were going to pick up in GA, TX, NC. I guess he forgot to check the landslides for Senators Chamblis, Cornyn, and Dole.He is still explaining his Kerry win.

Posted by: Karen | Feb 28, 2006 5:37:23 PM

Did Zogby say whether the sample was adjusted in any way?

Posted by: Karl | Feb 28, 2006 6:12:24 PM

So what is your bottom line? Without revealing anything he told you, could you at least tell us whether or not you consider the poll to be credible?

Posted by: Observer | Mar 1, 2006 1:55:28 AM

I don't know what to think of the survey at this point - thanks for giving me at least a little more background - but one line of your post really jumped out at me: "More disclosure could put the interviewers' lives at risk."

The *interviewers'* lives? From whom? I can't imagine that the insurgents will say "Ah, he's a pollster: kill him!" Or was this a reference to pollsters being detained and sent to Abu Ghraib?

Every post by imbeds I've read or heard contradicts this poll. What about the high rate of reenlistment by regulars in both the Army and the Marines stationed in Iraq. This poll in addition to requiring a large grain of salt also does not pass the smell test.

Posted by: Bill Dempsey | Mar 1, 2006 3:48:41 PM

If interviewers' lives are at risk, then it stands to reason that the interviewers' actions, if disclosed, must be one of the following: (a) exploitable by the terrorist insurgency -- i.e., make interviewers an easy mark for kidnapping; (b) aid and comfort to the enemy on a field of battle, making them de facto enemy combatants; or, (c) open belligerence warranting a deliberate US military response.

Posted by: craig | Mar 1, 2006 4:08:40 PM

A correction to your comments on the Zogby Poll, where you mentione that "The Center for Peace and Global Studies paid ...but otherwise played no role...." The Syracuse Post-Standard reports that Zogby says that the "Le Moyne faculty helped to develop and word the polls questions....", as well Barron Boyd, Director of the Center for Peace, is cited as having "helped Zogby develop the questions." Also, the Christian Science Monitor states that the project was funded by "an anonymous anti-war activist." There is also one statistic that is seldom mentioned that plays on what the troops may have acutally been saying, that is that "53% said the US should double the number of troops and bombing missions to control the insurgency."

Posted by: Bob | Mar 1, 2006 4:37:23 PM

To boil it down: **Zogby** says "trust me". Zogby. The mind boggles.

No. Not after his biased behavior and slanted polling from 2002 onward.

James, 56, is the most recognized political voice of the Arab American community, a key player in trying to reshape American policy in the Middle East. And John, 53, has made the Zogby name an internationally recognized trademark, the result of his success as a political and corporate pollster.
...

James is founder and president of the Arab American Institute, a Washington-based organization that promotes the political interests of Americans of Arab descent and works to better American understanding of Arab interests in the Middle East conflict.

The organization runs on a $1 million annual budget, about 30 percent of which comes from individual foreign donors.

John is an integral part of the work of the 17-year-old institute, even as he continues to expand the reach and services of his firm, Zogby International, from polling to public relations and telemarketing. From its base in Utica, it handles clients on five continents.

...

The turning point for James was the time he spent living with Palestinian refugees in 1971 as a graduate student on his way toward a doctorate in religion from Temple University. The experience of life in the Palestinian camps "has never left me," he said.

For John, the turning point was his anti-war activism. "I saw how some of the Jewish kids could be anti-war and pacifists and, at the same time, just relish the idea of kicking the hell out of the Arabs," he said.

Source: AAI in the News
-------------
Impartial polling? Yeah, right.

Posted by: John Steele | Mar 1, 2006 4:53:39 PM

As I wrote on my own blog:

I CAN'T SPEAK for anyone else, but I know that if a pollster came up to me while I was serving in uniform, and asked if I'd like to participate in an opinion poll, he'd get a two-word answer from me.

The second word would be "OFF."

...What I'd like to know, is what was the ratio of those refusing to take part in the poll to those agreeing to participate? Now that's a number from which I could draw some interesting analysis...

Even if we accept the numbers as being accurate, I tend to greet it with a great big "so what?" Not to say that the opinions of the troops are not important, but what, really, is the basis upon which we should accept the feelings of a random soldier/marine/airman/sailor concerning large-scale strategic and tactical matters? I hope I don't come off sounding all elitist, but the military isn't run as a democracy because it relies upon those on the groud making the ground-level decissions and those at 30,000ft making the 30,000ft-level decissions.

I have to agree with earlier comments -- my mind boggles at what kind of polling method would risk the _interviewers_ lives if it were revealed.

My first thought when the secrecy was mentioned was that Zogby had subverted military personnel into doing the interviewing, and he wanted to protect them against consequences (legal or career-wise). But I can imagine NO scenario where the interviewer's lives are at risk, unless he is so deranged to imagine the U.S. Military would execute them in secret...

You can't shed _any_ light on this?

Posted by: Dave | Mar 1, 2006 6:38:59 PM

I don't want to make too big a deal about this, but through inside techniques which I cannot disclose for obvious reasons, I can report that Zogby's staff of covert pollsters never polled ANYBODY in Iraq on this subject, but rather, filed their work while lounging by a hotel pool, sipping mint juleps in Niger. I know this sounds fantastical...but you're just going to have to trust me on this one. (By the way, my margin of error is 3.3)

Posted by: Brian | Mar 1, 2006 6:54:39 PM

What I'd like to know more about is this, and the answer would not endanger any troops (though it ought to embarrass the poll designers):

How were the questions phrased, and in what form were the answers given? In particular, the percentabe of respondents citing "revenge for Saddam's link to 9/11" as a reason for the invasion are much higher than could be expected if an open-ended question were asked. Therefore, it seems likely to have been an answer supplied on the questionnaire as one of multiple choices. The fact that another answer to the same question refers to "preemptive military action" makes me even surer this is so.

Supplying such an answer as one of the possible choices is crap, pure and simple. Troops who saw the invasion as revenge for the 9/11 attacks, despite knowing that there was no Iraqi involvement, would still choose that response, as coming closest to what they believe.

Yet the results make it look as if the troops surveyed are twice as brainwashed as U.S. civilians; the Big Lie of Saddam-Sept. 11 links is now "only" believed by less than a third of the public.

By the way...isn't Mr. Zogby of Arab background? Me thinks that Senators Clinton, Schumer, Corzine, and a host of other democrats, might like to know that an Arab is conducting such business...RIGHT WITHIN OUR BORDERS!!! Next thing you know, Arabs like Mr. Zogby will be OWNING AMERICAN POLLING COMPANIES!!! Please call your democrat senator on this one...

Posted by: Brian | Mar 1, 2006 7:05:48 PM

What I find so depressing about this unsubstantiated poll is that it will immeditately be taken up as Gospel by every anti-Administration politician and anti-war pundit for whom "80% of Iraqis hate our guts" has become a virtual mantra.

Zogby said, you just have to trust me. Ha Ha Ha. He was joking , right? Imagine someone Zogby really, really did not like politically. Say a poll done for white supremacists. Do ya think he would accept - "regarding the methodology - well I cannot tell you details, you just have to trust me." Ha Ha Ha. Sure. The methodology explaination is probably in the "media" vault right next to the Mohammed cartoons. Too bad the news is too sensitive for us to actually read - better just let the media/pollsters tell us what to think.

Posted by: Californio | Mar 1, 2006 8:06:23 PM

Sorry, all of your assessments are ignoring the other part of the poll that shows a vast majority of the troops think that Saddam was behind 9-11. These "singled-out" pollsters are not exactly getting this from the liberal MSM or what not, nor are they getting this from anywhere else as Bush himself has stated that Saddam was not related to the incident. Either way it is difficult to reconcile this part of the poll with your conspiracy theories.

Posted by: coomaraswamee | Mar 1, 2006 9:03:26 PM

In addition, we don't even need this poll, because Iraq is a Wilsonian Classically progressive wet dream...and conservatives used to be smart enough to avoid things that would never work out, hence being seen as better managers…but now they are ideologues stuck with their ideas like Hitler in the Bunker. I mean we didn't really need the study to find out that breast milk was better than formula for babies either.

Posted by: coomaraswamee | Mar 1, 2006 9:09:27 PM

In addition to the items you brought up, Mark, I would be interested in seeing the script of the survey-- what the precise wordings were, in what order, for all of the questions.

This same guy tells Kristoff (NYT) that his poll is "the first" conducted among U.S. troops in Iraq -- a contention that is patently and demonstrably untrue.

A far more comprehensive poll of nearly 2,000 soldiers serving there was published nearly two and a half YEARS ago. It used a sample design with sufficient dispersion-- 50 bases throughout the country -- that I'd be willing to bet is superior to that which Zogby's team employed (good luck getting him to be forthcoming on his own).

Go here for the first poll's details:

http://www.stripes.com/morale/groundtruth.pdf

BOTTOM LINE: if we can't trust Zogby on a claim as basic as the one he used to pitch his survey to Kristoff, why would we want to we trust anything else he says about his work?

Posted by: AlpedHuez | Mar 2, 2006 9:20:45 AM

Hmmm.

1. It's been brought up all over place on many blogs that 80% of the respondents want eliminate White Phosphorus and Napalm?

What color is the sky in Mr. Zogby's universe?

As an ex-Marine I love White Phosphorous. It's the principle method for infantrymen to make *smoke* as in *smoke grenades*! I also love Napalm and I can't quite figure out why anybody in uniform wouldn't like it.

IMHO I'd like to know what steps were taken to verify that these poll questions were actually *asked*. I'd also like some reassurance that the people asked these questions really were US military soldiers.

Frankly these poll responses look like they were put together by a bunch of college liberals who entered in such information that conformed to their personal biases and ideological viewpoints.

Only college liberals hate White Phosphorus, which was recently mislabled as "chemical weapons" by the liberal media. Only college liberals hate Napalm. Only college liberals, who would never see the inside of a recruiting station, would not know that every US infantryman goes into battle with at least one White Phosphorous grenade.

"Professional pollster Mark Blumenthal started Mystery Pollster to provide better interpretation of polling results and methodology... offers much needed help to Political Wire readers" - Political Wire