Demystifying the Science and Art of Political Polling - By Mark Blumenthal

December 02, 2005

The RT Strategies Iraq Poll

Yesterday's front-page analysis of President Bush's Iraq speech earlier this week by the Washington Post's Peter Baker included a reference to some recent survey data:

Administration officials believe much of the public is still eager for victory and open to persuasion if the president can make the case that he has made progress. They took heart in a survey last week by RT Strategies, a bipartisan polling firm, that found that 49 percent of Americans favor bringing troops home when only "specific goals and objectives" are met, 30 percent want a fixed timetable for pulling out and 16 percent support immediate withdrawal. The middle 30 percent, they figure, is the real political battleground.

In an item in last Sunday's "Politics" column, the Post's Chris Cillizza called the poll "one of the few pieces of supportive news the administration has had lately on Iraq." He led with the finding of another question showing that 70% of adults "said that criticism of the war by Democratic senators hurts troop morale." Cillizza also looked at the poll in a blog item (that included links to results, crosstabs and an RT Strategies press release).

Not surprisingly, these findings have already been the subject of much debate on Cillizza's blog and elsewhere, partly because of the perception that they contradict the findings of other polls. I do not believe they do. Instead they help underscore that when it comes to prospective policy on Iraq, polls are inconsistent consensus is elusive. This post (which continues after the jump due to its length) reviews the nuts and bolts of the RT Strategies Poll and how its results fit into the bigger picture regarding public opinion on the Iraq War.

What is the RT Strategies "Omnibus" Poll?

First, let's review what we know about this poll and how it was conducted. Last week, I linked to a new national survey, a "national omnibus poll" conducted by a new bi-partisan research company named RT Strategies. The company is a partnership of Democrat Thomas Riehle and Republican Lance Tarrance. Riehle worked most recently as head of IPSOS Public Affairs, where he helped create the AP/IPSOS survey. Earlier in his career he worked variously for Democratic pollsters Peter Hart and Pat Caddell. Tarrance founded and ran the well-known Republican firm that bears his name (originally Tarrance and Associates, now the Tarrance Group) although he is no longer affiliated with that company.

[Interest declared: Riehle is a personal friend and an occasional commenter on this site].

Their first survey -- the one that is getting all the attention -- is part of something called an "Omnibus Poll." Omnibus in this context means that RT Strategies conducts one poll, but allows multiple paying clients to buy questions. Here is the way the National Journal's Gregg Sangillo described it ("Consulting Game," 11/19/05):

The twist is that in any given week, any outside groups, such as political party organizations or lobbying firms, can ask RT Strategies to pose additional questions as part of the poll. Riehle says that traditionally, if an organization wanted a poll question answered, it would have to "call up a company and do a whole poll. Well, now you're talking real money. We're saying, we have a vehicle, it's going to be going out into the field every week, [so] just ask the questions you need to ask."

MP readers may recall that their first survey also included questions written and analyzed in partnership with Charlie Cook and his Cook Political Report. However, despite some initial confusion to the contrary, Cook had nothing to do with the questions on Iraq. They were not part of the Cook section of the poll. "I did not even know they even existed," Cook told me via email, "until after the survey was completed." [For those interested in Cooks questions, his analysis, the top-line results, crosstabs and a press release from RT Strategies are posted on Cook's website].

Such are the pitfalls of omnibus surveys. A client can buy a question or two without having to bear the costs of fielding a complete survey. The disadvantage is that others questions may precede it that a client knows nothing about (except for those with the clout to insist that their questions come first).

Omnibus polls are not new. The AP/IPSOS poll (which Riehle helped create) is part of a regular omnibus poll in which both AP and other paying clients ask questions. According to the folks at AP, their questions always come first. The "Battleground Poll" conducted by the bipartisan team of Republican Ed Goeas (of Lance Tarrance's old company, the Tarrance Group) and Democrat Celinda Lake (of Lake, Snell, Perry, Mermin) is also an omnibus survey that includes questions fielded for private clients of the respective firms.

Generally, commercial omnibus surveys include questions on a wide variety of different topics, which generally cover consumer products and services rather than political issues. However, it is always possible that a question asked early in an omnibus survey might have some biasing effect on a question asked later. Omnibus poll clients put their trust in the pollster to avoid ordering questions in ways that would create an obvious bias.

To be fair to the omnibus pollsters, I should point out that surveys conducted by the major national news media outlets also typically ask about a wide variety of different issues in the course of a single survey. They too must make subjective judgment calls about how to order the questions and whether that order may unintentionally influence the results. While academic survey methodologists have studied "order effects" using split sample experiments, their findings provide no more than a rough guide to survey practitioners. Public and commercial pollsters rarely have the time or budget for experimental research. Thus, we rely mostly on experience and judgment in determining how to order questions and, in tracking surveys, whether inserting new questions or cutting old ones will create unexpected "order effects."

Who Was the Client for the Iraq Questions?

That was the question I wondered about when I first saw the results. According to Riehle, there was no client. "Lance and I wrote them," he told me when I called, adding that they wanted to include some newsworthy questions to help generate buzz for their new survey. In a comment left on Cillizza's blog, Riehle also added the following:

Lance and I intended to touch on the most emotional topics in the news in our choice of questions to ask (see, for example, the intense and often emotional debate in the House these past few days!), and we hope the questions themselves are pretty straightforward in how they measure the division of public opinion on those particular highly-charged questions.

Which brings us to the buried yet nonetheless critical question...

What Should We Make of RT Strategy's Iraq Questions?

In his Sunday item, the Post's Cillizza implied that these results contradict other polls that show "significant majorities believe it was a mistake to go to war, as well as rising sentiment that Bush misled Americans in making the case for it." Is that a reasonable supposition?

To consider the questions we need to step back for a moment and think about public opinion more generally. Most Americans know that George W. Bush is our president, that we are at war in Iraq, and that most elected officials are either Democrats or Republicans. Thus, on these topics, most Americans possess general opinions. So when pollsters ask questions about how well the President is doing his job, whether the war in Iraq was a good or bad idea, or the general strengths and weaknesses of the political parties, pollsters can tap real pre-existing opinions. Questions on these general topics tend to be consistent across polls, even with slight variations in question wording.

However, most Americans have far more limited knowledge of more specific aspects of public policy and the day-to-day debate that captivates policy makers, political junkies and most MP readers. In a poll released Wednesday night, for example, Gallup found that only 10% of Americans reported seeing Bush's speech on Wednesday while 66% had heard nothing about it (subscription-only Gallup release here).

When pollsters move beyond general ratings to more specific questions about policy - as we do in almost every public political poll - we move to shakier ground. Here Americans often lack preexisting attitudes, yet most will work to answer our questions, often forming opinions on the spot based on the text of the question. When that happens, responses can be very erratic and contradictory across polls. Very small variations in wording, the number of answer choices offered or the order of the questions can result in big and often surprising differences in the results.

With that in mind, consider the three RT Strategies questions:

Thinking about the war in Iraq, when Democratic Senators criticize the President's policy on the war in Iraq, do you believe it HELPS the morale of our troops in Iraq or HURTS the morale of our troops in Iraq? (IF HELPS/HURTS, ASK:) And do you believe it (HURTS/HELPS) morale A LOT or just SOME
44% hurts a lot26% hurts some6% helps some7% helps a lot17% not sure

When Democrats criticize the President's policy in Iraq, do you believe they are (ROTATE) Criticizing the President's policy because they believe their criticisms will help the United States' efforts in Iraq, OR, Criticizing the President's policy to gain a partisan political advantage?
31% believe will help51% to gain advantage6% some of both (volunteered)6% neither (volunteered)7% not sure

And thinking about the future of our policies in Iraq, do you believe the U.S. military should.... (ROTATE FIRST TWO, ALWAYS ASK "Set a fixed timetable" last) Withdraw our troops immediately, regardless of the impact OR Withdraw our troops as the Iraqi government and military meet specific goals and objectives OR Set a fixed publicly available timetable for withdrawal.
16% withdraw immediately
49% withdraw when goals met
30% set fixed timetable3% none (volunteered)2% not sure

A few reactions: First, all three of these questions fall into that second category of issues about which many Americans appear to lack preexisting attitudes. Non-attitudes are most evident on the morale question (something that Armando at DailyKos picked up on). The telltale clue is that 17% were completely unable to answer the question, a sure sign that many more came up with an answer on the spot. The fact that nearly a third chose one of the softer "some" categories (26% hurts morale "some," 6% helps "some") is consistent with that argument. Also consider the respondent who believes such criticism neither hurts nor helps troop morale, but does not realize that "neither" is an o[tion. Odds are good they will end up in the "hurts a little" category.

On the partisan advantage question, nearly one in five respondents (19%) could not choose between the two offered answer categories. Finally, for reasons that I'll discuss below, I'd argue that the large number of respondents in the middle category of the future policy question (49%) suggests that it was an attractive choice for those respondents who were simply not sure how to answer.

Now MP is not averse to survey questions that offer new information and push respondents a bit to see where they might stand in debates they have not followed closely. And in this case, the results of the RT "morale" and "criticism" questions are more or less consistent with the similar questions asked elsewhere. For example, a Fox News poll in early November found that 58% of Americans agree that those "who describe U.S. military action in Iraq as a mistake" are "hurting U.S. troops." Only 16% believed they were "helping." The rest had mixed opinions (9%), believed the criticism had no effect (9%) or could not answer the question (8%).

It is also worth noting that Americans tend to dismiss much of the debate in Washington as attempts to gain "partisan advantage," so the results of the RT question are not particularly surprising. For example, back in September (9/8-11), Gallup asked about politics in the context of Hurricane Katrina:

"Do you think Democrats who criticize the way the Bush Administration has handled the hurricane response mainly want to find out what went wrong, or mainly want to use the issue for political advantage?" 36 find out, 60% use for advantage, 4% unsure

Seven years ago, ABC News and the Washington Postasked a similar question about the impeachment of President Clinton with nearly identical overall results:

"Do you think the House voted to impeach Clinton on the basis of the facts of the case, or on the basis of partisan politics?" 36% facts of the case, 61% Partisan politics, 3% no opinion

Questions that push respondents to consider questions for which they do not have pre-existing opinions do have a role in opinion research (one that should not be labeled as a fraudulent "push poll" -- but that's another subject for another day). However, in those instances pollsters need to take care to provide respondents with new information in a way that does not bias subsequent questions. For that reason, I am a bit surprised that RT Strategies asked two questions that mirrored the Bush administration talking points just before asking respondents their preference about prospective Iraq policy. Would the responses to the third question have been different if they followed a question about say, whether Bush "intentionally misled the American people about the presence of weapons?" We will never know, but it certainly seems possible that they would.

To be fair, Gallup asked a very similar question a few weeks ago (11/11-13) with similar results:

"Here are four different plans the U.S. could follow in dealing with the war in Iraq. Which ONE do you prefer? Withdraw all troops from Iraq immediately. Withdraw all troops by November 2006 -- that is, in 12 months' time. Withdraw troops, but take as many years to do this as are needed to turn control over to the Iraqis. OR, Send more troops to Iraq." 19% withdraw now, 33% withdraw within 12 months, 38% take as long as needed, 7% send more troops, 3% unsure.

Note that Gallup showed 19% ready to withdraw immediately; RT Strategies show 16%. Gallup shows 52% supporting withdrawal either immediately or within 12 months, RT shows 46% support withdrawal either immediately or on a fixed timetable.

Having said this, I want to caution readers against taking these these prospective policy questions at face value. I also tend to agree with those who argue that the questions on the RT poll are, in essense, the wrong questions, that other measures give a better sense of true, pre-existing opinions on the Iraq War. This is not necessarily a criticism of Riehle and Tarrance, merely a caution that focusing on these three questions alone can give a misleading impression. For example, review the questions asked since Labor Day as posted by the Polling Report and you will find some highly consistent results:

Approval of Bush's handling of the war in Iraq varied between 32% and 36%, with disapproval between 62% and 65%, as measured by six different pollsters.

Differently worded questions about the worthiness of the war (asked by Gallup, CBS, ABC/Washington Post and NBC/Wall Street Journal) found between 31% and 40% that found the war worth the cost and between 52% and 60% that said it was not.

Differently worded questions about whether the decision to go to war was right or wrong (asked by Gallup, CBS and the Pew Research Center), found 42% to 45% who say the US made the right decision in going to war, between 50% and 54% who say we made the wrong decision or should have stayed out.

However, look at the range of questions about prospective policy and the results are all over map. Here is a sampling (full details at the Polling Report):

CNN/USA Today/Gallup (11/30): "If you had to choose, which do you think is the better approach for deciding when the U.S. should withdraw its troops from Iraq: to withdraw U.S. troops only when certain goals are met, or to withdraw U.S. troops by a specific date and stick to that time-table, regardless of conditions in Iraq at that time?"
59% when goals are met, 35% by a specific date, 6% unsure

FOX News/Opinion Dynamics (11/ 29-30): "Do you think there should be a publicly announced timetable for withdrawing U.S. troops from Iraq?" 47% should, 41% should not, 12% unsure

Harris (11/8-13): "Do you favor keeping a large number of U.S. troops in Iraq until there is a stable government there OR bringing most of our troops home in the next year?"
35% wait for stable government, 63% bring home next year, 3% unsure

FOX News/Opinion Dynamics (11/8-9): "What do you want U.S. troops in Iraq to do? Do you want them to leave Iraq and come home now or do you want them to stay in Iraq and finish the job?"
36% leave now, 55% finish the job, 9% unsure

NBC News/Wall Street Journal(11/4-7): "Do you think that the United States should maintain its current troop level in Iraq to help secure peace and stability, or should the United States reduce its number of troops now that Iraq has adopted a constitution?"
36% maintain level, 57% reduce number, 4% both depends, 4% unsure

ABC News/Washington Post (10/30-11/2): "Do you think the United States should keep its military forces in Iraq until civil order is restored there, even if that means continued U.S. military casualties; or do you think the United States should withdraw its military forces from Iraq in order to avoid further U.S. military casualties, even if that means civil order is not restored there?"
52% keep forces in, 44% withdraw forces, 4% unsure

CBS News (10/30-11-1): "Should the United States troops stay in Iraq as long as it takes to make sure Iraq is a stable democracy, even if it takes a long time, or should U.S. troops leave Iraq as soon as possible, even if Iraq is not completely stable?" 43% stay as long as it takes, 50% leave ASAP, 7% unsure

Pew Research Center (10/6-10): "Do you think the U.S. should keep military troops in Iraq until the situation has stabilized, or do you think the U.S. should bring its troops home as soon as possible?" 47% keep troops, 48% bring home ASAP, 5% unsure

So there we have it: A consistent majority of at least 60% of Americans now disapproves of President Bush's performance on the Iraq war and believes it not worth the cost. A smaller majority now says that the war was a mistake. The consistency of the results suggests these are real attitudes, not opinions formed on the spot in the response to the language of the question.

However, when pollsters ask what we should do next in Iraq, results are highly inconsistent. Support for leaving sooner varies anywhere from 35% to 63% on the questions listed above. Support for staying the course (in one form or another) varies from 36% to 59%. Ask a question with three or more options (as RT Strategies and Gallup did above) and, not surprisingly, at least a third of Americans opts for the middle category. When it comes to prospective policy, Americans - like their leaders - are divided and collectively not quite sure what to do next.

Under the heading: Fighting A War With Both Hands Tied Behind Your Back, would someone please poll al Q on whether U.S. criticism of the U. S. mission in Iraq helps al Q's morale.

Posted by: Terry Gain | Dec 2, 2005 7:47:54 PM

There is a very strong implication from all these floating numbers. They show a large group of Americans unsure about what to do in Iraq, and suggest that the battleground in politics is for that large middle that could be convinced that leaving now or in the near future is right, but are equally likely to be swayed by arguments saying that the US can accomplish something by remaining in Iraq.

In all the surveys, there are about one third of Americans who always take the “dovish” position, [Minimum size of Dove segment -- in CNN/USAT Poll: 35% leave by a specified date, or Fox: 36% leave now] and a roughly equal third that take the more “hawkish” position [Minium size of Hawkish segment -- in Harris 35% wait for stable govt, in NBC 36% say US should maintain level of troops]. That leaves roughly a third that moves between the Dovish or Hawkish position based on the wording of the question.

It is that middle third that is “up for grabs” and will likely make the difference in the coming months, particularly in the election

So, I would say that the Post is right to report that Administration officials believe much of the public is still eager for victory and open to persuasion if the president can make the case that he has made progress. But equally correct are the Democratic officials who believe they can make the case that the US should figure out an exit strategy that is based on the belief that the situation will not get better.

Posted by: Arnon Mishkin | Dec 2, 2005 8:57:55 PM

In case there was any doubt, we can conclude from these polls that there's a good reason these things aren't decided by plebiscite.

To their credit, CNN did note in its coverage that Gallup found that only 10% of Americans reported seeing Bush's speech on Wednesday while 66% had heard nothing about it. Always nice when such is not hidden behind a subscription wall.

And since it seems that I cannot embed the link in text, here's the URL:

http://www.cnn.com/2005/POLITICS/11/30/iraq.poll/

Posted by: Karl | Dec 2, 2005 11:04:33 PM

Thanks for all that information. I too saw the WaPo piece by Cillizza and no amount of search gave me the info you reveal today. However, I did see Bush supporters seize on the RT poll as "proof" of the Bush side of things. The way we look at the polls on the war and Iraq in many ways mirrors the way we look at the war itself. Whether a partisan can look at anything without a bias may be the story of this whole tale. Oy.

On the other hand, when really pressed to think about it, i.e. they didn't have an opinion beforehand, they decide it does hurt morale.

Maybe it's just that simple. Some segment doesn't think about it, but after a moment's reflection they decide that, yeah, if someone was saying those things, it would hurt morale.

While it's important that pollsters want to 'capture' the moment and analyze it, it's also usefull to see how people react even to 'leading' questions.

Posted by: Aaron | Dec 3, 2005 9:54:05 AM

Nice analysis. Unfortunately even a perfectly written poll has one main flaw. We derive much peace in our democracy from our ability to compartmentalize our thinking in order to get along. I noticed in looking into polls on abortion the other day that 70% of Americans want RoevWade to remain but 70% want greater restrictions on abortion. That doesn't seem to make any sense until you realize that if people don't have any direct consequences to their opinions the will say just about anything - no matter how contradictory. Any politician who bases policy decisions on polls ignores the fact that their actions do have consequences and that is what makes them decisions and not just opinions. The least workable system I can think of is one that is poll driven.

Posted by: D Boyd | Dec 3, 2005 10:31:44 AM

The most horrifying thing to me, and the most inescapable conclusion from these results, if they are true, is that a majority of this country no longer believes that freedom is worth fighting for; that America is worth fighting for; that the Western way of life is worth fighting for. It will take much more horrific attacks than we have suffered to get their brains working, if even that is possible. With widespread opinions such as you reveal above, more horrific attacks are absolutely inevitable.

Posted by: Peg C. | Dec 3, 2005 11:41:40 AM

Fascinating.

Thank you for making this enlightening analysis available - free of charge, no less!

Posted by: Knemon | Dec 3, 2005 6:12:19 PM

The most revealing statistic was from the Harris poll when asked "Do you favor keeping a large number of U.S. troops in Iraq until there is a stable government there OR bringing most of our troops home in the next year?" 35% wait for stable government, 63% bring home next year, 3% unsure"

This explains EVERYTHING you need to know. The public wants a favorable outcome in Iraq. Thus, the initial reaction to a question "should we pull out or stay until we achieve our goals?" will always get a majority for "stay until we achieve our goals." But, when they are told of the consequences of that decision: i.e. that "large numbers" of ground troops will have to stay in Iraq until 2007, suddenly that option appears far less appealing.

This shows very soft support for continuing the war. The public doesn't know what to do, but doesn't support keeping large numbers of troops in Iraq for an extended period of time. This severely limits the Presidents' options. He can only posture and talk about withdraw troops "depending on conditions." But, what if the conditions aren't achieved?

If he actually does start to withdraw, not only will he receive criticism for failure to achieve his goals, but also will accelerate that perceived failure from a declining ability to control events on the ground in Iraq.

Posted by: Cugel | Dec 5, 2005 11:17:00 AM

I am a Sunni Triangle Combat veteran and am ashamed to even have to ask folks to please have faith in us an let us do our job. The left wing of our nation salivates at the thought of us cutting and running. They have always ran when the going got tough. I saw a bumper sticker yesterday with an American flag on it with the text saying "These colors don't run" I wish this were true, I guess it should have said "well, at least the red and the white dont run" Please dont make us look like failures! We can finish the mission if you just give us time.

One combat medic
Balad Iraq

Posted by: Phillip Rolen | Dec 5, 2005 5:57:45 PM

How much longer do you need to find the WMD, Phillip?

Posted by: Argonaut | Dec 5, 2005 11:18:49 PM

Well Argonaut think about this...imagine loosing your car keys in a child's sandbox...how long would it take you to find them? Thank you Phillip, for protecting our country. RIVERCREST DEBATE RULES!!!

Posted by: jason thomas | Dec 17, 2005 12:18:17 AM

Hi.

Question on impeachment polls - does comparing Qs now versus Qs used historically have value?

I’ve just linked to your site and am unabe so far to find any comparisons between wording used currently and that used in poll questions asked of Americans in our most recent historical experience of impeachment polling. [maye I’m not looking in the right places. If so, sorry!]

In any case, could you comment on these two excerpts from [1] old WaPo poll and [2] new Zogby polls? excerpts were put forward on the Media Matters site, where they are comparing wording between the last set of polls over impeachment and current polls. Thanks.
“[A] January 1998 Post poll conducted just days after the first revelations of Clinton's relationship with Monica Lewinsky asked the following questions: ‘If this affair did happen and if Clinton did not resign, is this something for which Clinton should be impeached, or not?’”

Here is the URL where I found those comparisons:
http://mediamatters.org/items/200512100004#20051220

It looks as though Media Matters makes a direct comparison between that wording methodology -- an equation, if A turns out to be what happened, should B occur, and finds it essentially equivalent to " ‘If the president did not tell the truth about the Iraq war, should he be impeached?’"

I realize that there were no bloggers around to criticize the Post’s wording in 1998, if in retrospect the wording and/or methodology might seem to have faults, but the methodology does seem to be the same between Zogby’s current poll and WaPo’s 1998 poll – and if I am not off-base in finding an equivalent level of reasonableness [or, some might think, bias] between the two, would it not seem fair to bring the Wash Post’s poll wording from that era into the debate?

"Professional pollster Mark Blumenthal started Mystery Pollster to provide better interpretation of polling results and methodology... offers much needed help to Political Wire readers" - Political Wire