May 6, 2007 - May 12, 2007

A new AP/Ipsos poll taken 5/7-9/07 is out today with approval at 35% and disapproval at 61%. With this addition my approval trend estimate stands at 34.8%.

There is not much to discuss here. The approval trend estimate has remained in the 33-35% range since January, and is still there. Once or twice we've bumped into those limits, only to promptly return to the range. At the moment we see a slight upward trend, but earlier this week saw a slight DOWNward trend. So I'd say the evidence so far gives no reason to think approval is doing anything other than remaining flat, with poll to poll variation bumping the trend up or down briefly.

Everything looks pretty normal in the diagnostics below, so I'll not add any more commentary. See the earlier posts on approval this week for more analysis, all of which holds true in light of today's AP poll.

A new InsiderAdvantage/Majority Opinion automated survey of registered voters in South Carolina (conducted 5/8 through 5/9) finds:

Among 500 Democrats, Sen. Barack Obama runs at 31%, Sen. Hillary Clinton runs at 27%, and former Sen. John Edwards runs at 16% in a statewide primary.

Among 500 Republicans, former Mayor Rudy Giuliani leads former Speaker Newt Gingrich (22% to 15%) in a statewide primary; former Gov. Mitt Romney trails at 10%, Sen. John McCain at 9%, and former Sen. Fred Thompson at 8%.

Thanks to Washington Post blogger Dan
Froomkin for tipping me off to this exchange from yesterday's press
briefing by presidential press secretary Tony Snow that leads indirectly to
some fundamental questions regarding public opinion and the Iraq War:

Q Tony, you mentioned the polls, and talked about
the Republican support. All the polls also show that big majorities of the
American public do not support the war. Have you heard the President talk about
how difficult it is to fight a war or prosecute a war without the public's
support?

MR. SNOW: The President understands the importance
of public support. What's also interesting is that you see numbers coming up again on, do you think we're winning or
do you think -- for instance, a pretty
strong majority now, when asked, do you think we're losing, say no. That's
an important data point. When it talks about, would you like the Americans to
succeed, the answer is yes. So you always have mixed feelings.

[Emphasis
added]

Is Snow right? Have the numbers really "come up" that much? Snow
is presumably thinking about two nearly identical results asked on two recently
released national polls:

CNN/ORC
(5/4-6, n=1,208 adults): "Do you think that the U.S.
war in Iraq
is lost, or don't you think so?"

41% lost
55% don't think so
4% don't know

Quinnipiac
(4/25-5/1, n=1,166 registered voters): "Do you think that the U.S. war in Iraq is lost, or don't you
think so?"

41% lost
49 don't think so
11% unsure

Putting aside a quibble about whether those results amount
to a "strong majority," they certainly show more Americans rejecting than
accepting the notion that the war "is lost." The pollsters asked these questions
after Democratic Majority Leader Harry Reid told
reporters on April 19, "I believe ... that this war is lost, and this
surge is not accomplishing anything."

As first glance, these new results certainly seem like a big
change from a question traced by the ABC News/Washington Postpoll:
"All told, do you think the United States
is winning or losing the war in Iraq?"
As recently as January, a nearly two-to-one margin (57% to 29%) believed the
U.S was "losing," a five point increase in the "losing" percentage since
December:

So have opinions on the war changed really that much in just
a few months? I doubt it. The more likely explanation is that, in the context
of the Iraq War, some Americans are interpreting the "losing" and "is lost"
very differently.

Consider the most recent results of questions tracked by the
Pew Research Center
and CBS News:

Pew Research
Center (4/18-22, n=1,508 adults): "How is the U.S. Military effort
going in Iraq?"

7% very well
31% fairly well
34% not too well
25% not well at all
3% don't know

CBS News
(4/9-12, n=994 adults): "How would you say things are going for the U.S. in its efforts to bring stability and
order to Iraq?
Would you say things are going very well, somewhat well, somewhat badly,
or very badly?"

So both organizations, in surveys conducted just before and
just after Reid's remarks, show the same general pessimism reflected in the January
ABC/Washington Post "is losing"
results. By roughly two-to-one margins, Americans say the War effort is going
"badly" or "not well."

More important, while both CBS and Pew show a slight up-tick
(roughly six percentage points) in March and April as compared to December
through February, the long term trend has been to greater pessimism. And the
current overall results essentially match the reading of the January ABC/Post
poll about the direction of the war. The following chart shows the results for
the Pew survey. The CBS trend data (available via Polling Report) shows the same pattern.

Again, my hunch is that Americans interpret the words "is
lost" very differently from "is losing," though the bigger issue is whether
this semantic distinction has political consequences. A lot of conservative
commentators seem to think so, given their reaction to Reid's
remarks. Tony Snow is likely right that most Americans want our military to
succeed in Iraq.
However, if the seemingly contradictory results cited above are about a subset
of American who believe that the war is not winnable yet perhaps not yet
completely "lost," then we have a distinction without much difference.

Consider, for example the recent NBC/Wall Street Journal
Poll (4/20-23, n=1,004 adults) that reported 55% saying it is "not possible" to
achieve the U.S. goal of
"achieving victory in Iraq."
Now recall that academics that have studied public support for wars past and
present - including one that has been advising the Bush White House - believe
that "prospective evaluations of mission success" are the key to continuing
support for the Iraq War.

All of this suggests some things my media pollster colleagues
might want to test on future surveys. First, the issue of whether my hunch is
right regarding the meaning of "losing" vs. "is lost." A survey could confirm
this easily enough with an experiment that divides the sample randomly for a
side-by-side test of two questions:

Do you
think that the U.S. war
in Iraq
is lost, or don't you think so?

Do you
think the United States
is winning or losing the war in Iraq?

It might also be useful to try to get individual respondents
to decide whether "lost" or "is losing" better describes their opinion. In
other words, ask if they believe the U.S.
"has won," "is winning," "is losing" or "has lost" the Iraq war. Do those who believe the U.S. "is
losing" in Iraq but has not yet "lost" see any hope of winning in the future?

UPDATE: Professor Franklin sends along a slightly improved version of my "how are things going" chart of Pew Research Center data that adds the very similar results from CBS News and plots a smoothed regression line through the points. Click the image below for the easier-to-read full size version.

If anyone actually bothered to read
the poll, it says an oversample of 203 African-Americans were questioned out of
799 baseball fans. Considering only 12.8% of the
population is African-American (according to the U.S. Census Bureau), and
potentially a far lower percentage of baseball fans are African-American, I
would say that the 25% mark of African-Americans used to conduct the study is
designed to mislead the public and generate racially charged results.

Not exactly.

Pollsters sometimes "oversample" a
survey sub-population in order to increase the reliability of the results
for that group. More interviews means less potential random sampling error. Before
tabulating the data for the full sample, however, they "weight" back the
oversample its correct proportion with the larger sample.

I checked with Gary Langer, the director of polling at ABC News, and he
provided a few additional details. The ABC Polling Unit started with a nationally
representative sample of 1,803 randomly selected adults interviewed between
March 29 and April 4. Of these, 660 described themselves as baseball fans
(on the survey's first question). Of these, 64 were African-American.

The pollsters wanted a bigger and more reliable sampling of African-Americans. So they
continued calling from April 5 to April 22 and interviewed another 476 randomly
sampled African Americans, of whom 139 were self-described baseball fans.

Thus (adding everything up), the ESPN/ABC survey interviewed 799 baseball
fans, including 203 among African Americans. Before tabulating the data, however,
they weighted the combined sample of 2,279 (the original 1,803 plus the
oversample of 476 blacks) in a way that reduced the proportion of
African-Americans to its correct value as determined by the U.S. Census.**

This practice is not at all unusual. The intent is to generate more statistically reliable results by
race, not -- as Brown puts it -- to "generate racially charged results."

One thing I will say in Larry Brown's defense. The two-sentence methodology
blurb at the end of the ESPN was not entirely clear. For one thing, it said
that the total sample of 799 baseball fans included "an oversample of 203
African-Americans." Technically, it included an oversample that increased the
sample of African American to 203 interviews.

More important - and here is a message for all who write poll releases - it
included the term "oversample" without any description of the weighting
procedure. A sentence like this one (quoting from Gary Langer's email reply to
me) would help reduce the confusion:

The combined sample (1,803 gen pop, oversample of
476 blacks) was weighted to Census norms, reducing the proportion of
African-Americans to its correct population value.

**The US Census reports African-Americans as 11% of all adults. Browns 12.% statistic is the percentage African Americans among the full population, including children.

P.S.: And speaking of ABC Polling Director Gary Langer, the site Freakonomics
posted some intriguing comments
from Langer yesterday on the subject of whether polls have historically
overstated support for minority candidates. It includes this accurate and
well-deserved compliment from Freakonomics author Steven Dubner:

Gary
is a force of nature. He not only runs ABC's polling but has become the
network's top cop for keeping bad data off the air, vetting many of the
surveys, studies, and polls that producers and reporters plan to use in their
stories. I don't know of any other news organization that has such a resource.
I am sure he is occasionally a thorn in the side of a reporter who's dying to
cite some sensationalistic study from some biased organization ... but as
consumers of news, we are all the better for it.

As Today's Washington Postreports
on a new in-person survey
of current residents of the New
Orleans area conducted by the Kaiser Family
Foundation. The survey, according to the release on the Kaiser site,

documents the devastating impact
that Hurricane Katrina and the failure to respond quickly and effectively to it
has had on the economic well-being, physical and mental health, and personal
lives of the people of the New Orleans
area. The survey also found a sharp divide in the way that African Americans
and whites in the New Orleans
area experienced the storm and perceive the recovery efforts, especially in
hard-hit Orleans Parish. Future Kaiser surveys are planned in 18 months and 36
months to monitor progress and changes.

The
methodology of this survey is as interesting as the findings, at least to polling
geeks like me. Telephone surveys present some obvious challenges in the areas
affected by Katrina, particularly in reaching those still living in trailers or
who otherwise lack landline telephone service. As such, the the Kaiser
researchers went back to basics and conducted in-person interviews using a "two-stage,
stratified area probability sample" and a team of 41 interviewers. Their
interviewers visited 456 randomly selected census areas and documented the physical
condition of nearly 17,000 housing locations, then attempted to conduct interviews
in randomly selected households. The methology is all explained here; the complete
100-page study is available for download here.

Among 543 Democrats, Sen. Hillary Clinton leads Sen. Barack Obama (28% to 20%) in a statewide primary; former V.P. Al Gore trails at 13%, former Sen. John Edwards at 8%.

Among 375 Republicans, former Mayor Rudy Giuliani leads Sen. John McCain (36% to 15%) in a statewide primary former Gov. Mitt Romney trails at 9%, former Sen. Fred Thompson at 7%, former Speaker Newt Gingrich at 5%.

New analysis from the recent USA Today/Gallup national survey of 491 Democrats and Democratic-leaners (conducted 5/4 through 5/6) looks at support for Sen. Hillary Clinton and Sen. Barack Obama and finds:

Among 265 Clinton supporters, 35% are more likely to support her over Obama because "Clinton is more experienced;" 23% "like Clinton's views on issues/agenda."

Among 194 Obama supporters, 19% are more likely to support him over Clinton because "Obama is a fresh face/has new ideas;" 18% "agree with Obama's issue politions/agenda."

Additional results from the recent Chernoff Newman/MarketSearch statewide survey of likely Republican primary voters in South Carolina tests "branding" and vote preference of the Republican candidates. Mark Blumenthal's analysis of the release of the Democratic candidates here.

Three new American Research Group statewide surveys likely voters in California, Florida, and Michigan (conducted 5/4 through 5/8) finds:

Sen. Hillary Clinton leads Sen. Barack Obama in California (37% to 28%), Florida (45% to 17%), and Michigan (38% to 25%); Sen. John Edwards trails in each state with 15% or less.

Former Mayor Rudy Giuliani edges out Sen. John McCain (27% to 24%) in California and leads him (31% to 18%) in Florida. In Michigan, former Gov. Mitt Romney runs at 24%, McCain at 22%, and Giuliani at 19%

Although we try hard to promptly report polling related new items,
we missed a good one yesterday. Conservative commentator Ann Coulter dropped
another one of her bon mots during a
Fox News broadcast, this one aimed at the pollsters at Newsweek. On his show At Large on Monday, host Geraldo Rivera asked
Coulter for a reaction to Newsweek poll
results showing Barack Obama leading the various Republican contenders. As reported
by News
Hounds (via colleagues on the AAPOR list serv):

She went on to say that in polls "where
Republicans are actually allowed to vote," they "do a lot better."
thus suggesting that Newsweek did not call any Republicans.

News Hounds also linked to a YouTube video clip:

Not surprisingly, reaction from the pollsters was swift. Rob
Daves, president of the American Association for Public Opinion Research (AAPOR), released the following statement
yesterday:

"Ann Coulter's
kill-the-messenger assessment of the Newsweek poll exceeds the bounds of
professional reporting and commentary," said AAPOR president Robert P.
Daves, director of polling and strategic research at the Star Tribune.
"It's pure-and-simple mudslinging on her part to question the ethics of a
reputable polling firm and news organization that abides by a longstanding Code
of Professional Ethics and Practices. Anyone who values solid public opinion
research and intelligent public discourse should be outraged and ignore
Coulter's irresponsible and groundless assertion that the results were
fabricated."

[Interests disclosed: I serve as a member of AAPOR's executive
council].

And finally, Evans Witt, the chief executive officer of
Princeton Survey Research, the survey firm that conducts the Newsweek poll, had
this comment for reported by the Associated
Press [that was originally posted to the AAPOR member-only listserv):

"As the 2008 election campaign
continues to heat up, I am sure that there will be informed and incisive
criticisms of polls from many observers," he said. Coulter's comments
"do not fit into this category," he added.

When it comes to Ms. Coulter's various lunatic
comments, I tend to agree with those say
that "she, like any bully, will go away if you ignore her." On the other hand, read
over some of the comments left on this site over the last few days and you will
see that some people will believe anything. So for those tempted to confuse the
valid skepticism expressed by reputablevoices
about the reporting of Newsweek poll
with Coulter's rant, let's restate what should be obvious: Coulter's remarks about
the Newsweek poll are unfounded and preposterous.

Additional results to the new CNN/Opinion Research Corporation national survey (story, results) of 1,028 adults (conducted 5/4 through 5/6) finds:

54% disapprove President Bush's veto of a bill providing additional funds for Iraq with a specific date for complete troop withdrawal; 44% approve.

When asked to choose among three bills, 40% prefer "a bill that would provide additional funds for Iraq that would set benchmarks for the Iraqi government but not set a specific timetable for troop withdrawal;" 33% prefer "a bill that would provide additional funds for Iraq that would set a specific date to start withdrawing troops;" 24% prefer "a bill that would not provide additional funds for Iraq and would require all troops to be withdrawn by March."

A new InsiderAdvantage/Majority Opinion national survey of 621 registered voters (conducted 5/4 through 5/6) finds 39% of Americans favor the impeachment of President Bush and Vice President Cheney; 54% do not.

The mysterious result de jour is a bit of a conflict between
the most recent USA
Today/Gallup and CNN/ORC polls on
the national Republican primary horserace result. The Gallup survey has shown Rudy Giuliani leading
John McCain by 14 and 13 points on their last two surveys (34% to 20% on the
survey released today). Meanwhile, CNN/ORC shows a much closer race, with
Giuliani leading by only 3 points last month and 2 points (25% to 23%) now. Both
organizations poll over exactly the same period this week, and nearly
overlapping periods a month ago. So why the difference?

Why the difference? I have been looking at the numbers and
see no obvious explanation. The sample sizes are relatively small (with a
reported margin of sampling error of roughly +/- 5%), but differences in the
Giuliani vote are statistically significant, especially given the April-to-May
consistency for both polls.

The most important difference between the two polls - as Hotline OnCall points
out - is that the Gallup polls asks its Republican primary vote question
among all adults that identify or
lean Republican, while CNN asks it only of self-described registered voters that identify or lean Republican. I might expect
such a difference to work in Giuliani's favor, on the assumption that he might
be less popular among the more hard core Republican primary voters. But there
are two problems with that theory. The first is that despite the labeling, the
percentage of adult respondents that were asked the Republican vote question is
only slightly smaller on the last two CNN surveys (40% and 39%) than on the
last two Gallup surveys (42% and 43%).

That result is a little odd, since other pollsters that screen
for registered voters who identify or lean Republican tend to ask their
Republican primary vote question of a smaller percentage of their samples. Looking
at recent surveys that provided comparable details, I get 35% (of the sample
asked the Republican primary vote question) for the Pew
Research
Center, 32% for LA Times/Bloomberg, 31% for Time/SRBI and 30% for the Cook/RT
Strategies.

Moreover, if we look at the other results from comparable
questions over the last month, the other pollsters that screen for registered
voters (including those with effectively tighter screens than CNN) have mostly
not shown the race as close as CNN/ORC.

Which makes these results a puzzle, except for this general
rule of thumb: When very small (even invisible) differences in methodology make
big differences in the results, it usually means that the underlying attitudes
are not strongly held.

In other words, most Republicans nationally have only begun
to consider the candidates, much less who they will support in 2008. So their
answers to pollsters are showing great variation. When voters start to make up
their minds -- next year -- these results will show more consistency.

The polls are really rolling in yesterday and today and they provide a striking contrast and hence a good lesson.

The latest to arrive is actually a bit old but only just released. The NPR poll conducted 4/26-29/07 is actually the sixth oldest of the recent polls. It found approval at 37% and disapproval at 59%. The NPR poll is the product of a bi-partisan team of Greenberg Quinlan Rosner Research (D) and Public Opinion Strategies (R). They sample likely voters, rather than adults or registered voters. This sample population mixes with house effects to generally puts the NPR poll about 1.9 percentage points above the trend estimate. In fact, the trend estimate is now 34.7 and NPR is 37%, a 2.3 point discrepancy which is reasonably close to what the house effect would predict.

The interesting comparison and lesson is that the Newsweek story over the weekend stressed a new all time low in approval of President Bush while other polling has failed to find any strong downward trend (except for a Harris poll taken 4/20-23 that also had approval at 28%.) In my trend estimator, when Newsweek was the endpoint, the trend estimator was pulled down to 33.2%. I warned then that Newsweek was an outlier and that an estimate of 34 or so was more likely. The flurry of new polls since Newsweek help reinforce that conclusion.

Here are the last twelve approval polls in order:

38 33 35 35 28 32 37 35 35 28 34 38

The mean is 34% and the median is 35%. The two polls at 28% stand out pretty strongly from the rest.

The lesson for interpreting polls is that a single poll does not a trend make. The strong interpretation Newsweek gave its poll, and the extra hype it got on Sunday talk shows, was not supported by other evidence. I stress that I'm complaining here about interpretation of poll results, and not about the poll itself. The Newsweek poll was an outlier, but outliers happen and that should not lead to a conclusion of anything other than a bad random outcome about the poll. The news story that was based on the poll, on the other hand, took the findings and made a very strong claim about approval at a new low when other evidence is clear that approval has not fallen into the 20s. One simply cannot ignore all other polls when making claims about a trend.

The ultimate irony of this is that Newsweek is right: President Bush IS at an all time low for his presidency, and has been since January! His stable approval rate of 33%-35% in my trend estimator is indeed the lowest of his presidency, and he has shown no signs of recovery (though no sign of further decline either.) But that all time low for the trend is 33-35%, not 28%. The desire to pick each poll number that happens to be well below the trend estimate and emphasize how low it is misleads us from attending to the central trend estimate which has far more to say about the actual level of presidential support. Each poll is a random draw around the trend. Some will be lower, some higher. But that poll to poll variation is largely random. It is the trend that captures the systematic path of approval.

And so I conclude as I did originally about the Newsweek poll. The best estimate of approval remains in the 33-35 percent range. At the moment the trend is at 34.7, but it could easily drop a bit with new polls, or possibly rise a little. My money is on an average of 34, for the moment.

Below are the usual poll diagnostics. I commented on these more extensively in the last two approval posts, so I'll not say more about them here.

A new National Public Radio national survey (story, results, highlights) of 1,000 likely voters (conducted 4/26 through 4/29 by Greenberg Quinlan Rosner Research (D) and Public Opinion Strategies (R) ) finds:

37% approve of the way President Bush is handling his job; 59% disapprove.

56% of Democratic primary voters say Congress and President Bush should pay the most attention to the "war in Iraq," while 30% of Republicans agree.

A 37% plurality of Republican primary voters say Congress and Bush should pay the most attention to "terrorism and national security," while 11% of Democrats agree.

34% of Americans approve of the way President Bush is handling his job; 63% disapprove.

Among 427 Republicans and leaners, former Mayor Rudy Giuliani leads Sen. John McCain (34% to 20%) in a national primary; former Sen. Fred Thompson trails at 13%, former Speaker Newt Gingrich at 8%, former Gov. Mitt Romney at 7%.

Among 491 Democrats and leaners, Sen. Hillary Clinton leads Sen. Barack Obama (38% to 23%) in a national primary; former V.P. Al Gore trails at 14%, former Sen. John Edwards at 12%. Without Gore, Clinton leads Obama 45% to 27%.

Three new polls put President Bush's approval at 34% or above, and bring my trend estimate to 34.3%. The Hotline poll, taken 4/26-30/07 but only released yesterday, has approval at 35%, disapproval at 62%. USAToday/Gallup, 5/4-6/07 has approval at 34%, disapproval at 63%. CNN/Opinion Research Corp., 5/4-6/07 has approval at 38% and disapproval at 61%.

These results contrast with the weekend release of a Newsweek poll at 28%. Newsweek's story on the poll stressed the new low rating for President Bush. A Harris poll some 10 days earlier had also pegged approval at 28%. However, other recent polls at 32-35% failed to offer evidence for a sharp decline in Bush's approval. With the addition of these new polls at 35%, 34% and 38%, the trend estimate has returned to "flat", continuing a long period of stable approval in the 33-35% range which has persisted since December.

The recent polls are plotted below. The CNN poll at 38% is a good bit above the trend estimate, while Gallup and Hotline are quite close. The CNN result is also a bit high compared to previous CNN polling which generally tracks the trend quite closely.

The addition of these new polls brings the trend estimate up, making the previous Harris and Newsweek surveys look like larger outliers than before. The upward shift also keeps the rather high CNN estimate within the 95% confidence interval, meaning that while it is high it is not a statistical outlier.

The current trend remains within the uncertainty range of the trend estimate. The current uncertainty, with polls ranging from 28-38 percent, is rather large but not suspiciously so.

Finally, the various estimates of trend for the last 20 polls shows a strong clustering in the 34-35 percent range, so the evidence remains good for approval in this narrow range.

A new SurveyUSA automated survey of likely primary voters in California (conducted 5/4 through 5/6) finds:

Among 456 Republicans, former Mayor Rudy Giuliani leads Sen. John McCain (34% to 21%) in a statewide primary; former Gov. Mitt Romney trails at 12%, former Sen. Fred Thompson at 11%, and former Speaker Newt Gingrich at 9%.

Frank Newport is breaking news on his Gallup Guru blog that the latest USA Today/Gallup job approval rating for President Bush is 34%. Earlier, Newport voiced some caution about the "all time low" in Bush approval in this weekend's Newsweek poll, including a link to similar advice from Charles Franklin. USA Today and Gallup should have more details tomorrow, and we will post all the links.

As long as we are on the subject of debates, I want to pick
a nit with something Joe Klein noted in his assessment
of the Republican debate last Friday:

Here's my gimmick: I listened to
the debate on radio. Not
that I wanted to...but I was on a forced march, driving north. I did this fully
aware that Nixon "won" the famous debate with JFK among radio
listeners as he was losing the presidency on TV.

My issue here that while entirely plausible, the evidence
that Nixon "won" among radio listeners is about as sketchy as the evidence that
Giuliani "won"
among Californians last week. I don't mean to pick on Joe Klein. Two Colorado State University
academics who reviewed the literature 20 years ago found that assertions Richard Nixon
winning among radio listeners "prevail in nearly all accounts of the first
Kennedy-Nixon debate" (Vancil and Pendell, 1987).

It turns out that aside from a few anecdotal accounts from
journalists, only one true survey attempted to gauge reactions to the debate
among both radio and television audiences. It involved a few questions asked on
a larger omnibus telephone survey conducted by Sindlinger and Company, and the
cross-tabulation by audience (television and radio) was published only once, in
the November 7, 1960 issue of Broadcasting
(from Kraus, 1996, p. 80):

Kennedy supporters may be grateful
that television was invented before the "Great Debates" took place. The
Sindlinger research showed that Mr. Kennedy was routed by Mr. Nixon on radio.

In answer to the question who won
the debates, 48.7% of the radio audience named Mr. Nixon and only 21% picked
Mr. Kennedy. Among those who watched the debates on tv, 30.2% named Mr. Kennedy
the winner and 28.6% picked Mr. Nixon.

According to the Sindlinger
projections, the total television audience was about 4 - ½ times the radio
audience - 270 millions viewers of 5v to 61.4 million listeners to radio.

That result, however, has a few problems, the most important
of which is the relatively small size and unrepresentative nature of the radio
audience. It amounted to must 282 responses from Sindlinger's sample of 2138
respondents, but the pollster apparently misplaced the original data, because no
information survived regarding the partisanship or vote preference of the radio
or television subgroups. That omission is critical because, as Steven Chafee, a
professor of communications at the University
of California Santa Barbara
has observed (2000, p. 334):

By 1960, those who could listen to
debates only on radio were far from a random lot. Situated for the most part in
remote rural areas, they were overwhelmingly Protestants and skeptical of
Kennedy as a Roman Catholic candidate.

University
of Minnesota
Political Scientist James N. Druckman, sums up (p.
563):

Put another way, relative to
television viewers, radio listeners may have been predisposed to favor Nixon
over Kennedy. This lack of reliable causal evidence means that a prime example
of the power of television images may be nothing more than "telemythology"
(Schudson, 1995, 116).

Or is it? After all, much of the power of this debate
anecdote is that most observers and commentators thought Nixon lost the debate
to Kennedy on the basis of his appearance. It certainly seems plausible that
had the same debate occurred on radio, Nixon may have fared better.

Druckman took the matter one step further. In the article
quoted above, he describes an intriguing experiment conducted about five years
ago. He recruited 171 respondents - mostly students - who demonstrated little
or no knowledge of the Kennedy-Nixon debates. He then randomly divided the
subjects into two groups. Half watched a video tape of the first Kennedy-Nixon
debate and the other half listed to just the audio. All later answered
questions about Nixon and Kennedy, yielding the results Druckman had expected (pp.
569-570):

Television images matter - they
prime people to rely more on personality perceptions when evaluating
candidates, which in turn, can affect overall evaluations. Images also enhance
political learning, at least among nonsophisticates. The experiment provides
evidence that Kennedy may have done better on television because of his
superior image.

So there we have it. A great example of how not to interpret
a post-debate survey, and a clever experiment by a social scientist that provides
a bit of evidence to support the underlying truth beneath the well established
"telemyth."

References used after the jump (links are to subscriber only
databases).

The Newsweek poll released this weekend was quickly criticized for under-representing Republican partisans. For the adult sample, Newsweek had 22% Republicans and 35% Democrats, while for registered voters their numbers were 24% and 36% for Reps and Dems. Independents were 36% of adults and 37% of registered, with 7% of adults and 3% of registered refusing or unable to answer the question.

How out of line are these results? The graph above shows the distribution of party id for all polls taken since January 1, 2007. The groups are for three populations (Adults, Registered Voters and Likely Voters) and for whether the party ID question is "leaned", that is does it include those who say they "lean" to one party or the other, after first saying they are independent or non-partisan. In the legend for the graph A.N is adults, not leaned, RV.L is Registered Voters, Leaned, and so on.

It is clear that the distribution of Republicans, Democrats and independents depends quite a bit on the population and the question wording. For the adult, not leaned, sample that Newsweek used for presidential approval, the 22% Republican they found is at the low end of samples in 2007, where the median is 25%. Among registered voters, not leaned, the median in 2007 is 29%, compared to Newsweek's 24%. For Democrats, the Newsweek sample at 35% for adults compared to a median of 35%, while among registered voters their 36% compares to a median of 35%.

So the Newsweek results are a bit low (-3%) for adult Republicans, and a bit lower (-5%) among registered voters, but quite close to the median for Democrats. While the adult Republican number is the lowest of the year in comparable samples and questions, it is only 3 points from the median. That is indicative of the relatively low spread of estimates of Republicans in the adult population this year, where half of all polls fall between 24% and 28% and only 3 of 24 polls fall above 30%. Among registered voters with unleaned questions, half of all polls fall between 27% and 32%, with four polls above 32% and six polls below 27%. Newsweek's registered voter sample is the second lowest reading for Republicans of this group.

The relationship between percent Republican and percent Democrat is not terribly strong within categories of population and question wording, as shown below.

If anything, there is a positive correlation of the two partisan groups, ranging from .44 to .81 for adults and registered voters. (The four cases of likely voters have little variation and produce negative correlations, and I discount this.) The positive correlation is a good indication that what is at work with partisanship is that some questions or survey organizations produce more or fewer independents, driving down or up both Reps and Dems simultaneously.

The variation we see above is a reminder that party identification is a central concept, but subject to quite a lot of variation in measurement depending on population and question wording (and probably "house effects" due to survey organization, a topic ignored in this post.)

Given the sample size of each partisan group, the range of variation we see across polls is not surprising, and that raises an important point. One of the first things we ask about a poll is what its partisan split is. That is an important diagnostic to have available, and Newsweek deserves to be praised for openly disclosing this bit of data. All pollsters SHOULD release this information but a number do not. The current criticism of Newsweek is exactly the disincentive these organizations have for such release. If your party id is even a bit off, it becomes an immediate target for criticism and partisan cries of "fraud". Such is an uninformed view. Normal sampling variation will produce a range of party id estimates, and the data above show that in general this variation is well behaved, doing about what we'd expect given the sample sizes involved. But it does provide a tempting target for anyone unhappy with a particular poll.

This is not to say we should not scrutinize pollsters based on their data. That is entirely what we do here, of course! And I think I was the first to point out that the latest Newsweek poll appears to be an outlier on presidential approval. But that conclusion was based on a great deal of evidence from over 1400 polls and a consistent methodology I apply to every new poll regardless of source and regardless of results.

Finally, it is easy to overstate the impact of the distribution of party id on other survey estimates, such as presidential approval or horserace results. While party id is a powerful individual level predictor of those things, a discrepancy of 3-5 percent in partisanship, as in this case, does not translate directly into a similar discrepancy in every question with partisan overtones. Partisans are not completely homogeneous in their views, so the impact of variation in party id is mitigated by the variation within partisan category in approval or candidate preference. If we want to analyze a poll's results, it is better to focus directly on the question of interest, rather than on another variable like party id which may influence but not entirely determine the result we care about.

A new WBZ-TV/SurveyUSA automated survey (story, results) of likely primary voters in New Hampshire (conducted 5/4 through 5/6) finds:

Among 551 Republicans, former Gov. Mitt Romney (32%) leads former Mayor Rudy Giuliani (23%), Sen. John McCain (22%) and former Sen. Fred Thompson (11%) in a statewide primary. In January Giuliani ran at 33%, McCain at 32%, and Romney at 21%.

The 2,500-word examination
of Barack Obama and race in today's Washington
Post touches briefly on the issue of whether polling, "once" considered "wildly
unreliable" in races involving black and white candidates might be more
trustworthy than in past elections. It quotes Scott Keeter of the Pew Research
Center, who co-authored a
report
on the issue back in February.

But what caught my eye is a bit of overlooked survey data
relevant to the larger story. Consider the tongue-in-cheek reaction to the Post
story this morning from Time's Ana
Marie Cox:

The results to this question show virtually no variation by
party. Among Democrats, 48% consider Obama black, 7% mixed/other, 7% mixed black
and white.

One thing unclear as of this writing: Did interviewers read
the two "mixed" categories to respondents or did respondents volunteer those
answers? I will check on that, but if "mixed" was not presented as an option,
some confusion is to be expected given Obama's famously "biracial and
multinational background" (to quote the Post article). [UPDATE: Doug Schwartz at Quinnipiac emails that this questions was entirely open-ended and did not prompt respondents with any categories].

Meanwhile, there is a very intriguing cross-tabulation I
would like to see: How much different are the results to the primary horserace question
and the Obama favorable rating among the 63% of Democrats that answered the
question correctly as compared to the 37% that did not. The Democratic sample
is relatively small (n=499) but still large enough for such a comparison. Unfortunately,
the survey has nowhere near enough interviews to allow the really interesting test: How African-American Democrats answered this question.

Additional results from the most recent Quinnipiac University national survey of 1,166 registered voters (conducted 4/25 through 5/1) tests thermometer ratings among 18 different countries among men, women, republicans, democrats, independents, and white evangelical/born again Christians.

Among 400 300 Republicans, Sen. John McCain edges out former Gov. Mitt Romney (19% to 15%) in a statewide primary; former Sen. Fred Thompson runs at 13%, former Mayor Rudy Giuliani at 12%, and former Speaker Newt Gingrich at 7%.

A new Newsweek national survey (story, results, Charles Franklin's analysis) of 1,001 adults (conducted 5/3 through 5/3 by Princeton Survey Research Associates) finds:

28% approve of way Bush is handling his job as president, "the lowest presidential approval rating in a generation;" 64% disapprove.

More Americans say Sen. Hillary Clinton, Sen. Barack Obama, and former Sen. John Edwards have "shown political courage in their current campaign" than have former Mayor Rudy Giuliani, Sen. John McCain, and former Gov. Mitt Romney; 48% say Giuliani has shown it "in the past," 43% say Clinton has.

A Newsweek poll taken 5/2-3/07 finds President Bush's approval rating fallen to 28%, with disapproval at 64%. This is a new all time low for Newsweek polling on Bush and ties the lowest readings of Harris polls (taken 4/20-23/07) and an older CBS poll taken 1/18-21/07. No poll has found a lower approval reading for the President.

With the addition of this Newsweek poll, my trend estimate stands at 33.2%, also an all time low.

Before concluding that Bush's approval has crashed, however, the Newsweek poll should be compared to the recent trend and to other recent polls. The last six polls have registered approval at 35, 28, 35, 32, 35 and 28. So the last six readings are three at 35, one at 32 and two at 28, a seven point range.

In the figure below, the Newsweek poll is clearly well below the current trend estimate, and more importantly, is below the typical Newsweek poll, which is normally about 2-3 points below the trend. The current poll is 5.2 points below trend.

This means that Newsweek is a bit atypically low, compared to both other polls and to its own normal trend.

A check of the outlier analysis shows that Newsweek, like Harris 10 days earlier, is in fact an outlier when compared to the variation around the trend estimate for all polls.

The Newsweek and Harris polls are exerting a significant pull on the trend estimate, even when balanced against the three recent polls at 35%. The current small downward slope in the trend estimate is due to the addition of Newsweek. It was flat prior to adding the new poll.

The uncertainty of the trend estimate can be seen in the gray area in the plot below. While the trend is moving a bit down since early 2007, it has remained well within the gray region, suggesting we have no reliable evidence that approval is moving either way. Since January the approval trend has sometimes moved up slightly and sometimes down slightly, but has never sustained either trend long enough to move out of the uncertainty band. The current downward slope may again be reversed with new polling (or not-- let the data speak.)

The sensitivity analysis below shows how the last 20 trend estimates have moved around as each new poll has been added. the substantial effect of the new Newsweek poll is apparent in the gap between the current estimate and the previous cluster of estimates between 34% and 35%.

The bottom line is that the evidence for a new decline in approval remains quite mixed, with two polls claiming a rather dramatic drop in approval, while several others suggest continued stability around 32%-35%. Given the outlier analysis and the fact that the Newsweek poll is clearly below that poll's own trend, we must still wait before concluding that approval has taken a sharp turn down. The evidence remains in favor of rough stability around 34%, though the trend estimate now stands a little below that, in deference to Newsweek's evidence.