Demystifying the Science and Art of Political Polling - By Mark Blumenthal

February 28, 2006

The Zogby Poll of Troops in Iraq

[3/1 - Note to regular readers: I have added additional thoughts at the end of this post].

MP has received email from several readers asking about a just released survey of troops currently serving in Iraq conducted by Zogby International and noted in Nicholas Kristof's column in the New York Times this morning.

According to the release, Zogby conducted the survey in collaboration with the Center for Peace and Global Studies at Le Moyne College. Those who click through to the Zogby summary will find the following methodology statement:

The survey included 944 military respondents interviewed at several undisclosed locations throughout Iraq. The names of the specific locations and specific personnel who conducted the survey are being withheld for security purposes. Surveys were conducted face-to-face using random sampling techniques. The margin of error for the survey, conducted Jan. 18 through Feb. 14, 2006, is 3.3 percentage points.

I wrote about another survey of troops in Iraq conducted last year by the Military Times newspapers (here and here) and know that polling active duty troops is no easy task. So I sent an email to John Zogby this morning asking if he could describe the survey a bit more in general terms if not in specifics.

Zogby returned my call this afternoon. While not exactly a fan of this site (to put it mildly) he was courteous enough to provide a more in-depth explanation of what his company did and why he is unwilling to disclose more publicly. Unfortunately, the ground rules for our conversation prevent me from sharing much of what he told me. But I did come away convinced that Zogby has good reason to withhold the details of how he was able to interview U.S. troops the way he did. More disclosure could put the interviewers' lives at risk.

Here is what I can say:

The Center for Peace and Global Studies paid Zogby to conduct the study but otherwise played no role in conducting interviews or gathering the data. [Correction: In our conversation Zogby indicated that an "anti-war" sponsor paid for the survey but played no role in conducting interviews or gathering the data. I wrongly assumed he meant the Center for Peace and Global Security, whose faculty according to a story in yesterday's Syracuse Post Standard, did help "develop and word the poll's questions." In fact, the funder was "a wealthy war opponent who [Zogby] would not name." Thanks to reader Bob for pointing out the Post Standard story. Apologies for the error. See this update for more details].

According to the procedure Zogby described, respondents were intercepted randomly (e.g. they were not self selected) at multiple locations throughout Iraq (e.g. not just in the so-called "Green
Zone") and interviewed using a paper questionnaire that they filled out with the assistance of an interviewer.

Zogby was willing to share the specific geographic locations where they collected data on the condition I not repeat them. I passed on the offer as my knowledge of Iraq and military operations there is cursory at best, but I have no doubt his offer was genuine.

Zogby provided Nick Kristof and others reporting on the poll full details about his methodology on an "off-the-record" basis.

So in short, I can tell you that Zogby found a creative solution to the difficult problem of polling troops in Iraq, but I promised to say no more than that. I asked Zogby what advice he would offer data consumers who find this all puzzling. In this case, he said, "you have to trust me."

PS: I neglected a hat tip to the reader who blogs at Fickle Minded and first emailed me about the Zogby poll.

PPS (3/1): Readers should not interpret my commentary above as either an endorsement or criticism of Zogby's methodology. However, in response to some of the very reasonable questions posed by readers in comments and email, I want to clarify a few points.

First, to be clear, the Center for Peace and Global Studies is in effect a "partisan" sponsor in that, according to Zogby, they opposes the war in Iraq. [Correction: As noted above, the survey had a partisan sponsor that opposes the war in Iraq. However, that sponsor was not the Center for Peace and Global Studies, which collaborated on the study but did not fund it. Apparently, the Center, despite its name, is a non-partisan academic institution that takes no formal position on the Iraq war. The confusion on this point was mine. Again, apologies for the error].

Second, while Zogby says his interviewers selected respondents randomly at various locations, he makes no claim of random selection with respect to the locations involved. I apologize for being so vague, but the most I can say is that the method Zogby used to gain access to those locations constrained his ability to make random selections.

So to evaluate this survey, one important question is whether the troop populations accessible at the locations Zogby selected are representative of all troops in Iraq. This issue is analogous to the question of whether precincts selected for an exit poll are representative of all precincts in a given state (although exit poll precinct selection is usually random). Zogby believes the locations involved provide a reasonably representative sampling and, as noted above, he offered to share the names of specific locations on an off-the-record basis. I declined largely because I lack the knowledge and resources to make an independent assessment. So for me, this question remains open.

Third, even if consumers of this data knew all that I know about how Zogby's interviewers "walked up to troops" (as commenter Karen puts it), they would still have questions about the impact of such an interaction might have on the kinds of troops most likely to agree to participate in the survey. Consider the exit poll example again. Even though exit pollsters have disclosed the procedures they use to train interviewers and select respondents, we still debate the effect of those procedures on the kinds of voters that choose to participate. Disclosure in that case cannot resolve all questions, but it at least enables an informed debate. Unfortunately, such discussion and debate is impossible in this case.

The survey procedures Zogby described to me involved compromises analogous to those used in surveys of Katrina victims conducted by the Washington Post/Kaiser/Harvard and CNN/USAToday/Gallup. A survey of troops in Iraq would be impossible without some sort of methodological compromise along these lines, and Zogby's approach may be the best available. Yet my "bottom line" on this survey remains uncertain. I hope that those with more military expertise will assess how well it represents the troops in Iraq in terms of the age, rank, branch of service and any other similar characteristics (and no, I did not ask Zogby whether he weighted or adjusted his results).

The issue of disclosure is more difficult. Note that for the Katrina surveys, news organizations clearly characterized the results in terms of the limitations of the survey design. For example, the Washington Postreported results from "evacuees living in shelters." USAToday reported on the opinions of "residents who sought help from the Red Cross." Both organizations spelled out the methods used to contact respondents so inquisitive consumers could reach their own conclusions about the value of the data. Yes, Zogby has good reason for withholding the details in this case, but the secrecy limits the ability of consumers to evaluate the data and of news organizations to report it.

The toughest question here is whether it is appropriate for news organizations to report on a survey with partisan sponsorship that requires readers and viewers to place an unusual degree of trust in unspecified methods and procedures. I will admit I do not have a good answer for that one -- different news organizations will apply different standards -- but big "grains of salt" are certainly in order.

PPPS: In addition to Democracy Project (which has track-back links below), see also the posts by The Officer's Club and Murdoc Online for a different perspectives on the Zogby poll.

Also a reader emails with a very good question that I probably should have put to Zogby:

What is the problem with releasing the demographics on the poll? Were it truly
representative, it would have a breakdown of service branches proportional to
the military in Iraq. Same with the breakdown for gender, age, etc. I can't
imagine any security risk in releasing the demographics, or motive for keeping
them hidden other than they would expose the lack of validity of the survey.

Neither the poll release nor (I'm told) the full report available for sale provides statistics on the demographic composition of the sample.

Ports: CBS News & Cook/RT

Two new polls confirm the Rasmussen result discussed yesterday that shows Americans overwhelmingly oppose the deal to allow an Arab company to operate U.S. shipping ports. The new surveys from CBS News (story, Bush results, Katrina results) and the Cook Political Report and RT Strategies (Charlie Cook column, results) also show significant drops in the approval rating of President George Bush since January.

The CBS and RT Strategies polls explain the ports issue in more detail, yet still show more than 60% of Americans now oppose to the deal:

CBS News - As you may know, the Bush Administration has agreed to let a company from the United Arab Emirates run six shipping ports in the U.S., including ports in New York and New Orleans, that are now being run by a British company. Critics of the plan say that allowing a company from an Arab country to operate U.S. shipping ports is dangerous to national security. The Bush Administration says security will be protected by the U.S. and that the United Arab Emirates is a U.S. ally. Do you think the U.S. should or should not let a United Arab Emirates company operate U.S. shipping ports?

21% Should
70% Should not
9% Don't know

RT Strategies - Here is something that has been in the news recently. An Arab-government owned company has been cleared by the U.S. government to run major shipping operations at six major seaports here in the United States. Some believe that this proposal could lead to a potential security threat at these U.S. ports, while President Bush disagrees. Bush says the government has carefully made this decision and there is no security threat. Do you think (ROTATE:) Congress should take special action to block the government's decision, or we should trust President Bush and his Administration in their decision?

Both CBS and RT Strategies provide tabulations of these results by party ID, which suggest that the different question language used by the two pollsters matters mostly to Republicans. On the CBS survey, 58% of Republicans say the U.S. "should not let a United Arab Emirates company operate U.S. shipping ports," while RT Strategies shows only 32% of Republicans believe "Congress should take special action to block the government's decision." The difference is mostly likely the different language used in the alternative posed by the RT Strategies: "or we should trust President Bush and his Administration in their decision?"

All three survey organizations now show President Bush with a statistically significant decline in his job approval rating since their last survey:

CBS News shows an eight point decline Bush's approval rating, from 42% in late January to 34% on the most recent survey.

RT Strategies shows a seven point decline, from 47% approval in late January to 40% now.

Rasmussen's average result has declined from 47% earlier in February to 44% over the last six days.

Tabulations of the Bush rating by party identification show that the declines cut across all partisan groups in both surveys:

Some will no doubt seize on the fact that the latest CBS News sample is a few points more Democratic on party ID (37%) than on their last three surveys (34% in late January, 33% in early January and 32% in December), although the Republican percentage (28%) is about the same as the last three surveys (27%, 29% and 28% respectively). However, the difference in the party results does not explain the drop in the Bush job rating, which occurs across all three categories.

In fact, even when MP recalculates the CBS job approval results for the most recent survey using the average party composition reported on their last three surveys (33% Democrat, 28% Republican, 39% independent or other), the Bush approval percentage still rounds to 34%. The reason is that my recalculation just increases the number of independents at the expense of Democrats. However, Bush's rating is now so low among both subgroups as measured by CBS that the adjustment makes little difference.

February 27, 2006

Rasmussen: Dubai Ports Deal

The Rasmussen Reports automated poll was the first out of the blocks late last week with results on the Dubai ports deal. Although unmentioned in most of the "mainstream" media, the results did get references over the weekend in the New York tabloids (Daily News and the Post) as well as a host of prominent (Kaus, Sullivan, RealClearPolitics, The Corner, The Plank). We will, no doubt, hear much more about the ports deal in the next round of conventional public polls, but for now, Rasmussen's results are the only available. The question is, how seriously should we take them?

Reasons for Skepticism: First, Rasmussen himself is skeptical. I exchanged email with Scott Rasmussen over the weekend, and he cautioned that they conducted their survey on Wednesday and Thursday nights last week, "at the height of furor" as he put it. Rasmussen said they plan to track their Dubai ports questions again in a week or so.

Second, many Americans are probably not following the issue closely. Nineteen percent (19%) of those surveyed were unable to answer Rasmussen's question on the Dubai Ports deal. Compare that to his presidential job approval question, which normally produces a "don't know" of about 1% (much lower than other conventional surveys). Moreover, as the Rasmussen summary notes, on a subsequent question, less than half of those surveyed know that foreign firms currently operate U.S. ports:

Just 39% of Americans know that the operating rights are currently owned by a foreign firm. Fifteen percent (15%) believe the operating rights are U.S. owned while 46% are not sure.

For all of these reasons, the conventional pollsters will probably ask questions that -- like those asked about the NSA wiretapping -- explain the issue to respondents in some detail. The wording they choose may produce results different results from what Rasmussen obtained.

Finally, there is the conventional wisdom that Rasmussen's polls "need to be taken with a bucket-full of salt," as Andrew Sullivan put it last week. This is a bigger can of worms than MP has time for this morning, but it does reflect a deep skepticism about Rasmussen among conventional survey practitioners. That skepticism is not helped by the paucity of disclosure on Rasmussen's website about his methods for sample selection, weighting and question wording. To be fair, Scott Rasmussen has been responsive to my requests for more information and MP hopes to look more closely at Rasmussen's methods in the comings months.

Reasons to take these results seriously: First, the initial result -- in which 17% believe the "Dubai Ports World should be allowed to purchase operating rights to several U.S. ports" and 64% disagree and say the sale should not be allowed -- is not close. Yes, this one-sided reaction may change over time. It may look different when measured by a more rigorous sampling methodology. Different wording may get a different result. But it is hard to imagine a complete reversal.

Second, the results that drew the most attention from bloggers -- those noting Bush's surprisingly low rating on "national security issues" -- came from a question asked before any mention of the ports sale issue:

For the first time ever, Americans have a slight preference for Democrats in Congress over the President on national security issues. Forty-three percent (43%) say they trust the Democrats more on this issue today while 41% prefer the President.

It is important to note that the question about trust on national security issues was asked first, before any mention was made of the Dubai Ports issue [Emphasis in original].

I emailed to ask Rasmussen how this results compares to those from previous surveys. His answer:

We asked the specific question frequently in 2003 and early 2004. Occasionally in early 2005. Bush always on top by high single digits to low double digits.

Third, Rasmussen's party weighting procedure makes his samples a few points more Republican than other national surveys of adults. In an email exchange in early January, Rasmussen explained that since 2004, he has weighted his adult samples to match the partisanship of voters on Election Day 2004 as measured by the final adjusted NEP exit polls (37% Republican, 37% Democrat, 26% independent). While an odd choice to weight samples of adults (as opposed to "likely voters"), the effect is to make Rasmussen's samples consistently more Republican than most other public polls of adults. We can debate the wisdom of this procedure, but in this case it argues for taking negative reaction to the Dubai ports sale a bit more seriously.

[And yes, MP has been procrastinating about explaining Rasmussen's weighting scheme in full. Apologies, but I'll get there soon].

We should be patient, as the conventional surveys on this issue will begin appearing over the next week or so. However, Rasmussen's early results suggest that the initial public reaction has been sharply negative. Stay tuned.

February 24, 2006

Gallup's Newport on Real Wiretapping Polls

Let's stay on the subject of the domestic eavesdropping issue, but go back to what real polls have to say about it. Today, Gallup's Frank Newport posted a highly relevant analysis looking at results from a number of different polling organizations, as well as a shorter summary of the same material in their daily video briefing. MP took a similar look at this issue (with the help of the Numbers Guy) a few weeks ago, but Newport does a very systematic analysis of the different ways pollsters have asked about the wiretapping issue. His in-depth written report is definitely worth reading in full, but do so quickly. Like all Gallup reports, it is free for today (and in this case the weekend), but will turn into a subscription-only pumpkin on Monday. The video briefing should remain available to all.

Here is the crux of Newport's analysis:

The fact that arguments for and against the wiretapping program focus on two powerful principles -- defending the nation against terrorism and protecting the privacy rights of individuals -- adds to the complexity of this issue

It is reasonable to assume that when Americans are asked about the wiretapping program, they are sensitive to cues in the question wording that stress one or the other of these dimensions. As a result, different ways of asking about the program could, in theory, quite easily yield different response patterns.

A review of questions asked on wiretapping across a number of polling organizations over the last month or two shows that while there is some variation in public opinion on the issue, it is not as large as might be expected. In other words, despite the newness of the issue, attitudes appear to be generally similar regardless of how the question is asked. The data from a number of different questions suggest that the American public is roughly divided on the wiretapping issue, with the most recent survey results suggesting a slight tilt toward approval of the program.

Those bored with my "Roboscam" obsession are advised to go read Newport's report right now. It's worth the click.

For those of you still reading, consider the implications of Newport's analysis for the ongoing debate (that includes bloggers Gerry Daly and Gordon Fischer and various MP readers) who have been scrutinizing the language used in the recorded calls. One big challenge in the debate is the lack of an audio recording or true verbatim transcript of any of the calls. We have only the recollections of those who were called to go on (again, if you do have a recording captured on an answering machine or know someone who does, please email me).

Most recipients who have described the details remember hearing the wiretapping described as "illegal." Others remember hearing that they were "domestic" or that they involved "American citizens." Some heard that they were done "without a warrant," others are certain they never heard that phrase. Some remember hearing Al Gore's name invoked as a critic, others are certain the calls never mentioned Gore. And one reader who emailed remembers hearing that their Congressman "supports" the Bush wiretaps "for national security purposes."

Now as noted previously, these differences may be due to memory lapses among the recipients or variations in the calls themselves. Obviously, we have no way of knowing.

However, if we assume that all of the calls refer to the wiretaps as "illegal" or done "without a warrant" in one breath, and as done "for national security purposes" in another, then the calls reference both of the "two powerful principles" that Frank Newport referred to. If so, the intent of the calls' language -- regarding only the wiretapping issue -- may be to try to simulate the national political dialogue on wiretapping for the purposes of "data harvesting" rather than to "push" opinion one way or another.

The really big, but sadly ambiguous clue to the Roboscammers's identity is the way they consistently describe the Democratic members of Congress as "supporting" the Bush wiretaps, who in many cases do not. I believe that tells us that one of the motivations (though certainly not the prime motivation) of whomever is doing this is to prompt liberal Democrats to put pressure on their member of Congress to oppose the program. This tells us the calls are coming from either someone on the Left who wants to see more Democrats stand up to the President, or someone on the Right who wants the same thing in order to set up Karl Rove's hoped for debate on wiretapping and to just generally sow discord in the Democratic base.

At this point, I have my guess, yours may be different, but we really cannot know for certain.

February 23, 2006

TPM Cafe-Cross Post: The Roboscam Calls

[Note: MP is cross-posting the following at the site TPM Cafe. It mostly reviews material that will be familiar to regular readers, but does include another update on the current list of members of Congress whose districts have been identified as having received the "Roboscam" calls. The count is now five Democrats and six Republicans]

Over the last few days, on my blog Mystery Pollster, we have been following the story (here and here) I have dubbed "Roboscam." It involves an apparently widespread campaign of automated telephone calls placed into competitive House races that ask just two "questions" about President Bush's domestic wiretapping program. The automated calling scheme falls somewhere between classic push polling and an unethical and possibly illegal effort to collect personal data under the false guise of an opinion poll.

Although reports from recipients have been inconsistent on some details, a common pattern has emerged: Over the last few weeks, someone has placed telephone calls under the guise of an automated survey to voters in at least a dozen competitive House districts. Those who respond hear just two questions that they are instructed to answer by pressing the buttons on their touch tone phones. The first asks whether they support or oppose President Bush's wiretapping program after a statement that names their member of Congress and claims they "support" Bush on the wiretapping issue. Some recipients of the calls say the statement also mentions former Vice President Al Gore's opposition to the wiretapping. The poll then asks whether the respondent plans to support their member of Congress for reelection. After two questions, the "survey" ends.

The relatively few who can remember say the calls initially identify the sponsor as "USA Polling." Although a USA Polling Group does exist (an academic survey call center at the University of South Alabama), an official there tells me they have no ability to place automated calls and have been bombarded with calls and emails from angry recipients of the recorded calls who tracked them down over the Internet. The apparent attempt to impersonate the work of the legitimate automated pollster SurveyUSA has led their president Jay Leve to offer a $1000 reward to the first person who can provide a "clean tape recording from beginning to end."

I have so far logged reports from readers, bloggers and media outlets of eleven members of Congress whose districts have been called. Five are Democrats and six are Republicans (any public reports are linked in parentheses):

One key characteristic of the calls is the consistent claim that the local member of Congress -Democrat and the Republican alike - "supports" Bush's domestic wiretapping program. As such, the calls certainly misstate the positions of both Salazar and Boswell (and possibly others), who have both spoken out against the wiretapping program. As discussed in much greater depth on Mystery Pollster, these calls fit the classic pattern of the so-called "push poll," which is not a poll at all but a fraud that aims to spread a usually false claim under the guise of a survey.

The intentions of those sponsoring the calls are unclear, but in the case of the Democratic districts the goal may be to help spur Democratic partisans who strongly oppose the Bush wiretapping to call their representatives to complain about their alleged "support"' of the program. The call sponsors may be hoping to push Democratic candidates further to the left on wiretapping, to sow dissention in the Democratic ranks, or both. Strident attacks by Democrats on the wiretapping issue would help create the "debate" that Bush advisor Karl Rove signaled he is eager for in a speech to the Republican National Committee last month.

The goal of the calls in the Republican districts is less clear. My own theory is that these calls are part of a widespread effort to "harvest data" on reactions to the wiretapping issue to be appended to voter files for the sort of micro-targeting that is now all the rage among campaign cognoscenti. While a political canvass under the guise of a survey may not be illegal, it is certainly the type of "abuse" that the American Association for Public Opinion Research (AAPOR) has condemned for "exploit[ing] the legitimacy and credibility of the scientific research process."

It is worth remembering, of course, that we do not know for sure who is making these calls. The tactic seems more consistent with the current strategy of the Republicans for 2006 than the Democrats, but I certainly cannot rule out the possibility that some entity on the far Left -- someone who wants to push moderate Democrats to more strident opposition on the wiretapping issue -- may be the sponsor.

Regardless of the motivation, all of the calls appear to violate the federal Telephone Consumer Protection Act (TCPA) of 1991 that requires all automated calls to identify the entity sponsoring the call and provide a working telephone number that recipients can call to request they be placed on a do not call list. None of those who have so far reported receiving the calls remember hearing either a telephone number or the name of the entity actually making the calls (the apparently phony "USA Polling" reference does not count).

So what can we do to fight the Roboscam calls? I am told that either the FCC or a State Attorney General has the power to investigate or file suit over violations of the TCPA. I would imagine any such investigation would have the power to seek a warrant to have the calls traced through telephone company records (oh the irony in that possibility).

Of course, before that can happen those of us in the blogosphere need to help gather the facts and tell the story by linking to this page. I strongly urge anyone who has received similar calls to post a comment, email me with the details, or both.

Roboscam: More from the Real USA Polling Group

Another quick Roboscam update. See the post earlier today for the context, but as a result of my calls to the staff of the real "USA Polling Group" at the University of South Alabama, I received an email this morning from their Associate Director, Keith Nicholls, that will be of interest to those following this story. It appears in full after the jump.

I spoke to Professor Nicholls briefly this morning, and something he said intriguing in regards to the pervious post by SurveyUSA's Jay Leve. Like the Iowa Poll director I quoted on Tuesday, Nicholls' first thought of SurveyUSA when he heard about these calls. "When people first started complaining about being called by an automated poll," he told me, "I assumed it was SurveyUSA." So the concerns that Jay Leve articulated regarding potential damage to his brand are well founded.

Readers will note that Professor Nicholls is far more skeptical of the IVR methodology than MP (see my posts on IVR surveys and my article in POQ). That's another worthy debate for another day. The critical and relevant point to the Roboscam story is this: Pollsters may disagree vehemently among ourselves about which methodologies are best, but we all agree about "push polls" and other sham calls conducted under the guise of research. They abuse respondent confidentiality and the research process, and we deplore them.

Email from Keith Nicholls:

Mark,

Indeed you are correct that the automated calls in question do not come
from our organization. USA Polling Group is a small survey research
center located on the main campus of the University of South Alabama in
Mobile. We conduct person-to-person public opinion polls by telephone
for our local newspaper, the Mobile Register. We consider this a public
service. We also conduct survey research for faculty member and
telephone polls and marketing surveys for outside clients. In no case
do we try to sell anything. We are not telemarketers.

Under no circumstances do we or would we conduct polls using an IVR
system. We do not have an IVR system and we do not believe such systems
have any legitimate role to play in public opinion polling. The results
of such polls typically aren't worth the paper they're not written
on. In actuality, I suspect that the most common use of those systems
involves unethical push-polling.

Certainly we have suffered from the case of mistaken identity. We have
received numerous complaints demanding that we cease and desist. Many
of these complaints are really over the top, hostile and belligerent,
threatening FCC action, lawsuits, etc. We respond to each of these
complaints, assuring the complainant that we are not to blame.

Should you have any further questions, please do not hesitate to
contact me.

Roboscam: More From SurveyUSA

Another quick update. MP received an email this morning from Jay Leve, president of SurveyUSA regarding the "roboscam" phone calls we have been following. Leve wanted to make it clear that the research conducted by SurveyUSA is entirely legal under the Telephone Consumer Protection Act, as his company follows the letter of the law in both properly identifying their calls and providing a number for respondents to use to call back if they wish.

His comments provide some useful background on this emerging story. He includes a link to a complete audio recording of a very similar call captured last July during the 2004 campaign. He also reports on yet another first person account of one of the recent calls. This one comes from a SurveyUSA employee, no less, who lives in the 7th District of New Jersey District, represented by Republican Mike Furgeson.

The full text of Leve's email appears after the jump.

SurveyUSA wants to make these points:

SurveyUSA does not know whether calls are being made by Republican, by Democrats, or by both parties. SurveyUSA takes no political side in the story.

Election lawyer Ezra Reese observes that the calls in question may violate federal law. SurveyUSA wants to put Reese's point in context. Legitimate opinion research, as conducted by SurveyUSA, which a) identifies who is calling and b) in every scenario provides a phone number for the respondent to call us back if he/she has concerns about the research, is not illegal.

An employee of SurveyUSA, who lives in New Jersey's 7th CD, received the following phone call on his home answering machine or about 7/20/05. To my knowledge, this is the only preserved tape recording of the kind of calls in question. This, then, is the smoking gun:

It is very important to take apart this phone call into its component pieces.

This phone call clearly attempts to confuse the recipient into thinking that the call is coming from a polling firm, and specifically my polling firm, SurveyUSA. SurveyUSA is a registered trademark. When SurveyUSA registered the domain www.surveyusa.com back in 1995, we anticipated that others would try to start businesses with confusingly similar names and trade on our reputation. So, just as SurveyUSA registered the ".net" extension and the ".org" extension for SurveyUSA.com, SurveyUSA also registered the domain name www.usasurvey.com (and others similar to it). Therefore, anyone who received this, or any other phone call, from a "USA Survey," and who attempted to locate that organization, would find only my organization, and assume the calls had come from me.

I do not believe the primary purpose of the call was to harm the reputation of SurveyUSA; I believe that SurveyUSA is just collateral damage in this exchange. Nevertheless, the damage to my firm is real and material, and the damage to all opinion researchers is real and material.

Next, this phone call, to the vast majority of those who received it, would in fact sound like a poll. It is not a poll. This message was received in its entirety on my employee's home answering machine. Pollsters do not typically interview answering machines. Answering machines can't participate in a poll. Even if the call were not a poll, but (under the guise of being a poll) had as its primary purpose the gathering of respondent data (whether to "harvest" or "enhance" a database), there would have been a pause after Question 1, to allow the respondent to answer. Because there is no pause whatsoever, it is clear that there is no desire on the calling party's end to collect any information. Quite the opposite. The calling party wants to be certain that Q2 gets laid down on the answering machine in its entirety before the answering machine cuts off. For Q2 is the shiv. (Note: I have no knowledge about whether the buried allegation contained within Q2 is true or false or entirely made-up). Separate note, just to be on the record: As soon as this phone call was first brought to my attention, back in July 2005, I immediately called the Chief of Staff for Congressman Ferguson to make certain that he knew that SurveyUSA was not associated in any way with these phone calls.

Next and more recently: the same SurveyUSA employee, in NJ 7th CD, received another phone call at or about 4:45 pm ET on Saturday 1/28/06. Because the SurveyUSA employee was at home and answered the phone, that call was not preserved on his answering machine and is not tape recorded. However, contemporaneous transcription reveals the following: the call came from "USA Polling." The call asked two questions. Question 1, paraphrased, was "Former Vice President Al Gore has said that President Bush's domestic wiretapping without obtaining warrants is illegal. Do you support President Bush's domestic wiretapping without obtaining warrants?" Yes, press 1. No, press 2. Undecided, press 3. Question 2 paraphrased: "Congressman Mike Ferguson supports President Bush's domestic wiretapping. Knowing that, would you vote to re-elect Congressman Mike Ferguson? Yes, press 1. No press 2. Undecided, press 3. [click].

SurveyUSA has an acute interest in learning who is trying to pass itself off as SurveyUSA. That is a crime against SurveyUSA.

SurveyUSA shares a more general concern that phone calls such as these make it increasingly difficult for all legitimate opinion researchers to secure the trust and cooperation of a respondent. That is a crime against us all.

February 22, 2006

Roboscam Update

Tonight MP can report quite a bit of news on yesterday's "roboscam" post: The calls are probably illegal, readers have emailed with reports of calls received in five more congressional districts and an academic call center named "USA Polling" convincingly denies any involvement.

To review, some unknown entity has been placing a large number of "survey" calls into congressional districts across the country. The calls share a common pattern: All use an automated, recorded technology that asks respondents to enter their responses with the telephone key pad. All ask just two questions. The first asks whether the respondent supports President Bush's wiretapping program after stating that the local member of Congress - whether Republican or Democrat - "supports" Bush's program. The second then asks whether the respondent supports their representative for reelection. Those who can remember say the calls identify "USA Polling" as the sponsor.

Here are the new details:

Ezra Reese, a Democratic election lawyer with the DC firm Perkins Coie, emailed me today to say the calls are probably illegal if because they do not appear to identify a sponsor and contact number as required of all automated calls by regulations associated with Telephone Consumer Protection Act of 1991. Reese sent the text of the relevant FCC regulations (copied below, on the jump). The gist is that any automated call (a) must accurately identify who initiated the call and (b) must provide a phone number at the end so that people may request removal of their names from the caller's lists. None of those who have so far reported receiving the calls has reported hearing any telephone number at the end of the call.

These regulations are distinct from the laws against "sugging" that I discussed yesterday. Also, according to Reese, the content of the calls has no bearing on the identification requirement. Although the FCC regulations expressly exempt non-profit organizations, Reese does not believe that political calls have any such out: "There may be arguments for why this provision doesn't apply to political calls, but I've yet to hear them." [Correction: Reese's point is that while most FCC regulations exempt non-profits, this one does not. All entities -- including non profits -- must comply with the identification requirements for automated calls].

Reese also points out that while the FCC has yet to actively "crack down" on violations of this rule, they could investigate if they receive complaint. He also points out that the statute "includes a civil liability system and even a private right of action to sue the offenders." In other words, a recipient of the calls has the standing to sue, provided they can identify the offender.

As very alert reader RD (aka JunctionDailyBlog) discovered last week, the academic survey center at the University of South Alabama is named the USA Polling Group. I spoke to an official there today who categorically denies making any automated calls. They do all their survey work with live interviewers and have no ability to conduct automated interviews of any kind. They also report having received 20 to 30 angry calls from recipients of the automated calls who went to look for "USA Polling" on the Internet. Their report provides rather strong evidence that the calls are providing a false identification and do not provide a proper phone number as required by FCC regulations.

Based on new reports from alert readers, I can now expand the list of specific districts where calls have been received from two to seven (the source in parentheses):

I would again urge any readers who may have received calls fitting the pattern to please email me with details.

As noted yesterday, I also spoke to a Democratic pollster who requested
anonymity but reported similar calls received in the districts of four more Democratic members of Congress. Although I will say tonight that the four are not listed above,
in order to honor my commitment to that source, I will need to refrain
from further comment on these districts unless a reader in one reports independently having received one of these calls.

Reader RD points out that Democrat John Salazar "has never supported the President's wiretapping program, as the lead-in [question] suggested." A Yesterday post noted the Radio Iowa report that Democrat Leonard Boswell also spoke out in opposition to the program, contrary to the assertion of the call question.

Well, one more comment: All but one of districts either specifically named above or cited by my anonymous source appear on the Cook Political Report's list of competitive races. The exception is Michigan's Mike Rogers.

Not surprisingly, those who report receiving the calls disagree on some of the details. For example, some report hearing a statement about former Vice President Al Gore's opposition to the program, others do not. Whether this discrepancy reflects memory lapses or systematic differences in the calls is anyone's guess. However, I can report that readers and others have reported the Gore references in both Democratic and Republican districts.

The Gore references are certainly intriguing, and more than a few commenters see great significance in them. But this "clue" is ultimately ambiguous. "The stuff about Al Gore," concludes Democratic Iowa blogger Gordon Fischer, "could almost trick someone into thinking it's a Democratic survey." But a loyal Republican reader emails to suggest just that. He believes the mention of Gore rather than another prominent Democrat "sets [Gore] up as the primary foil to (the struggling in the polls) George W. Bush." He believes this is a Gore "push poll" because "it advertises that Gore is leading the opposition." [Update, with his permission I can report that the GOP reader is blogger Gerry Daly of Dalythoughts - see also his comments below].

Finally, two more points worth making in light of some of the comments to yesterday's post. Those of us following the issue need to keep in mind that we really cannot say for certain that we know the political leanings of those who are making the calls. As explained yesterday, the entity involved seems to be conducting not survey research aimed at sampling public opinion, but a large scale effort to "harvest data" under the guise of a survey, presumably to identify voters who appear persuadable on the wiretapping issue. The sponsors are also misstating the positions of Democratic members of Congress in a way that matches the classic "push poll" pattern. By always asserting that the Democratic representatives "support" Bush on wiretapping, they may be trying to pressure moderate Democrats to attack Bush more directly.

Let me say it plainly. The tactic seems to me to be more consistent with the current strategy of the Republicans for 2006 than the Democrats, but I certainly cannot rule out the possibility that someone on the far Left of the political spectrum -- someone who wants to push moderate Democrats to more strident opposition to Bush -- may be the sponsor. We really cannot know for certain.

Second, Mickey Kaus asked a reasonable question: "Why is conducting an anonymous poll so much worse than, say, writing an anonymous blog?" The answer is that anonymous blogs usually present themselves as just that. They do not try to pass themselves off as a well known media brand in order to embellish their credibility. Moreover, anonymous blogs do not typically ask readers to provide personal information that the blogger sells to a database vendor or political campaign. The better analogy here is not the anonymous blog, but the website that secretly deposits spyware on your hard drive.

I concede a pollster's bias on this issue. My sense of outrage comes mostly from the way the "data harvesters" abuse the trust among respondents that a "poll" or "survey" will protect their privacy. This abuse, as AAPOR President Nancy Belden put it last year in an official statement condemning the collection of signatures under the guise of a survey, "exploits the legitimacy and credibility of the scientific research process." Whether initiated by Republicans or Democrats, the "roboscam" tactic now in play is unethical, stupid and probably illegal.

Late update: The Pueblo Chieftan reported last week that Colorado Democrat John Salazar personally received one of the calls:

"I got a call about this late last night and it made me angry," Salazar said
Wednesday. "I don't support the wiretapping program because Congress set up a process for
getting court approval for these wiretaps and that process hasn't been followed."

b) All artificial or prerecorded telephone messages shall: (1) At the beginning of the message, state clearly the identity of the business, individual, or other entity that is responsible for initiating the call. If a business is responsible for initiating the call, the name under which the entity is registered to conduct business with the State Corporation Commission (or comparable regulatory authority) must be stated, and (2) During or after the message, state clearly the telephone number (other than that of the autodialer or prerecorded message player that placed the call) of such business, other entity, or individual. The telephone number provided may not be a 900 number or any other number for which charges exceed local or long distance transmission charges. For telemarketing messages to residential telephone subscribers, such telephone number must permit any individual to make a do-not-call request during regular business hours for the duration of the telemarketing campaign

Numbers Guy: Palestinian Exit Polls

Carl Bialik, the "Numbers Guy" for the Wall Street Journal Online," has a comprehensive review of the Palestinian exit poll snafu now online (that, as always, is free to all).

MP blogged about this issue briefly earlier in the month, but frankly, lacks a good working knowledge of the working of the election or the mechanics of the exit polls conducted there. Bialik, on the other hand, dug deep. Here is his bottom line:

The pollsters had experience and strong track records. So what went wrong? Two of the pollsters, who have been reviewing the results since the election, have different explanations. One suspects that Hamas strategically gamed the polling numbers to suppress Fatah worries. But another blames himself for failing to properly analyze the data, a mistake exacerbated by an unusual election system that magnified small differences between the two leading parties.

February 21, 2006

RoboScam: Not Your Father's Push Poll

The plot thickens. The automated calls we noted Friday received in the New York Congressional District of Republican Congressman John Sweeney (as reported by the Glen Falls, NY Post Star and the Albany Times Union) do not appear to be an isolated incident. Very similar calls have been received in Iowa and at least three other congressional districts held by Democrats that match the pattern of a classic "push poll" dirty trick. Why such calls were also made about a Republican remain unclear, but the answer may be a new high tech development in the inglorious history of political dirty tricks. Details will follow, but for now, let's call it "robo-scam."

First, the latest details: Last Thursday, Iowa attorney, blogger and former Iowa Democratic Party Chair Gordon R. Fischer reported receiving a call very similar to those received in New York (hat tip to alert MP reader Drew Miller). There were jut two questions asked by an automated recorded voice. He took notes of the call and posted an approximate transcript online (Fischer tells me by email that his transcript is "close to verbatim, but they did talk fast." He estimates it as "75% accurate"):

"Here is the second -- and last -- question. Do you support the re election of Congressman Leonard Boswell?

If you support Congressman Boswell's re election, press 1.

If you do not support Congressman Boswell's re election, press 2."

[I pressed 1.]

Thank you for your time." CLICK.

On Friday, Fischer posted an update reporting that he had been "swamped by folks who received the exact same call."

MP was also contacted by a Democratic pollster who requested anonymity but reported that similar calls have been received in the districts of at least four Democratic members of Congress in addition to Boswell. All of the calls involved automated surveys that asked only two "questions:" One asked about President Bush's NSA wiretapping program after mentioning that the member of Congress supported it. The second asked whether the respondent would support the member's re-election.

Memories were hazy on the stated identity of the pollster. Here is Fischer's report, again via email:

Unfortunately, [I could remember] nothing -- it was very fast. And the "identifier" was only at the beginning of the call. I think it was something generic like, American Opinion, or Acme Polls, or something.

MP also heard independently about these calls from two pollsters in Iowa who both recieved similar calls. One was J. Ann Selzer whose company regularly conducts the "Iowa Poll" for the Des Moines Register. She received an automated poll call, but hung up before listening to the questions. However, she definitely remembers hearing that the survey would involve just two questions and it was being conducted by "USA Polling." She explains via email:

I definitely heard USA Polling and of course because I'm in the industry and I know Jay [Leve of SurveyUSA], I made particular note of the name, wondering if this was a spinoff of his enterprise. Because I am associated with a prominent local poll (The Des Moines Register's Iowa Poll), I do not respond to polls of a political nature in order to maintain my independent status.

These reports are consistent with those from the calls into John Sweeney's district in New York. The Albany Times Union reported one recipient who "recalled hearing something about 'USA,' and thought perhaps USA Today was conducting the poll." Another recipient "thought she heard 'USA' mentioned on the call as well."

The second Iowa pollster is now retired and emailed MP to say he too had received an automated survey call that asked two questions, one about Bush's wiretapping program and a second about whether he would vote for Boswell. He did not remember any reference to Boswell's support for the wiretapping program (and so did not consider the call a "push poll"). He also remembered the pollster being identified at the beginning of the call as "SurveyUSA." [Update: With his permission, I can report that the retired pollster referenced above was Glenn H. Roberts, former and long-time head of the Iowa Poll at the Des Moines Register].

I contacted Jay Leve of SurveyUSA who confirms that his company has not fielded any surveys in Iowa during 2006 about either Congressman Boswell or the NSA wiretapping program. They did conduct interviews in Iowa from February 10 to 12 as part of their 50-state tracking program, but again, that survey asked no questions about either wiretapping or Boswell. However, as Leve points out, the apparent confusion of the retired pollster,

just confirms anew how invidious this is for my organization, (that somebody IN the business could confuse it with SurveyUSA), and then more generally, it speaks to the destructive power of the phone calls themselves (as they should concern all public opinion pollsters, who need trust and cooperation from respondents.)

So here are the facts now in the public domain: Automated "poll" calls were made into at two congressional districts, one represented by a Democrat, one by a Republican. In each case, the poll involved just two questions, one about Bush's wiretapping program and one about support for the incumbent member of Congress. Most of those called (including a lawyer taking notes) remember hearing that the member of Congress "supports President Bush's wiretapping program." MP has heard from a Democratic pollster who prefers to remain anonymous that very similar calls have been received in recent weeks in at least four more districts represented by Democrats.

As I will explain in an update, the calls into the Democratic districts fit the classical pattern of the dirty trick "push poll," albeit delivered by an automated recording. The puzzling part involves the calls made into the district of Republican John Sweeney of NY, for reasons that require more explanation, do not fit the usual "push poll" modus operandi. MP has a theory on that one, which I will explain in an update later today.

UPDATE: So what is going on here? To try to connect the dots, a little explanation is in order.

Let's start with the definition of a "push poll." Many organizations
have posted definitions (AAPOR, NCPP, CMOR, CBS News, Campaigns and
Elections, Wikipedia), but the important thing to remember is that a
"push poll" is not a poll at all. It's a fraud, an attempt to
disseminate information under the guise of a legitimate survey. The
proof is in the intent of the person doing it.

To understand what I mean, imagine for a moment that you are an
ethically challenged political operative ready to play the hardest of
hardball. Perhaps you want to spread an untruth about an opponent or
"rumor" so salacious or farfetched that you dare not spread it yourself
(such as the classic lie about John McCain's supposed "illegitimate
black child"). Or perhaps your opponent has taken a "moderate" position
consistent with that of your boss, but likely to inflame the opponent's
base (such as Republican voting to raise taxes or a Democrat supporting
"Bush's wiretapping program").

You want to spread the rumor or exploit the issue without leaving
fingerprints. So you hire a telemarketer to make phone calls that
pretend to be a political poll. You "ask" only a question or two aimed
at spreading the rumor (example: "would you be more or less likely to
support John McCain if you knew he had fathered an illegitimate child
who was black?"). You want to make as many calls as quickly as
possible, so you do not bother with the time consuming tasks performed
by most real pollsters, such as asking a lot of questions or asking to
speak to a specific or random individual within the household.

Again, the proof is in the intent: If the sponsor intends to
communicate a message to as many voters as possible rather than measure
opinions or test messages among a sample of voters, it qualifies as a
"push poll."

We can usually identify a true push poll by a few characteristics
that serve as evidence of that intent. "Push pollsters" (and MP hates
that term) aim to reach as many voters as possible, so they typically
make tens or even hundreds of thousands of calls. Real surveys usually
attempt to interview only a few hundred or perhaps a few thousand
respondents (though not always). Push polls typically ask just a
question or two, while real surveys are almost always much longer and
typically conclude with demographic questions about the respondent
(such as age, race, education, income). The information presented in a
true push poll is usually false or highly distorted, but not always. A
call made for the purposes of disseminating information under the guise
of survey is still a fraud - and thus still a "push poll" - even if the
facts of the "questions" are technically true or defensible.

So let's start with the calls to Iowa and the other districts
represented by Democrats. Please keep in mind that this is a blog and
the judgments that follow reflect MPs opinion. Obviously, we cannot
know for certain the motives of those who placed the calls, but they
have all the hallmarks of a classic "push poll" dirty trick.

One big clue comes from the length. The calls asked just two
questions and included no demographic items. Another is the nature of
the first question. It is hard to imagine a real pollster phrasing a
question like the Gordon Fischer describes whether testing current
opinions or reactions to some potential message. Even if the confluence
of Gore's opposition plus the Congressman's support added up so some
sort of message worth testing, a real pollster would first ask about
support for the wiretapping program absent the "message" in order to
gauge its effect.

Probably the most important clue comes from the claim that the
Democrats in question "support" the president on wiretapping. In at
least one case reported to MP, the Democrat has not yet made a public
statement on the Bush wiretaps. In the case of Leonard Boswell, Radio
Iowa reported just last Friday that he criticized the wiretapping program
during the taping of a Public Television program (I am told that it typically tapes on Thursday). While it may exist, I cannot find any other statement online
by Boswell on this subject. So the possibility exists that he spoke out after most of the "USA Polling" calls were made into his
district.

Why does that matter? At very least, the calls misrepresent Boswell's current position. However, consider these calls in the context of Karl
Rove's recent speech to the Republican National Committee in which he
signaled his intent to make disagreements over the NSA surveillance
program a centerpiece of the Republican campaign:

Let me be as clear as I can: President Bush believes if al Qaeda is
calling somebody in America, it is in our national security interest to
know who they're calling and why. Some important Democrats clearly
disagree. This is an issue worthy of a public debate.

So again my speculation: Someone out there would like to see Rove's
hoped for "debate" occur in as many Congressional races as possible. So
they are making thousands of calls into Districts held by moderate
Democrats spreading the rumor that those officeholders support Bush on
wiretapping. They know that Democratic partisans (like this one) will
be outraged and put pressure on their representatives to harshly
criticize Bush on wiretapping. If the members respond to the pressure,
the dirty tricksters get the debate they hoped for. If not, the
Democrats are forced to put out fires ignited by the push poll in their
base.

OK, you say, that may explain the calls to districts held by
Democrats, but what about the calls into the district of Republican
John Sweeney stating that Sweeney "supports" the president on
wiretapping? What possible motive could those calls have?
MP's admittedly speculative theory requires some explanation of another
form of fraudulent survey: Those that involve political canvassing
under the guise of a survey.

"Canvassing" is a political activity once performed mostly by volunteers.
They would knock on doors or call voters on the phone, identify their
affiliation with a candidate and then ask whethrer the voters planned to support their candidate. Supporters might get a "get-out-the -vote" (GOTV) reminder call
on Election Day. Undecided voters might receive a
follow-up mailings to win them over.

One problem with the honest approach to canvassing is that many
voters choose to hide their true intentions from partisan volunteers.
Over the years, as political campaigns came to rely more on paid
telemarketing firms to conduct canvassing, the purveyors discovered
that they would get fewer "undecided" responses by pretending to conduct a poll.
Although the paid canvassers never make an explicit promise to keep the responses
confidential, their failure to reveal their true identity combined with the use of the terms "survey" or "poll" convey the same
implicit message. That is why it works.

Survey researchers have a name for this particular fraud when used
to sell products: "sugging," an acronym for "selling under
the guise" of research. Sugging not only violates the codes of organizations like AAPOR and CASRO, it is also now prohibited
by federal law.

The telemarketing laws that outlaw "sugging" do not apply to
explicitly political activity such as push polls or canvassing, in part
because of the constitutional protections of political speech. But
legal or not, canvassing under the guise of a survey is an ethical
breach that exploits the credibility of legitimate polls that protect
respondent confidentiality. Pollsters may disagree among themselves
about methodology, but nearly all agree that protecting respondent
confidentiality is sacrosanct. Canvassing under the guise of a survey
makes a mockery of that principle.

Unfortunately, the latest technical innovations in automated polling
make canvassing under the guise of polling even cheaper and easier than
in the past. Now campaigns can call every voter in a district or state
in a matter of days. And in a climate where the use of "high tech
micro-targeting" based on commercial data appended to voter files is
all the rage, such activity may be proliferating.

MP's theory is that some or perhaps all of the "USA Polling" calls
discussed in this post may be part of a such a "data harvesting"
scheme. Again, this is pure speculation, but more specifically:

In
some districts, such as the John Sweeney's in New York, the calls are
intended only to identify individual voters who approve of his
presumed support for the wiretapping program.

In other districts
the sponsors get a two-fer: They can identify voters who approve of the
Bush wiretapping plan while also using the classic "push poll" tactic
to sow discord in the Democratic base.

Either way, these calls look like a sleazy, unethical
program of unusual scope. Calling it a mere "push poll" seems
inadequate. So henceforth, let's give it a name more fitting of its
high tech origins:

Roboscam.

PS: The scammers rely largely on stealth. They do their business in such a subtle way that is is almost subliminal. Those who do suspect foul play usually have nowhere to go with their concerns. If those of us who
object want to do something about it, we need to use the power of the Internet to pool what we know. So I hope that bloggers who read this
message will link to it and urge anyone who has received similar calls
to email me with the details. Big points (and possibly a reward) to
anyone who can capture an audio recording of one of these calls or any details via caller ID.

"Professional pollster Mark Blumenthal started Mystery Pollster to provide better interpretation of polling results and methodology... offers much needed help to Political Wire readers" - Political Wire