Articles and Analysis

Ever since Election Night this past November, I have been ruminating about a story that I've been meaning to post. Its connection to pollsters and poll methodology is indirect, but seems especially relevant right now as we formally turn the page from the incredible election year 2008 to a new administration and its travails in 2009. So before we get back into our usual routine, I want to share it.

In May 2002, I sat down to draft a questionnaire for a new client, a wealthy businessman who was then considering a challenge to an incumbent U.S. Senator two years hence. It was just six years ago, back in the days before my MysteryPollster blog, back when conducting such surveys for Democratic candidates was my full-time occupation. And, as it turned out, this client would be my last "major" race (a designation that, in my mind, falls somewhere between a contest with an uncertain outcome and one with some national significance).

My client's eventual campaign struck some as a microcosm of conventional (and thus flawed) politics. One of the consultants for one of our opponents called our campaign "an exercise in the technology of politics...They do polls and see what people want to hear and then they put ads on the air and run them again and again. There's nothing new about what they're doing." There was truth in this critique, and the ultimate demise of this campaign played a not insignificant role in my evolution from consultant to blogger.

But it was something that happened at the very beginning of that campaign that sticks in my memory now. On that day in May, I started with my usual process when drafting a benchmark "message testing" survey. I took all of my notes from initial discussions with the client, suggestions from other consultants and information gleaned from news clippings and the Internet and synthesized it into a single document listing all of the "messages" -- candidate profiles and arguments about them pro and con -- we hoped to test on the benchmark survey.

I found my "notes for questionnaire" for that project on my hard drive yesterday, still there after moving with me through at least three different computers over the last six years. My first cut consisted of roughly 700 words. About half involved the incumbent Republican, about a third concerned our client. I had also typed a two sentence profile of the potential primary opponent, a former Democratic Senator that had been turned out of office a few years before.

Finally, the document also included the names -- just the names --of three potential candidates that most everyone considered extreme long-shots. The paucity of detail on these three spoke to their status as very long shots in this particular Senate race. Anyone looking over my notes right now, however, would immediately notice one now familiar name among the also-rans:

"Barack Obama."

The survey we fielded a few weeks later showed that just 4% of likely general election voters had a favorable impression of Obama, while 5% rated him unfavorably. Nearly four out of five (79%) had never heard of Obama. He did a little better among likely Democratic primary voters -- 11% favorable, 6% unfavorable -- but still ran far behind, winning just 6% of the vote, in a five-candidate primary matchup that also featured former Senator Carol Moseley Braun. Obama did a little better (rising to 12%) when we omitted Braun, but Obama still ran far behind state Comptroller Dan Hynes (with 34%).

Of course, my client, Chicago businessman Blair Hull, started with even less recognition and support (2% of the vote, to be precise), but his candidacy and its ultimate demise was another story altogether. The consultant who dismissed our campaign as "nothing new" was a truly forward looking guy named David Axelrod. But I digress.

It is still hard to believe that a State Senator who seemed like the longest of long shots for the U.S. Senate so recently is about to be sworn in as the 44th President of the United States. I look back at my notes from six years ago, ponder all that has happened since and just shake my head in wonder.

This story teaches many lessons, of course, but the most relevant to readers of this site is that no poll or statistical model could have predicted Obama's ascent. Yes, we could see from the "internals" of that first survey that Obama had the potential to be a formidable contender in the 2004 Senate primary, especially after both Braun and incumbent Peter Fitzgerald announced that they would not run. But no survey or model in 2002 could have predicted Obama's 52.8% majority in the seven-candidate 2004 Senate primary or the nearly 70% of the vote he won in the general election, much less all that followed in bid for the White House in 2008. I'm certainly a believer in opinion surveys and statistical models, but they have their limits.

The Obama story from 2004 provides no direct corollary to the ongoing arguments about the "electability" of potential Senate candidates in Illinois, New York or elsewhere, but I do see a warning against leaping to conclusions about a candidate's prospects based only on early favorable ratings or horse-race numbers. "Electability" is an appropriate topic whenever a party selects its nominee (or appoints someone to serve out the term of a departing legislator), but quantifying electability through polling, or through predictive models derived from it, is a shaky enterprise at best. Obama's rise -- in the face of early data that led many to question his electoral potential in both 2004 and 2008 -- is a testament to such challenges (see also this pertinent example from September 2007).

This story also says something bigger about the potential for "change" within the American electoral system. Typically the day-to-day business of politics is mundane and static. We fight the same fights over and over, and little seems to change. Politics seems to be mostly about compromise, mostly "the art of the do-able." Yet every once in awhile, usually in the context of a presidential election, some out-of-the-blue candidacy reshapes our perception of what is possible. It shows us that sometimes, if we're lucky, we get to see history being made in the midst of mere electoral politics.

I look back to my own experience in the 2004 Senate election and realize that there was a silver lining in an unsatisfying campaign that amounted to, admittedly, little more than "an exercise in the technology of politics:" It offered me the opportunity to witness (and, yes, literally chart) one of the most meteoric success stories in American political history from the very beginning. And from this perch at Pollster.com, I got the chance to keep that ringside seat, charting and writing about the most exciting campaign of my lifetime.

So I want to thank Pollster.com's readers for coming along with us for the ride, and especially those who have stayed now that things have calmed own a bit. We hope you will stick around and let your friends know as we continue to track public opinion in 2009. You will be seeing a transition in Pollster.com over the next month or so, as we move the 2008 data off the front page and begin to feature charts and data that track the performance of the new administration and the concerns of American voters. But as we do, let's not lose sight of an idea we have tried to stress here from our first post: Using survey data well requires that we understand its limitations.

Comments

Basil:

Thanks Mark. I'm interested to see what you come up with. The '08 campaign will be seen as a watershed event in many dimensions, and you (and 538 and RCP) were there in a role that had never existed before. I had the sense that the Pollster site's following evolved (or at least changed) rapidly as it became better known, and that several of the more technically-minded posters withdrew as the site became more of a venue for advocacy as opposed to analysis. Still, with a few exceptions, the tone was generally one of enthusiastic civility and intellectual curiosity rather than the mere ranting that most sites devolve into. You are to be commended for that.

Polls conducted two years out can not predict "electibility”. Here is summary of what our Chicago Tribune polls showed that began only three months before that March 17 primary.

In January it was a 4-way race between Pappas, Obama, Hynes and Hull, ranging from 10% to 14%, with a plurality of voters undecided. Three other candidates were also in the race. Familiarity with those candidates ranged from 31% for Obama to 58% for Pappas statewide. A heavy media buy by big-spender Hull put him out in front as of Feb. 11-17 with 24% of the vote but 34% were still undecided.

After then, Illinois was treated to one of the more bizarre incidents in Illinois politics. Details of Hull’s divorce included an order of protection for his ex-wife. His defense in print ads included this “I only hit her once”, obviously not the kind of advice one would get from a campaign consultant.

In our final poll March 3-6, about two weeks out from the primary, Obama was out in front – 33% to Hull’s 16%. Hull was known to 94%, but opinion of him was decidedly mixed as expected. Obama was known by only 75% with a positive opinion ratio of 47%/4%. He was known by only half of Illinois Democratic primary voters outside the 6-county Chicago metro area which we call “downstate”, an area that accounts for 24.0%-27.6% in the last three Democratic primaries. We should have conducted a fourth poll.

So there you have it. An open seat election with relatively unknown candidates, one of whom self-destructed after holding the lead for a little while. I think our polls did well.

Polls conducted two years out can not predict "electibility”. Here is summary of what our Chicago Tribune polls showed that began only three months before that March 17 primary.

In January it was a 4-way race between Pappas, Obama, Hynes and Hull, ranging from 10% to 14%, with a plurality of voters undecided. Three other candidates were also in the race. Familiarity with those candidates ranged from 31% for Obama to 58% for Pappas statewide. A heavy media buy by big-spender Hull put him out in front as of Feb. 11-17 with 24% of the vote but 34% were still undecided.

After then, Illinois was treated to one of the more bizarre incidents in Illinois politics. Details of Hull’s divorce included an order of protection for his ex-wife. His defense in print ads included this “I only hit her once”, obviously not the kind of advice one would get from a campaign consultant.

In our final poll March 3-6, about two weeks out from the primary, Obama was out in front – 33% to Hull’s 16%. Hull was known to 94%, but opinion of him was decidedly mixed as expected. Obama was known by only 75% with a positive opinion ratio of 47%/4%. He was known by only half of Illinois Democratic primary voters outside the 6-county Chicago metro area which we call “downstate”, an area that accounts for 24.0%-27.6% in the last three Democratic primaries. We should have conducted a fourth poll.

So there you have it. An open seat election with relatively unknown candidates, one of whom self-destructed after holding the lead for a little while. I think our polls did well.

Post a comment

Name:

Email Address:

URL:

Comments: (you may use HTML tags for style)

Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.