Thursday, September 16, 2004

The poll numbers are all over the place these days. Some polls show Bush ahead by 8 points or so. Most other polls show Bush and Kerry tied or at least within the margin of error. There are even some polls in the last week that show Kerry ahead among "registered voters."

This has lead to a kind of "my poll can beat up your poll" argument among prognosticators both amateur and professional. This leads to a series of questions:

1) Are all polls equal? Answer: No.

Some polls are much more carefully done than others. A friend of mine at the University of Miami (Fla) has told me some hair raising tales about what passes for a usable poll for the Miami Herald. He told me he's seen their raw data and its a joke. They just make up the margin error some times. They hire professors from local community colleges to do the polls even though these folks may not have the training, experience or staff to conduct an unbiased poll. My advice would be to very suspicious of any poll done by a newspaper without a national reputation. NYT, WSJ, Washington Post and maybe the LA Times are probably OK but don't read to much into a poll by the Indianapolis Star for example.

Also, some polls are "push polls." In this age of image dominated TV journalism, polls can BE the story. That means that both parties have incentive to show their guy ahead as much as they can. Polling groups with ties to one party or the other will give polls that intentionally push a particular result: For example they'll ask a bunch of questions about corporate corruption, Haliburton, Enron etc and then ask if you have a favorable opinion of Dick Cheney. Or they'll ask a lot of questions about protecting Americans from terrorism and defending "the Homeland" and then ask you if you intend to vote for Bush. Strategic Vision is the most commonly reported poll that has a clear tie to the Republican Party. Their results have differed from independent polls by as much as 10 percentage points - always in Bush's favor.

Finally, there are polls that fiddle with the sample. Polls are presented as "random samples" but they aren't really random. What they are is "normalized" which is a lot different. Polls actually have quotas for Republicans, Democrats, Independents, women, men etc. The recent Time poll that showed Bush ahead by 12 points had 10% more Republicans in their sample than there are registered Republicans nation wide. A number of pollsters criticized this poll for biasing the results.

2) What is a "Likely Voter?" Answer: Depends on the poll.

Usually, pollsters give screening questions about past voting history to determine if a respondent is a likely voter or just a registered voter. But there is little confidence that the screening really identifies people that are going to vote in the next election. Usually Republican candidates do better among likely voters than among registered voters. Here are some demographic characteristics that political scientists have identified with higher voter turnout (note: not a prediction of individual behavior as in the pollsters' "likely voter" terminology):
The higher the age group the higher the turnout.
Turnout is higher among men than women.
Turnout is higher among Whites than non-Whites.
Turnout is higher among higher income brackets.
So what does this mean? Well, it generally means that when turnout is low, the share of the votes cast by older, White, rich men goes up (guess which party they vote for). When turnout is high, the share of the votes cast by this group goes down. Most of the polls for this election say that turnout is going to be unusually high in 2004. Latino, Black, overseas and youth turnout are all expected to be MUCH higher than in the past. This could be interesting because these are all groups that will be more likely to vote for Kerry but will be unlikely to be included in "likely voter" polls. Indeed, overseas voters aren't counted at all. Remember how I said some polls of "registered voters" show Kerry slightly ahead? This could explain that result.

3) Can any one poll tell us anything? Answer: Not really.
You shouldn't look at just one poll and declare that you've learned anything with any precision. You have to dig a little and find out as much detail about as many polls as possible. Look at what kind of voters are supporting each candidate. Look at the trends over time. Check to see if a poll you're looking at is consistently more favorable to one candidate or the other.

5 comments:

This is where the "normalizing" comes in. Zogby and everyone knows their samples are biased so they try to correct those biases by setting quotas for various types of voters. But they don't tell you what the quotas were. So you have to rely on the reputation of the pollster and trust them. In the last several elections Zogby has been the best pollster. Gallup used to be the best but has been slipping for several election cycles now. Zogby is the official pollster for the Wall Street Journal. Gallup is co-conducted with CNN and USA Today (fluff journalism) so you know they've slipped. Gallup has gone from being the top pollster in the country to being the official polling operation of Airport waiting areas.

This reminds us that good information is not free. Also, the media is only interested in covering (and having) a horse race with ups and downs. They aren't doing serious research. To understand the electorate right now requires reviewing all the current polls, understanding their limitations, and comparing them with polls over the past several months, in addition to state-by-state polling. Because it's their bread and butter, the Kerry camp and Bush camp are doing this. In an internet age, we can too.

Okay, so the polls "normalize" to set quotas for different types of voters. But the point is that they aren't including ANY voters without a land line! Therefore their quota is zero.

No problem, you might say, this is taken care of in the system. I'm not so sure. It is unlikely that the only thing they have in common is that they are totally cell phone-reliant. What if you had a poll that left out blacks? Or senior citizens? Soccer moms? Would you trust such a poll? You can't argue that such a group just votes like the general population, so it's okay to leave them out. My suspicion is that it's the same situation with the no-land-liners (is there a word for these people? We need to invent one).

I'm not a pollster so I don't KNOW but I suspect that the way they do it is the have very good surveys done by academics over many years that show what the proportions of Republicans, Democrats, indpendents, Blacks, Whites, Men, Women, etc are in the pool of registered voters. They use those numbers to set their quotas.

This is still a problem but less serious than the one Bell Curve points out if the thing that totally wireless people have in common is one of those demographics. The problem would be worse if the thing the totally wireless have in common is something that is NOT accounted for in the normalization...That is probably the case.

Totally wireless people are probably three things: 1) residents of major urban areas 2) more likely to vote for Democrats and 3) more likely to be under 35 years old. The first two things are probably accounted for in the polls' quotas. The second thing though is not only not accounted for it is actually filtered OUT by the "likely voter" restriction. If the totally wireless crowd is going to vote in greater numbers than other young people, not counting them could be a substantial source of pro-Bush bias in the polls.