How Today’s Political Polling Works

So far this year, pollsters have put out more than 400 national Presidential general election polls, an average of about two per day; if the 2012 election is any guide, pollsters will be releasing as many as four per day as Election Day nears. While these poll releases are the subject of intense media interest, that interest is focused on the results of the polls, with much less attention given to how the polls are actually carried out. Political polling is one of the few areas in American life which has survived basically unchanged since the advent of the Internet, and while that may be about to change, it’s not at all clear that it would be a good thing.

Before the widespread use of caller ID, telephone polls were much easier to carry out. In 1997, a national sample of about 800 respondents required between 2,000 and 2,500 calls. Today, getting that same number of respondents requires between 7,500 and 9,000 calls to get a reasonably sized sample, a precipitous decline in what’s called the response rate, which is seen as a crisis in the industry. This is a problem because of the increased cost, but also because it could mean that the sample doesn’t actually reflect the group that the poll is trying to measure.

It may seems strange to assert that a poll can measure the preferences of an entire country by talking to as few as 800 people, but the math works if those 800 people are selected through a truly random process. Think about it this way: a certain proportion of Americans currently intend to vote for Donald Trump in the upcoming Presidential election: we can’t directly observe it, but we know it’s there. Polling is how we indirectly estimate that proportion. Over the course of the poll, you might, by pure chance, call a cluster of people who all intend to vote for Trump, but so long as the sample is random, they’ll be averaged out by those who don’t intend to do so, and, as the sample gets bigger, the average of the sample gets closer and closer to the true proportion. The problem comes if the sample isn’t really random. After what was perceived as a disastrous first debate for Barack Obama against Mitt Romney in 2012, Democratic voters became less likely to answer surveys, as they just didn’t want to talk about politics, while newly enthusiastic Republicans did. As a results, poll results shifted towards Romney, even though his actual support didn’t increase: it’s just that the samples were biased towards Romney voters. If the sample is bad, the results follow.

To achieve that randomness, telephone polls generally make use of a random-digit dial system, in which a computer uses a pre-determined set of area codes and exchanges (the middle three digits, which identify the service provider and area), and randomly generates the last four digits. If trying to reach a particular population, like a racial or occupational group, researchers can purchase lists of numbers, then randomly pick from that list. Of course, since about half of American households now only use cell phones, pollsters have to include cell phones in the sample, despite the fact that federal regulations require cell numbers to be dialed manually, rather than with an autodialer, dramatically increasing the cost of calling. Pollsters have also had to increase the number of times that they call numbers back, trying to get a completion: seven or more attempts are now standard in the industry. Despite these efforts to achieve a random sample, though, response rates remain shockingly low, especially among younger people, Spanish speakers, Evangelicals, and African-Americans.

Low response rates aren’t necessarily a problem, so long as the people who pick up their phones aren’t different in some way from those who don’t. But as the response rates have dropped, to the point where only about 10% of the calls actually end in an interview, it’s become harder to assert that the people who answer the phone aren’t somehow different from those who don’t. If they are different, then the sample is likely to be biased, and the results of the poll wrong.

To correct for this sort of bias in the samples, pollsters make use of weighting. In the simplest form, a pollster might find that only 6% of the sample is African-American, compared to 12% of the American public. As such, a pollster might use weighting to, in effect, count African-American response twice towards the overall results. When weighting is done this way, based on known demographic factors like race, age and gender, it’s not too problematic, but it’s still as much art as science. For one, pollsters can’t be sure that the members of a group they’ve reached are representative of the group generally. For instance, most political polls survey only in English, to avoid thorny translation issues and the higher salaries of bilingual interviewers, but 73% of Latinos speak Spanish at home, so the sample of Latinos reached by most surveys isn’t representative of all Latinos, even if the overall sample is generally good.

It’s also a bad idea for any group to be upweighted or downweighted too much, but how much is too much? Young African-American males are usually the hardest demographic to reach in political polling, and any that happen to be in the sample are likely to be upweighted because of their race, their gender, and their age. But since the weights are cumulative, that one person could respresent as much as half a percent of the overall results, potentially letting a few people throw off the entire sample.

There’s also not complete agreement about what factors should weighted: if a poll has a low proportion of Democrats, or Republicans, should weighting be used to correct for it? Decisions like this give some pollsters the opportunity to push their results one way or the other, for partisan purposes, or to avoid being too far from what other polls are saying. As polling averages have become more prevalent, some pollsters have become nervous about putting out results that are too far from that average, leading them to weight strategically to get their data back towards the mean, or, in some cases, to choose not to release results that look weird. As a result, the polls, in the aggregate, can miss shifts in public opinion.

The best telephone interviewers are highly experienced and college educated, and paying them is the main cost of political surveys. It’s no surprise, then, that there have been attempts to automate the process in order to save money. The most common form of this is Interactive Voice Response (IVR) polling, in which the live interviewers are replaced with recorded prompts, and respondents give answers by speaking to the computer. These services make polling much faster and cheaper: they can complete a survey in hours, and charge about 1/10th the cost of live interviewers. However, as anyone who’s called a customer service line can attest, voice response systems are far from foolproof. They also have even lower response rates than traditional phone sampling, seem to encourage more false responses, and cannot legally reach cell phones. IVR may work for populations of older, whiter voters with landlines, such as in some Republican primary races, but they’re not generally useful.

Online polls have presented another cheap, fast alternative to live caller polls, but they still face enormous challenges. The biggest is that 16% of Americans don’t use the internet, requiring an additional layer of weighting to try and get close to a representative sample of the public. The best online polls get around this by contacting a sample via mail or phone, then providing internet service to anyone in the sample who doesn’t already have it. This helps, but it drives up the cost and requires even more sophisticated weighting to correct for the fact that samples are being taken of samples. Recent studies show that even after weighting, online polls tend to overrepresent men and the unemployed, perhaps because online surveys generally require people to opt in, and unemployed men may be more motivated to share their views and more likely to have the time to do the surveys. This may help explain why Donald Trump seems to do better in online polls than in telephone polls.

While live caller telephone polls face enormous challenges, they still seem to provide more accurate results than the alternatives, and perhaps more importantly, many major media outlets don’t yet consider other techniques reliable enough to report on. This is important, because public polling is generally done for publicity, so pollsters are incentivized to do good work in order to get more media coverage. Media organizations trust live caller polls over other methods largely because their cost and difficulty means that the groups doing them have a bigger incentive to produce high quality results and get things right.

However, if low response rates mean that live caller polls stop working, the media and the public will no longer be able to easily tell good polls from bad ones. There’s already a surfeit of low-quality IVR polls flooding the market, and if they’re seen as being just as good as the expensive live caller polls, and are treated the same way by the media and the public, the incentive to do good polling will go away, and the social benefits good polling provides will go along with it. In the Presidential election taking place four years from now, it’s likely that there will be even more polls released than there will be this year, but it’s not at all clear how many of them will actually be any good.

Dan Cassino is an associate professor of political science at Fairleigh Dickinson University, researching public opinion and political psychology. His new book, Fox News and American Politics, will be released at the end of April.