Forensic Polling Analysis

By CHRISTOPHER SHEA

Published: December 13, 2009

The American Association for Public Opinion Research censured a Georgia-based firm called Strategic Vision L.L.C. in September for failing to reveal information about how it conducted its polls during the 2008 presidential race. The company's chief executive promptly threatened to sue, which struck Nate Silver, a polling specialist and political blogger, as a bizarre response.

Wondering if the company had anything to hide, Silver, the proprietor of fivethirtyeight.com, stayed up all night keying all of Strategic Vision's poll results over the last four years into a Microsoft Excel spreadsheet.

To test the polls, Silver made use of a statistical truism. As he puts it, ''Tell a human to come up with a set of random numbers, and they will be surprisingly inept at trying to do so.'' They unwittingly fall into nonrandom patterns.

Silver took the results of every Strategic Vision poll question -- from more than 100 polls on political races and issues of every sort -- and analyzed the ''trailing digits'' in the results. (If a poll found that one candidate led another by 52-48, the trailing digits were 2 and 8.) Silver thought that, given the wide range of poll topics, the distribution of trailing digits should be more or less random. Instead -- shades of ''C.S.I.'' -- he found a highly abnormal distribution of digits. For example, there were nearly 60 percent more 8s than 2s. The probability of such a distribution occurring in authentic polls, Silver calculated, was ''millions to one against.''

Silver concluded that the firm's data were not random. ''It's not close to random,'' he wrote. ''It's not close to close.''

When readers asked for a comparison study, he presented a similar analysis for the well-respected Quinnipiac poll. In that case, there were ''a few too many 2s and 3s,'' but nothing outside the realm of chance.

In the coup de gr?, a retired physics professor at the University of Illinois, Michael Weissman, stepped in, deploying more sophisticated tools (Fourier analysis). If Strategic Vision's polls were legitimate, Weissman concluded, the odds that they would produce the numbers Strategic Vision published were 1 in 5,000 -- better than Silver found, but still suspicious.

Strategic Vision has threatened to sue Silver, too, but the company has yet to release documentation of its methods.