Silver Linings and the Statistical Playbook

We asked historian of science Christopher J. Phillips, an expert on quantification in American public life, to reflect on the role of statistics—and Nate Silver—in the coverage of the 2012 election. He was kind enough to write us the following guest post; you can find out more about his work here.The 2012 election was a "Moneyball Election" and Nate Silver its big winner. Or so proclaimedthe New Yorker's Adam Gopnik. He was certainly not alone. Deadspin's David Roher lamented the "braying idiots"detracting from Silver's well-deserved limelight; President Obama jokingly praised Silverfor having "nailed" the prediction of this year's Thanksgiving Turkey; and Wired's Angela Watercutter perhaps gave the ultimate complimentby calling Silver a "Nerdy Chuck Norris." Silver, for anyone who has spent the last few years under a rock, is the creator of the (mostly) political blog FiveThirtyEight. Picked up by the "New York Times" just before the 2010 midterm elections, FiveThirtyEight has become one of the go-to sites for political junkies.

Source: https://pbs.twimg.com/media/A7Cbu_ZCQAE0m6o.jpg:large

An unlikely fate, to be sure, for an unknown consultant at the accounting firm KPMG a decade earlier. Silver tells it as an ersatz rags to riches story, a bored employee and mediocre online poker player who designed a model for evaluating baseball players and then took on the pundits—mainly because in both baseball and politics the majority of "experts" knew close to nothing. This account is too modest on the one hand—his evaluative system for baseball, PECOTA, became one of the central predictive tools of the respected Baseball Prospectus operation. On the other, it overstates his statistical creativity—his overwhelming contribution has been to introduce clearer measures of confidence to poll predictions. As he explained to new readers in 2010, his blog was "devoted to...rational analysis" and "prioritize[s] objective information over subjective information." Hardly a revolutionary statistical approach.

Notwithstanding these pedestrian goals, his message has certainly struck a chord. His 2012 book, The Signal and the Noise: Why So Many Predictions Fail—But Some Don't, was perfectly timed to capture the election-season hype and indeed is still on the "New York Times" bestseller list. Few books could possibly be blurbed by both Bill James (of "Moneyball" fame) and Peter Orszag (of perhaps lesser O.M.B. fame). Silver has, in effect, become a hero to thinking folks everywhere (particularly those with only passing statistical knowledge), who happily point to his columns as if to say to the unwashed masses, "I told you so."

But what does this ascendance of "rational" thinking represent? For one thing, it is not about the spread of statistical knowledge. Silver himself almost never fully explains the mathematics behind the models he uses. While he is footnote-happy in his book, he doesn't include any notes or appendices which would begin to teach others how to use the basic statistical concepts he deploys like regression to the mean, probability distributions, or the "nearest neighbor" algorithm.

That's hardly criticism—each equation in a book is rumored to reduce readership, although to my knowledge no one's ever done the formal regression analysis—but it is telling that Silver is certainly not trying to get more people to understand the models themselves. Rather, he seems to be saying, "Trust me."

Source: http://www.newyorker.com/online/blogs/books/nate-silver.jpg

Like most in the futurology business, however, he is also saying, "Distrust others." Indeed, one lesson of his book, despite Silver's avowed optimism, is that predictive models are often worthless and almost always fail to live up to their promoters' claims. Even well-defined rule-bound games like chess and poker turn out to be hard to predict in practice, a fact Silver acknowledges by going Zen: "The closest approximation to a solution is to achieve a state of equanimity with the noise and the signal, recognizing that both are an irreducible part of our universe, and devote ourselves to appreciating each for what it is."

Such concessions are not uncommon. Like nearly all modelers, Silver readily admits that models require active tinkering. He happily combines quantitative and qualitative information in his political predictions, using the Cook Political Report to arbitrarily assign a code of +1 or 0 within his models, for example. And he emphasizes that Bayes's Theorem and conditional probability generally (the probability of certain data given a hypothesis is not usually the same as the probability of a hypothesis given the same data but the probabilities are mathematically related) suggest the importance of context and assumptions, although he perhaps overstates the theorem's importance.

Nevertheless, Silver doesn't really analyze the feedback loops of modeling that often interest historians of science. Following the language of Donald MacKenzie, models can act as a camera, giving a snapshot of a particular process, but can also act as an engine, driving changes in the very thing being modeled. Silver admits such loops occur in finance and fashion, and even in disease reporting, but does not address the possibility that all non-trivial models might work like this: more models mean more noise, more noise will require more adjustments. Models are, after all, irreducibly human creations. And even after years of computer models analyzing "Big Data," the list of triumphs is strikingly short.

Source: http://mitpress.mit.edu/covers/9780262633673.jpg

Silver's rapid rise may ultimately represent a fear of declining discourse more than a triumph of the nerds. After all, this is an era in which Rep. Paul Ryan is considered a budget finance guru for, apparently, basic arithmetic calculations. Silver's reputation has grown in no small part because he makes predictions in areas—sports, poker, politics—in which the loudest, wildest, crassest "expert" opinions normally take center stage. Maybe members of the "reality-based community" embraced Silver because they're tired of being told, "that's not the way the world really works" in the twenty-first century. On the other hand, Silver predicted a Patriots-Seahawks Super Bowl in 2013. Oh well.

3
comments

Am I right in detecting an underlying question/critique of what Ted Porter talks about as the problem of "technicality": that too much science ends being reserved for the consideration of a technically trained few, and not enough exists in a place that's truly susceptible to wide-spread discussion and debate? So Silver becomes a champion of "reality," but it's a reality that few others can understand or recreate...

I think you're right about the large-scale effects of model proliferation. I was, still, generally impressed with Silver's 538 posts precisely because he did entertain (and discuss and investigate) the performative aspects of modeling and polling. He mentioned often that polls tend to cluster around consensus numbers (which seems to denote fiddling the numbers on some pollsters part so as not to stand out). And he also noted that his predictions were shaping the Intrade betting lines (and other betting markets too) that he sometimes used as a point of comparison.

On the math/equations line of discussion, I also take your point and I think it speaks more to the "technicality" of the situation that the book and 538 had so few particulars about mathematical mechanics. Still, I often appreciated 538 most b/c of the very deep in the weeds discussions there of criteria for evaluating and rating data from a variety of sources. This struck me as old school "statistics"---the sort of stuff that a pre-WWII statistician spent his or her time doing most often. And I sometimes think we have grown too enamored with fancy mathematics, to the detriment of thinking about the collection and evaluation of data in the first place. In that way (but only that way), I appreciate the hidden mathematics.

I also appreciated the way Silver acts as a guide through the weeds and couldn't agree more on the fact that fancy doesn't imply quality or substitute for careful thought. It is not too hard to come up with examples of complicated models that failed even in their basic assumptions, let along predictions. I definitely wanted to think more carefully, though, about the claim that Silver is showing us the way, acting as a prophet with access to somehow deeper truths. Or even the idea that we are slowly but surely converging on some ultimate truth through these models. I think that's the problem with his idea of "convergence" or with the number fiddling--that somehow with enough of it we're going to get to the bottom of things. The claim that he makes in his book that the truth stays constant as the noise increases--so we just need to get better at filtering--comes across as a rather naive epistemology.