Why Nate Silver Won, And Why It Matters

Political pundits, mostly Republican, went into a frenzy when Nate Silver, a New York Times pollster and stats blogger, predicted that Barack Obama would win reelection.

But Silver was right and the pundits were wrong – and the impact of this goes way beyond politics.

Silver won because, um, science. As ReadWrite’s own Dan Rowinski noted, Silver’s methodology is all based on data. He “takes deep data sets and applies logical analytical methods” to them. It’s all just numbers.

Silver runs a blog called FiveThirtyEight, which is licensed by the Times. In 2008 he called the presidential election with incredible accuracy, getting 49 out of 50 states right. But this year he rolled a perfect score, 50 out of 50, even nailing the margins in many cases. His uncanny accuracy on this year’s election represents what Rowinski calls a victory of “logic over punditry.”

In fact it’s bigger than that. Bear in mind that before turning his attention to politics in 2007 and 2008, Silver was using computer models to make predictions about baseball. What does it mean when some punk kid baseball nerd can just wade into politics and start kicking butt on all these long-time “experts” who have spent their entire lives covering politics?

It means something big is happening.

Man Versus Machine

This is about the triumph of machines and software over gut instinct.

The age of voodoo is over. The era of talking about something as a “dark art” is done. In a world with big computers and big data, there are no dark arts.

And thank God for that. One by one, computers and the people who know how to use them are knocking off these crazy notions about gut instinct and intuition that humans like to cling to. For far too long we’ve applied this kind of fuzzy thinking to everything, from silly stuff like sports to important stuff like medicine.

Someday, and I hope it’s soon, we will enter the age of intelligent machines, when true artificial intellgence becomes a reality, and when we look back on the late 20th and early 21st century it will seem medieval in its simplicity and reliance on superstition.

What most amazes me is the backlash and freak-out that occurs every time some “dark art” gets knocked over in a particular domain. Watch Moneyball (or read the book) and you’ll see the old guard (in that case, baseball scouts) grow furious as they realize that computers can do their job better than they can. (Of course it’s not computers; it’s people who know how to use computers.)

It’s happening in advertising, which used to be a dark art but is increasingly a computer-driven numbers game. It’s also happening in my business, the news media, prompting the same kind of furor as happened with the baseball scouts in Moneyball.

Who wants to believe that machines can tell us which stories to write, or which stories people want to read? Who wants to believe that machines can actually write stories? But they do. Forbes, my former home, started running computer-generated stories earlier this year.

Backlash

Each time this happens, there’s always lots of sputtering and outrage. When you strip away the rhetoric, at the core is always the same fear – that machines will take away jobs from humans. Baseball scouts want to keep working. So do journalists and chess masters.

So do pundits, but after this election it’s getting harder to see what role they should play.

Why listen to Joe Scarborough and his crew on MSNBC’s Morning Joe show bloviate about who’s going to win and why, when Nate Silver and his computers can just give you the correct answer?

Scarborough bet Silver $2,000 on the outcome of the election after accusing Silver of being an “ideologue” who was predicting an Obama win simply because that’s what Silver wanted to happen.

Scarborough has lost a lot more than the two grand. He’s lost his reason for being, and he has taken a lot of others down with him.

Scarborough, not Siliver, turns out to be the wishful-thinking ideologue. Nate Silver and his computers may not put Scarborough and his ilk out of business – there’s loads of airtime to fill, and windbags are still needed for that.

But Silver has exposed those guys for what they are, which is propagandists and entertainers.

And that’s fine. We still need entertainers. Computers haven’t learned to do that yet.

Related Posts

The other day I was sitting in the bank watching a clerk copy information off a paper bank transfer to initiate a new wire transfer. Being a busy person I hate inefficiencies, and this was just plain bad. When I asked why the bank didn’t use an electronic copy to speed up the process, the clerk replied that using an electronic copy can create…

I’m really enjoying writing my series on international Web apps and I’ve gotten some great feedback that others are enjoying it too. The previous post about Russian Web apps has gotten the most interest so far, including from some people in Russia who took issue with it. The comments on my post are informative, but also Alexey Andreyev wrote an…

We’re witnessing another sea change in Web publishing. From Pinterest at the beginning of this year to the launch this week of a new product from two Twitter founders, Medium, 2012 has been a year where the norms of publishing are being challenged. It wasn’t that long ago that Tumblr and Wordpress were the cutting edge of publishing. But there’s…

On the most recent episode of ReadWriteTalk, I sat down with Bret Taylor and Paul Buchheit of FriendFeed. Both have been successful ‘intrapreneurs’ inside of Google. Buchheit created GMail; while Taylor created Google Maps with another one of the 4 FriendFeed founders, and then led the Google developer tools program. After leaving Google…

As part of the lead-up to our inaugural event this Thursday 15th October, The ReadWrite Real-Time Web Summit, we’re pleased to announce our list of the top 100 Real-Time Web companies. A couple of weeks ago we posted our top 50. Thanks to the comments on that post and messages we received in the days after, we discovered 50 more companies! The Top…