Big Data comes of age

J. Walker Smith writes:

One of the biggest winners in the US Presidential election was New York Times’ political blogger, Nate Silver. His success tells us something important about the way the world is heading, in marketing as well as politics.

Silver is a statistician – a quant. In 2008, he called correctly 49 of 50 states in the Presidential election, and all 35 Senate races. In this year’s tight Presidential contest, Silver had a clean sweep – 50 of 50 states plus the District of Columbia. As a result, his star is rising.

Silver builds sophisticated models by aggregating the polls of other firms. Throughout the 2012 campaign, he always had the President ahead, which made him a target for a legion of patronizing pundits. Yet pundit afterpundit incorrectly predicted a Romney landslide. Why? Philip Tetlock’s much-referenced 20-year study of the accuracy, or not, of expert predictions comes to mind.

Silver’s success is a marker that quants have come of age, in parallel with the rise of Big Data. An Economist video reports that the quantity of global data is forecast to be a staggering 7,910 exabytes by 2015, over 60 times greater than 2005. In this future, says The Economist, people will live in a world of sensors and software in which their “every move is instantly digitized and added to the flood of public data.” Suddenly, the world is made for quants.

The impact goes beyond politics. In marketing, one of the big new developments is dynamic real-time pricing, that moves to reflect short-run shifts in buying patterns, competitive pricing, and contextual factors. As a Wall Street Journal story reported, online prices for everything from white goods to children’s clothing to consumer electronics to shoes to jewelry to detergents to razor blades are in constant flux. One online pricing software company, Mercent Corp., says it changes the prices of two million items every hour. The era of ‘the programmable store’ comes ever closer.

Nate Silver’s success has put quants on everyone’s radar. A Forbes news analysis the day after the election noted three ways he has made a lasting impact in politics, ways that are also true more broadly.

First, quants are now a permanent part of the scene, and they focus on numbers not on presumptions. Second, quantitative analysis removes much of the mystery that has shrouded elections, marketing and other futures. Finally, expertise is being redefined. Knowing how to handle Big Data will be essential to proficiency and credibility.

Marketers are managing brands in a world of huge complexity. Big Data requires quants who can wrestle it to the ground. Nate Silver showed this to the world on one of its biggest stages, and now the world is demanding more of the show.

A longer version of this article, published as a Futures Company ‘ProofPoint’, is available via the pdf link below.

The photograph at the top of this post is a Wikimedia image showing the Empire State Building turning blue when the Presidential election was called for Obama. It was taken by David Shankbone and it is used with thanks.