small data

No one talks about big data any more, says Slate's Will Oremus. "Five years ago," he writes, "an article in the New York Times' Sunday Review heralded the arrival of a new epoch in human affairs: 'The Age of Big Data':"

Society was embarking on a revolution, the article informed us, one in which the collection and analysis of enormous quantities of data would transform almost every facet of life. No longer would data analysis be confined to spreadsheets and regressions: The advent of supercomputing, combined with the proliferation of internet-connected sensors that could record data constantly and send it to the cloud, meant that the sort ¬of advanced statistical analysis described in Michael Lewis' 2003 baseball book Moneyball could be applied to fields ranging from business to academia to medicine to romance. Not only that, but sophisticated data analysis software could help identify utterly unexpected correlations, such as a relationship between a loan recipient's use of all caps and his likelihood of defaulting. This would surely yield novel insights that would change how we think about, well, just about everything.

But we're less likely to use the term big data these days--we just call it data. We've begun to take for granted that data sets can contain billions or even trillions of observations and that sophisticated software can detect trends in them.

Oremus cites Cathy O'Neil's Weapons of Math Destruction and Frank Pasquale's The Black Box Society as illustrations of "the fetishization of data, and its uncritical use, that tends to lead to disaster," and suggests "Another possible response to the problems that arise from biases in big data sets:"

Small data refers to data sets that are simple enough to be analyzed and interpreted directly by humans, without recourse to supercomputers or Hadoop jobs. Like "slow food," the term arose as a conscious reaction to the prevalence of its opposite.

There is some hope, then, that in moving away from "big data" as a buzzword, we're moving gradually toward a more nuanced understanding of data's power and pitfalls. In retrospect, it makes sense that the sudden proliferation of data-collecting sensors and data-crunching supercomputers would trigger a sort of gold rush, and that fear of missing out would in many cases trump caution and prudence. It was inevitable that thoughtful people would start to call our collective attention to these cases, and that there would be a backlash, and perhaps ultimately a sort of Hegelian synthesis.