Maybe You Should Try Taking a Walk in My Shoes

The title of this post should really be, “Maybe He Should Try Taking a Walk in Your Shoes.”

The he I’m referring to is economist and author, Tim Harford. The you is the people who use NewSQL and NoSQL approaches to mine big data with database platforms like MySQL and MongoDB (or, preferably, our high-performance distributions of them, TokuDB and TokuMX).

Why should Mr. Harford take that walk? Well, he recently penned an article on big data in The Financial Times that caught my attention. It’s titled “Big data: are we making a big mistake?” and I encourage you to read it. In it, Mr. Harford thoughtfully asserts big data isn’t really solving big problems — we’re just “…making the same old statistical mistakes on a grander scale.”

He also says, “…big data do not solve the problem that has obsessed statisticians and scientists for centuries: the problem of insight, of inferring what is going on, and figuring out how we might intervene to change a system for the better.” Hmmm… Which systems is he referring to?

To be fair, not all of his observations are negative. He even cites “Google Flu Trends” and Google Translate as examples of big data successes. Perhaps Mr. Harford is concerned only with grand analyses of macroeconomic issues and trends. He is an economist after all. If so, he should be clearer about that early on in his article. Even if his commentary is aimed at the people who are using big data approaches on a macroeconomic scale, I suspect there are other folks at Google — and Apple, and Twitter, and Facebook, to name only a few — and the companies he lists as analyzers of “digital exhaust” who’d lend him those shoes.

Thankfully, César Hidalgo of M. I. T. Media Labs responds to Mr. Harford, and others like him who’ve recently published similar articles, in a piece published in the Scientific American Technology Forum. It’s titled “Saving Big Data from Big Mouths” and it too is thoughtful and well written. He points out — quite correctly in my humble opinion — that Mr. Harford and his contemporaries are missing the point and that they’d be better informed if they actually used modern big data tools to analyze something before penning their articles.

I urge you to read both of these articles. If you do, be sure to read the comments too. Some are as insightful and thought provoking as the articles.