The author is a Forbes contributor. The opinions expressed are those of the writer.

Loading ...

Loading ...

This story appears in the {{article.article.magazine.pretty_date}} issue of {{article.article.magazine.pubName}}. Subscribe

Crack is being served in Silicon Valley. An enthusiastic crowd of geeks and suits -- all of them "data scientists" -- just spent three days at the O'Reilly Strata conference (#strataconf) in Santa Clara. All over the event's menu is the crack cocaine of our day: big data.

A couple decades ago, when "big data" was mostly the dominion of a few credit-rating and mailing-list management companies such as Acxiom, Experian, InfoUSA (now Infogroup) plus some disparate, disconnected monster databases built by telcos, physicists and national security agencies, I used the metaphor of catnip instead of crack. I'd describe how access to rich data about customers was like catnip to executives, managers, statisticians and others in the consumer-mass-marketing economy. It was irresistible.

But that was long before most of us had poured our hearts and lives into Blogger, Facebook, Twitter and dozens of other online sites, for all to see -- and capture. Catnip now evokes images of LOLcats and frenzied cuteness, which is too lighthearted for the topic at hand. The bits of data that a few players had about us have turned into a torrent of data that's out and about. And less expensive than before. It's addictive like crack.

Big data is strategic now. Facebook is valued at around $100 billion because it has collected a treasure trove of data that may unlock the secrets of selling more things to more people. Most other companies would like to have whatever they're having. Google offers free email, word processing, mapping, analytics, video, videoconferencing and much more because they're selling us to advertisers. The byword these days is, "if you're not paying for the service, you're the product." Before turning to the ethical choices at hand, let me offer a mini paean to big data. I love big data, and not just because of my background in econometrics. A few things to love about it include:

It's helping solve big problems. Early detection of epidemics. Automated spell-checking. Crowdsourced astronomy. We seem to have entered the Age of Big Data.

It's creating useful feedback loops. In participatory medicine, people opt in to share data so they can analyze it and continually improve their health outcomes. This kind of feedback is spreading from field to field.

It's eroding the culture of expertise. Read Daniel Kahneman's new book and you'll stick to statistics. Add a pinch of Taleb and you'll never speak to tie-wearing experts again.

It's nurturing a culture of collaboration. From participatory medicine to open science and open government, scientists and citizens alike are resetting the terms of innovation.

It's a major new source of employment. Ok, maybe not major, but one of the few bright spots on the job horizon is the desperate need for those "data scientists."

The dark side: Now we can talk about big data's dark side, one aspect of which was spotlighted recently in a piece by Charles Duhigg that must have caused Target's senior management to break out in a cold sweat. I'll focus mostly on personal data here, even though there are dark sides to other collections, as well.

We are in the cross-hairs: When companies say they are "customer-centric," we might be tempted to think that it means that they honor customers above everything. That the customer's wish is their command.