New Gumshoes Go Deep With Data

By

Holly Finn

Updated Oct. 22, 2012 11:28 a.m. ET

A two-foot-long Lego model of the Imperial Star Destroyer from 'Star Wars' perches on the center table in the Hello Kitty-themed boardroom. Elsewhere in the building, taped-together cardboard boxes are piled to the ceiling. Amid the jumble of Care Bears, soda cans and packs of playing cards for "Magic: The Gathering" are camping tents and sleeping bags. This isn't kindergarten. It's "homesteading" week at Palantir, the "big-data" company that's the talk of Silicon Valley.

Once a year, employees (over 700 of them, 75% engineers) ward off corporate culture by living like this: a reminder of their rise from apartment start-up. Palantir occupies a growing number of buildings in downtown Palo Alto, and world-wide. It was recently valued at $4 billion.

Palantirians are the new Googlers. What search algorithms were to the 1990s, big data is today: a game changer. Imagine statistical analysis on steroids. Now multiply that.

It's increasingly clear that our problem today is data obesity. We ingest too many facts without converting the useful ones to mental muscle. Big-data technology treats this condition, transforming massive amounts of mismatched information into digestible, sometimes lifesaving, intelligence. But more than just a brutish mining mechanism, big data can be a subtle couples counselor: It repairs the relationship between man and machine. It may even rouse our better, more charitable selves.

In a seminal 1960 paper, computer pioneer J.C.R. Licklider observed: "About 85% of my 'thinking' time was spent getting into a position to think." His activities were mostly tedious; his "choices of what to attempt and what not to attempt were determined to an embarrassingly great extent by considerations of clerical feasibility, not intellectual capability." He wrote "Man-Computer Symbiosis" as a result, predicting a more hopeful scenario.

It is now coming true. The rise of cloud-based storage and the falling cost of gathering information is "augmenting intelligence," enhancing what we and our devices can do together by maximizing what each does best.

It isn't easy. Consider the content being spewed, around the globe, as you read this—digital images, purchase records, GPS signals, social media posts: an estimated 2.5 quintillion bytes of data every day. That's big. But "most of 'big data' is a fraud, because it is really 'dumb data,' " Peter Thiel, a co-founder of Palantir, told me. "For the most part, we would need something like artificial intelligence to turn the 'dumb data' into 'smart data,' and the reality is that we are still pretty far from developing that sort of artificial intelligence."

Still, user-friendly platforms layered atop the depths of information—like the ones that big-data companies offer—help. Done right, they make vital connections while freeing up human brain space for more intuitive, interpretive tasks. We can think.

And maybe—liberated from the clerical—we can do some good. A raft of companies from i2, bought by IBMIBM0.64% last year, to Splunk,SPLK1.00% which went public this year, are in the big-data business. So now a grain company in Missouri can sell specific seeds based on vast but granular soil samples and a baseball team in San Francisco can analyze the details of every pitch, every hit, every fielder's behavior. Less talked about is how the nonprofit sector (where I've worked for the past year) is touched.

There's been a global call for philanthropic accountability—from Ireland, where Labour senators are pressing the government to regulate the country's 7,800 charities, to Australia, where the Senate is to vote on a national regulator to enforce nonprofit transparency. Big data's pro-bono work—big benevolence—could be exactly what's needed. Scrappy groups are already making inspired inroads. Kaggle has hosted competitions among the world's best data scientists to analyze daunting sets of information, including from nonprofits. DataKind has crunched data for the public good: At the New York Police Department, it addressed questions of racism by analyzing the 111 data points collected at every "stop and frisk."

The technology of larger-scale companies like Palantir, named after the "seeing stones" in "The Lord of The Rings," is also doing its part. Designed to improve how clients draw conclusions from masses of integrated information, with clear visualizations and myriad ways to find and use facts, its platform turns anyone into a gumshoe.

Palantir-driven investigations have been used not just in counterterrorism, disaster recovery, fraud exposure and finance, but also to crack an infiltration of the Dalai Lama's email account. The company helps the National Center for Missing & Exploited Children fuse and search dozens of data sets and Humanity United weave together everything from satellite imagery to academic research, to accurately capture events in Sudan and South Sudan.

So big data may be dumb. But it's getting smarter.

Corrections & Amplifications An earlier version of this story misspelled the last name of J.C.R. Licklider as Linklider.

This copy is for your personal, non-commercial use only. Distribution and use of this material are governed by our Subscriber Agreement and by copyright law. For non-personal use or to order multiple copies, please contact Dow Jones Reprints at 1-800-843-0008 or visit www.djreprints.com.