The Era of Big Data Is Here

06/18/2012 06:36 pm ETUpdated
Aug 18, 2012

As we see the recovery sputtering because of the European debt crisis, we await the next big thing. What will power this economy the way returning servicemen, the housing boom, urbanization and the Keynesianism of presidents from Eisenhower to Nixon powered the U.S. from the 1950s to the early 1970s? Or the way liberalization, low oil prices and the tech boom created 21 million jobs in the 1990s? A clue may lie in my industry. As our economy has moved from manufacturing to service-based over the last century, commercial real estate has traced the same arc, mutating from a sector focused largely around industrial buildings to one that's about the high-rise and suburban offices that dot our commutes home. But recently, a new sector and catchphrase has emerged that indicates a major new spur in our country's growth: Big Data.

Our nation's publicly traded REITs now see cellular towers and data centers as the stars of their real estate portfolios (along with high-end malls and apartments). CALPERS, the largest public pension fund in the U.S., recently announced that it is setting up a $500 million fund to invest in data centers. McKinsey & Company predicts a 40 percent growth annually in the data being generated. Among companies of more than 1,000 employees in 15 out of the economy's 17 sectors, the average amount of data is a surreal 235 terabytes. That's right -- each of these companies has more info than the Library of Congress. And so, why should we care? Because data is valuable. The growth of digital networks and the networked sensors in everything from phones to cars to heavy machinery mean that data has a reach and sweep it has never had before. The key to Big Data is connecting these sensors to computing intelligence which can make sense of all this information (in pure Wall-E style, some theorists call this the Internet of Things). The sexiest manifestation of all of this is natural-language processing, pattern recognition and machine learning, all of which is crystallized in Siri, the backtalking, mind-reading application in iPhones -- a kind of new-millennium sassy personal assistant.

Data Driven

So, why is there so much data out there? Firstly, there's the data companies generate as they go about their business called "exhaust data" (hello, Amazon); the updates and pictures of our vacations that we breathlessly upload onto our social networks; and then all of that multimedia content we stream. With 60 percent of the world owning a mobile phone (and 12 percent of them tapping a smartphone) this is a worldwide phenomenon that has enormous implications for U.S. companies and our economy. In health care, electronic health records have been vaunted as the best way to integrate care, predict the onset of disease, eliminate medical errors, and optimize follow-up care. McKinsey thinks data could be worth $300 billion to health care. Currently, health care providers throw away 90 percent of their data partly because they don't have a place to keep it. No surprise, then, that a venture of the Blue Cross Blue Shield Association (which has health care information on 110 million people) has raised more than $37 million to create an information warehouse with 3.5 billion pieces at a time when insurers are combing patient records for ways to cut costs and improve medical care. And, there are jobs to be had. The Bureau of Labor Statistics estimates that the number of health IT jobs across the country will increase by 20 percent from 2008 to 2018, a pace much faster than the average for all occupations through 2018. The same goes for retailers who McKinsey says could potentially increase their margins by as much as 60 percent. A report by the World Economic Forum in Davos, Switzerland, titled "Big Data, Big Impact" goes as far as to declare data a new class of economic asset, like currency or gold.

As we try to get our arms around all of this, we inevitably run into exponential math. Firstly, according to IDC, a technology research firm, the aggregate amount of data is growing at 50 percent a year, or more than doubling every two years. Then, there's Moore's Law, named after Intel co-founder Gordon Moore, which states that the amount of computing power that can be purchased for a certain amount of money doubles every two years. Also, the volume of business data worldwide, across all companies, doubles every 1.2 years, according to estimates. If all these numbers and hyperbole give you a touch of vertigo, you're not alone. This is the dromosphere described by the French dystopian theorist Paul Virilio, which depicts modern society mapping the world in tiny fragments in an unending quest for speed and progress driven by technology. Putting aside the concerns about all this data, the hopes for Big Data are not unlike those for the Human Genome Project or for science itself -- a provable, positivist system to unlock age-old mysteries and a nifty way to raise profits. Incidentally, decoding the human genome originally took 10 years to process; now it can be achieved in one week.

What's in it for us?

Erik Brynjolfsson, an economist at Massachusetts Institute of Technology's Sloan School of Management, published a 2011 report with two colleagues that suggests that data-guided management is spreading across corporate America and starting to pay off. Looking at 179 large companies, they found that those adopting "data-driven decision-making" achieved productivity gains that were 5 percent to 6 percent higher than other factors could explain. Retailers now mine huge data sets to analyze sales, pricing, customer profiles, even weather data to tailor their pricing and markdowns and to make supply-chain decisions about how to get the product to the point of sale. But the more exciting frontier may be in solving humanitarian crises. As the World Economic Forum puts it,

"By analyzing patterns from mobile phone usage, a team of researchers in San Francisco is able to predict the magnitude of a disease outbreak half way around the world. Similarly, an aid agency sees early warning signs of a drought condition in a remote Sub-Saharan region, allowing the agency to get a head start on mobilizing its resources and save many more lives."

The Forum sees Big Data affecting and intervening in education, agriculture, health and global finance. One of the most extraordinary stories emerged from Haiti after the 2010 earthquake when researchers at the Karolinska Institute and Columbia University obtained data on people fleeing Port-au-Prince by tracking nearly 2 million cell-phone SIM cards in the country. By reading the sensors, they were able to pinpoint the location of more than 600,000 people, and made this information available to government and humanitarian organizations. Later that year, the same team tracked the movements of people during a cholera outbreak allowing aid organizations to mobilize.

Despite concerns with privacy and fraud, you can't argue with results like that. Data centers continue to be built in places like Phoenix, North Carolina, Santa Clara and Northern Virginia. And the U.S. is short by close to 200,000 of people with the deep analytical skills that Big Data requires, according to McKinsey. And that doesn't include the hundreds of thousands of jobs to build, equip and manage these facilities. Big Data could be the next Big Thing.