The author is a Forbes contributor. The opinions expressed are those of the writer.

Loading ...

Loading ...

This story appears in the {{article.article.magazine.pretty_date}} issue of {{article.article.magazine.pubName}}. Subscribe

Jason Waxman

Recently I had the opportunity to chat with Jason Waxman, the General Manager of Intel's Data Center Group. Waxman is responsible for the chip behemoth’s initiatives around Cloud Computing and is currently helping to organize an Open Data Center Alliance event in New York next week. It was an enlightening conversation covering a wide range of topics. What struck me was his level of excitement around not only cloud computing, but also what he describes as the fundamental driving force behind companies moving to the cloud. Hint, it’s not what you think it is. What I took away from the discussion is Big Data is the new cloud.

My immediate question is what does this phenomenon mean? Is the increasing volume of data simply evidence of an increasingly data driven world? Is privacy no longer even an option in a world where your every step can be analyzed in real-time? How does the ever-increasing volume of data change the landscape for today’s businesses? More importantly, what is the economic impact?

A recent report by The McKinsey Global Institute (MGI) titled “Big data: The next frontier for innovation, competition, and productivity” attempts to answer some of these questions. The report notes, “like other essential factors of production such as hard assets and human capital, it is increasingly the case that much of modern economic activity, innovation, and growth simply couldn’t take place without data.”

Big Data is all about the rapidly increasing information being created on a moment-by-moment basis. McKinsey outlined a few of the macro trends driving the industry;

5 billion mobile phones in use in 2010

235 terabytes data collected by the US Library of Congress by April 2011

15 out of 17 sectors in the United States now have more data stored per company than the entire US Library of Congress

1.5 million more data-savvy managers needed to take full advantage of big data in the United States

My next question is what exactly is Big Data? How big is big? A Gigabyte? A Terabyte? More? The reality is the term, like cloud computing, is loosely defined and interpreted by those who claim it as part of their marketing. In a 2001 research report, Gartner analyst Doug Laney defined data growth challenges and opportunities as being three-dimensional: increasing volume (amount of data), velocity (speed of data in and out), and variety (range of data types and sources). Gartner, and now much of the industry, has adopted this model for describing Big Data. Although I prefer a simpler and generally accepted definition: data sets so large and complex that they become awkward to work with using traditional database management tools.

A 2010 Economist article provided some of economics driving the industry indicating, “In recent years Oracle, IBM, Microsoft and SAP between them have spent more than $15 billion on buying software firms specializing in data management and analytics. This industry is estimated to be worth more than $100 billion and growing at almost 10% a year, roughly twice as fast as the software business as a whole.”

So what’s driving these massive numbers? The answer can be found in emerging markets. The Economist noted, “Between 1990 and 2005 more than 1 billion people worldwide entered the middle class. As they get richer they become more literate, which fuels information growth. The amount of digital information increases tenfold every five years.”

To put this into perspective, Moore’s law, which many in the IT industry now take for granted, says that the processing power and storage capacity of computer chips double while prices halve roughly every 18 months. This has been a fundamental factor driving our industry. Cloud computing accelerates this trend further enabling massive outsourced computational capabilities. What was once limited to the largest corporations are now available to anyone with a credit card. Moore’s Law is no longer limited. It has gone global in massively expanding connected world. No longer are we constrained by a single chip, device or computer.

What the PC revolution started, the Internet has super charged - information creation. The combined explosion of data, immense data creation and cheap, globally available, on-demand computation has enabled a new rapid transformation of raw data into usable information on scale we’re never seen before. More information is now being created in the time it takes me to write this article than was previously created in all of humankinds combined history. Unfortunately the majority of the information we’ve created has until recently, not been accessible. Most of this raw data or knowledge has been sitting in various silos -- be it a library, a single desktop, a server, a database or even a data center. Big Data is changing this.

MGI estimates that “enterprises globally stored more than 7 exabytes of new data on disk drives in 2010, while consumers stored more than 6 exabytes of new data on devices such as PCs and notebooks.” One exabyte of data is the equivalent of more than 4,000 times the information stored in the US Library of Congress.

A survey by Avanade digs deeper, “An overwhelming majority of companies (73 percent) have already leveraged data to increase revenue. Of those companies that have already increased revenue, 57 percent used data to increase an existing revenue stream. Notably, the remaining 43 percent used data to create entirely new sources of revenue.”

The firm notes, “This could mean an online shoe retailer using customer data to expand into adjacent clothing segments or a company improving service levels to win new customers or deepen relationship with existing ones. 65 percent of respondents report that their company leadership views data management and analysis as a source of real value, rather than a drain on company resources. In the old debate about IT being either a profit driver or cost center, big data is widely perceived to fall on the profit side of the argument. Results show that data is already producing business returns, and many executives see the promise of more.”