The Challenge – and opportunity – of Big Data

Data management has grown from traditional methods of managing data to what it is today, Big Data is all the rage these days, and more than a few organizations are at least wondering what sort of business intelligence they could derive from all the information at their disposal. But while awareness of Big Data is growing, only a few organizations-like Google or Facebook-are really in position to capitalize on it now. However, the time is coming and organizations that expect to leverage Big Data will not only have to understand the intricacies of foundational technologies like Apache Hadoop, they’ll need the infrastructure to help them make sense of the data and secure it.

The proliferation of data has always been part of the impact of information and communications technology. Now, as computers and cell phones continue to pervade our daily activities and as millions of networked sensors are being embedded in these devices (as well as in automobiles, “smart” meters, and other machines), the amount of data available for analysis is exploding. The scale and scope of the changes that such “big data” are bringing about have reached an inflection point. Companies capture trillions of bytes of information about customers, suppliers, and operations.

According to McKinsey quarterly report in 15 of the US economy’s 17 sectors, companies with more than 1,000 employees store, on average, over 235 terabytes of data—more data than is contained in the US Library of Congress.

Today, the data is ever growing in every organization and organizations are inclined to grow with Big Data as it is seen a next big thing for enterprise customers. New research from the McKinsey Global Institute (MGI) finds that collecting, storing, and mining big data for insights can create significant value for the world economy, enhancing the productivity and competitiveness of companies and the public sector and creating a substantial economic surplus for consumers