Nearly All Big Data Ignored, IDC Says

In 2012, only 3% of data in "digital universe" is tagged -- and even less is analyzed, according to IDC.

13 Big Data Vendors To Watch In 2013

(click image for larger view and for slideshow)

The volume of digital information will nearly double every two years between today and 2020, reaching 40,000 exabytes, or 40 trillion gigabytes, in just seven years. But for organizations to glean valuable insights from their growing stockpiles of data, they'll have to do a much better job of analyzing it.

In fact, only 3% of data today is tagged and a scant 0.5% is analyzed, according to a new study by research firm IDC. The report, "Big Data, Bigger Digital Shadows, and Biggest Growth in the Far East," was sponsored by data storage giant EMC. It explores many facets of the emerging big data phenomenon, including the type of information collected, the cost required to maintain it and growing security concerns, particularly in emerging markets.

The study's authors, IDC analysts John Gantz and David Reinsel, define the digital universe as one that contains a vast assortment of unstructured data: YouTube videos and photos from mobile phones; HD movies; ATM banking data; security footage; VoIP calls; and even subatomic impacts recorded by CERN's Large Hadron Collider.

"The digital universe lives increasingly in a computing cloud, above terra firma of vast hardware datacenters linked to billions of distributed devices, all governed and defined by increasingly intelligent software," Gantz and Reinsel write.
But despite a growing interest in the digital universe, the study finds that very little of its data is analyzed.

"If we're exploring for digital oil, it is early days indeed," said Chuck Hollis, EMC vice president and global marketing CTO, in a phone interview with InformationWeek. "We're only tagging 3% and analyzing 0.5% -- and we're struggling to do that as well. Boy, a lot of work we've got to do."

The types of data "ripe" for analysis include: surveillance footage; embedded and medical devices, including sensors implanted in the body; entertainment and social media; and consumer images, which would benefit from advanced tagging algorithms that analyze images in real time, the study says.

The rise of big data brings security concerns as well. A third of the data in the digital universe requires some sort of protection, either to prevent snooping or theft, or to adhere to regulations, IDC reports. But today only about 20% of this information is adequately protected, with data in emerging markets being the least secure.

"Therefore, like our own physical universe, the digital universe is rapidly expanding and incredibly diverse, with vast regions that are unexplored and some that are, frankly, scary," write Gantz and Reinsel.

EMC's Hollis points to one potentially controversial study finding: IDC predicts that only 13% of data will be stored in the cloud by 2020.

"Conventional wisdom is, 'Oh, it's all going to the cloud,'" said Hollis, who added that IDC's 13% estimate "is a much smaller proportion than a lot of industry pundits would estimate."

Other study findings include:

-- Most of the data in the digital universe -- 68% in 2012 -- is created and consumed by consumers.

-- By 2020, up to 33% of the digital universe will contain information that might be useful if analyzed, versus 25% today.

-- Emerging markets will boom: Between 2012 and 2020, their share of the expanding digital universe will expand from 36% to 62%.

-- IT spending on the infrastructure of the digital universe, including hardware, software, services and staff, will grow by 40% between 2012 and 2020.

-- Machine-generated data is projected to increase 15 times by 2020.

Join Cloud Connect for a free webcast with "Cloudonomics" author Joe Weinman. Cloudonomics is a new way to discuss the benefits of private clouds. Many have focused on the cost reduction possibilities while others have focused on business agility. However, private clouds can play a strategic role, as well. The Cloudonomics webcast happens Dec. 12. (Free registration required.)

Most IT teams have their conventional databases covered in terms of security and business continuity. But as we enter the era of big data, Hadoop, and NoSQL, protection schemes need to evolve. In fact, big data could drive the next big security strategy shift.

Why should big data be more difficult to secure? In a word, variety. But the business won’t wait to use it to predict customer behavior, find correlations across disparate data sources, predict fraud or financial risk, and more.