Think of This: Most of the World’s Data is Unanalyzed

The idea that we use only 10 percent or less of our brain is one of those persistent myths that stubbornly refuses to go away. In reality, that bit of gelatinous grey matter between our ears has been extensively mapped and it appears that most of it has a function. So the notion that if only we could harness that underused 90 percent of our brain we could develop superhuman mental abilities, needs to be relegated to the urban legend scrap heap.

But what about the idea that the world of Big Data is in even worse shape – that only one percent of what’s actually out there is being analyzed?

That’s the contention of Gurjeet Singh, a guest contributor to a recent issue of GigaOM.

Singh is the co-founder and CEO of Ayasdi, a discovery platform built on topological data analysis technology. He says that to make use of the untouched 99 percent of available data – some one quintillion bytes that are collected every day (or, according to IBM, a daily harvest of 2.5 quintillion bytes) – we have to “fundamentally change the way we gain knowledge from data.”

Singh calls on the next wave of Big Data solutions to: empower domain experts such as biologists, geologists, etc. – not just the data scientists; accelerate the pace of discovery so we can get insights faster; create new levels of machine intelligence – human thought is too slow for many Big Data operations; and design systems that are able to efficiently analyze both structured and unstructured data, especially the latter in all its myriad forms.

When it comes to the evolution of big data, we’ve only begun to scratch the surface,” Singh writes. “It stands to reason that if we continue to analyze 1 percent of data, then we’ll only tap into 1 percent of its potential. If we’re able to analyze the other 99 percent, then think about all of the ways that we can change the world. We can accelerate economic growth, cure cancer and other diseases, reduce the risk of terrorist attacks, and many other big ticket challenges that we’re faced with. That’s something that we can all rally around.”

Resource Links:

Latest Video

Industry Perspectives

In this episode, the Radio Free HPC team splits on the topic of Net Neutrality. The FCC will soon publish its new rules for ensuring an even playing field for Internet Bandwidth. "Dan doesn't like the idea one bit. Henry disagrees and thinks we need Net Neutrality to keep the Comcasts of the world from running amok. As for Rich, he just finds the whole argument rather amusing since it's pretty much a done deal." [Read More...]