The idle musings of a former military man, former computer geek, medically retired pastor and now full-time writer. Contents guaranteed to offend the politically correct and anal-retentive from time to time. My approach to life is that it should be taken with a large helping of laughter, and sufficient firepower to keep it tamed!

Wednesday, January 2, 2013

'Big Data' may be so big it helps protect our privacy

The Register suggests that the sheer quantity and volume of data being collected about us may serve to protect our privacy, because we simply can't analyze it all.

We can't seem to get enough of Big Data. In its Digital Universe in 2020 report (PDF), IDC forecasts Big Data-related IT spending to rise 40 per cent each year between 2012 and 2020, as the digital universe, now at 2.8 zettabytes (ZB), or 2.8 trillion GB, explodes to 40 ZB.

That's very, very Big Data. It's a pity, therefore, that we currently analyse a mere 0.5 per cent of it all.

Not that all of these data are useful. IDC expects that by 2020, just 33 per cent of the world's data will be useful if analysed. But the delta between today's 0.5 per cent of actually analysed data and 33 per cent that could be useful if analysed is unlikely to get dramatically better. We like to think of ourselves as hyper-analytical, what with our quantified selves and "measure everything" approaches to business.

Not only are we bad at regulating our intake of information, to paraphrase Nick Carr, but we're also really bad at separating signal from noise. In hindsight, we think we see clearly, but even then we tend to miss the point.

The bit about how hard it is to separate signal from noise is quite insightful. The successes that have been had are almost always very, very simple cases. What people dream about in their pie-in-the-sky plans is massive correlation to flag "grey" areas.

What that means in the Real World is a massive false positive rate. So many things get flagged for follow up that the system gets turned off.

By giving us a year, 2020, I feel better. I have been waiting for the world authoritarians to shut down our access to cyber communication. In addition, to have so much info as to not be able to analyze it all is sweet! Many thanks.