A modest proposal: Australian ISPs should JUMP at the chance to store metadata

The Big Data industry says it'll make them RICH and perhaps the government will even pay

Scarcely a day passes on which Vulture South is not told that an investment in Big Data software is the first step accruing undreamed-of riches.

You know the argument: organisations that collect lots of data are sitting on a trove of un-articulated insights. A little Hadoopery and analytics will quickly reveal why customers prefer vanilla to chocolate on Thursdays or churn to a rival when Mars occludes Saturn.

That sounds a promising prospect for internet service providers (ISPs), who over the years have tried – and largely failed – to build businesses beyond basic carriage.

Big Data advocates say they can turn that around because the piles of data lying around within Australia's ISPs are just the kind of thing Data Alchemists Scientists can turn into gilded app stores, movie download services and subscription security-as-a-service plays.

All of which sounds lovely, except for the small fact that ISPs are not keen on storing all the customer data that could deliver this outcome. Personal liberty and the cost of storing it all, they have recently argued, make it most undesirable for governments to insist they retain their customers' metadata.

But what if the government decided to help with those costs? Canberra's current controllers have, after all, declared it unfair to impose red tape on business. Indeed, while pensioners are being asked to make do with lesser increases in their payments, business has been spared any reduction in a subsidy on diesel fuel.

ISPs can therefore argue that tape doesn't get much redder or stickier than when it comes in the form of a few petabytes of storage. It seems only decent that the government helps ISPs with some sort of scheme to defray the cost of metadata storage facilities. That ISPs are part of the terribly-important-jobs-of-the-future-that-aren't-about-digging-things-up bit of the Australian economy surely qualifies them for some help.

Here's how things then play out for the ISPs: after a bit of lamenting the terrible spot the government has put them in and the threat to their customers' privacy, they come to some kind of subsidy deal and spend the cash on storage that handles metadata and also just happens to work well with Big Data software. Then they buy Big Data software with their own coin.

The more metadata Canberra compels them to collect, the more analytics they can perform. And the more successful they become.

ISPs end up getting rich off Canberra's coin, which helps pensioners because everyone's pension fund will invest in the booming ISPs. Australians end up with a government that knows they don't watch Rick Astley videos by mistake. Big Data advocates get another success story to which they can refer.