Why shouldn’t governments and spies make sense of all this data?

The digital revolution is a boon in many ways, but it’s also a thicket bad guys can hide in

From Tuesday's Daily Telegraph

It might reassure you to know, as you wonder what dark forces are snooping on your Ocado order, that we have long since passed the point of no return. The amount of digital data produced worldwide doubles every two years or so, and already far exceeds human ability to make any sense of it. There is far more information about us out there than governments, let alone spy agencies, know what to do with. In the age of Big Data, we provide it willingly every time we swipe our travel pass or click to agree a website’s terms without bothering to read them.

Politicians delude themselves that privacy can still be protected. They talk about state intrusion into our lives as if grey men with earphones were hiding in the loft listening to our bedtime conversations while large spools of reel-to-reel tapes turned silently in the background. When Eric Schmidt, the chairman of Google, asked a Hay festival audience recently what it thought of swallowing a pill that would beam information about our bodies to computers by Wi-Fi, there were nervous mutterings. “Too late,” he said, “it’s already being licensed.” We have not kept up with reality.

The excitement generated by the claims of Edward Snowden, the 29-year-old American computer spy, has exposed how little we understand about the revolution in digital information going on around us. It has also thrown up how ill-suited our politics is to the challenge posed by a transformation that is both beneficial and overwhelming. The supply of information grows exponentially, and astonishing uses for it are being developed every day, most of which improve lives. At a very basic level, the danger is less that we will be oppressed and more that we – and states – will succumb to info overload. Whatever some politicians might say, it is no longer credible to talk of preventing the collection of data. What the latest revelations underscore, rather, is the question of what should be done with it.

But the debate prompted by Mr Snowden’s revelations has also illustrated how acute the political challenge is on the Right. Traditional conservative beliefs in institutions of the state are being tested by an anti-state populism that – paradoxically – thrives on the internet and values the power of the individual over manifestations of the collective will as traditionally embodied in governments, their agencies, and their elected representatives.

Mr Snowden leaked details of apparent American programmes to monitor internet data. A close reading of his manifesto, with his talk of a “federation of secret law” ruling the world, CIA hit-squads, surveillance nets on the verge of activation and his right to act against a duly constituted, democratically elected government, suggests he has spent too much time watching Hollywood DVDs on his laptop and studying conspiracy theory forums on the web. Whether he is naive, deluded or malicious, he has generated a drama that is more about the fantastical steps he took to put himself beyond America’s grasp than the content of the classified information he released. Much sport is being had over his choice of Hong Kong as a bolthole (“They have a spirited commitment to free speech and the right of political dissent,” he says, suggesting he is not as familiar with the power of Google as he could be), and the US government’s habit of putting its secrets in the hands of quixotic young men.

But before we mock too much, and certainly before we give credence to Washington’s doom-laden complaints about the damage he has done, we should be thankful for the insight he has given us into the dilemma governments and spies face: if defending its citizens is a state’s first duty, and if information is power, then modern-day governments have to find ways of keeping an eye on the information available. A century ago, it meant a steaming kettle in the customs shed at Dover to open the odd letter from the Continent. Now it means supercomputers able to mine vast stockpiles of data, quickly, for clues that might prevent disaster.

If the amount available is more than we can handle, what next? The scientists at Cern, for example, who use the Large Hadron Collider to conduct super-clever experiments on the very beginnings of time, keep less then 1 per cent of the data they produce; the rest they just throw away in some giant electronic waste bin because they haven’t anywhere to put it. Consider too all the emails, tweets, Facebook updates, shared photographs, Tesco Clubcard points, CCTV footage, dental records, Google searches and every single piece of information entered into a computer anywhere, and you start to get a sense of the tera-haystack security agencies have to search these days to find the needle that might save a life somewhere. Big data is a boon, but it is also a thicket the bad guys can hide in.

Appreciating the mind-boggling volume of electronic bits zipping around the ether or stored in computer servers around the world is a necessary first step towards understanding why states are struggling to keep up with technological change. It’s the point the data scientists Viktor Mayer-Schönberger and Kenneth Cukier made in a recent lecture on the benefits and challenges of Big Data: there’s so much of it that it’s useful, but also messy. In their view, those who fear a malicious Orwellian dystopia should think more of the film Brazil and its vision of pointless bureaucracies. Just as individuals must cope with a rising torrent of information pouring through their lives, so governments are pedalling furiously to avoid being overwhelmed by the digital revolution. Of course, as more of what we know and do is turned into electronic pulses, opportunities arise to harness all that knowledge. The process throws up unlikely successes. Take the spread of a recent flu epidemic: statisticians were able to study the patterns of particular Google searches – for example, “what are the symptoms of flu” – to work out a way of predicting its course. Business is just as clever: a hedge fund bought access to data on weekend traffic patterns around a particular retailer in order to work out whether its shares were worth investing in. Yesterday, three mobile phone providers agreed to pool data they hold about their users to target advertisements at them.

But the growth in data runs in parallel with the growth of individual power, with the internet as the magnifier. Where in the past we deferred to institutions as depositories of information, we are all now experts, campaigners empowered – as Mr Snowden declared – to act when we think we are right and the state is wrong.

Politicians, and a particular strand of libertarians on the Right, have revelled in the anarchic freedom and democratic power the internet provides. To them, it is a corrective to the might of the state and the self-interest of politicians. But if institutions, be they parliaments or spy agencies, cannot be entrusted to make sense of this ocean of data on our behalf, then who can? Why should Mr Snowden, or The Guardian, or David Davis, be a better judge?

The question concerns Conservatives before all others. William Hague was eloquent in the Commons yesterday about the legal rigour and proportionate judgments that rule how the Government handles our data. “This is not a casual process,” he said; threats against us are launched in secret, so the methods to combat them must be secret too. There is too much information out there for one unknown young man to be able to tell us what is right or wrong.