Internet surveillance. Everything we do online is tracked.
Photograph: Alamy

Whatever else 2013 will be remembered for, it will be known as the year in which a courageous whistleblower brought home to us the extent to which the most liberating communications technology since printing has been captured.

Although Edward Snowden's revelations initially seemed only to document the extent to which the state had exploited internet technology to create a surveillance system of unimaginable comprehensiveness, as the leaks flowed it gradually dawned on us that our naive lust for "free" stuff online had also enabled commercial interests effectively to capture the internet for their own purposes.

And, as if that realisation wasn't traumatic enough, Snowden's revelations demonstrated the extent to which the corporate sector – the Googles, Facebooks, Yahoos and Microsofts of this world – have been, knowingly or unknowingly, complicit in spying on us.

What it boils down to is this: we now know for sure that nothing that you do online is immune to surveillance, and the only people who retain any hope of secure communications are geeks who understand cryptography and use open-source software.

This is a big deal by any standards and we are all in Snowden's debt, for he has sacrificed his prospects of freedom and a normal life so that the rest of us would know what has happened to the technologies on we now depend. We can no longer plead ignorance as an excuse for alarm or inaction.

The scale and intrusiveness of the snooping have been shocking, even to technical experts who understood, in principle, what could be done. From there it is but a short step to demonising the NSA, GCHQ and their partners in surveillance. But to do so is to miss the point.

Security services are military agencies and they do what military forces do, which is to try to accomplish the missions they have been assigned given the resources they have been allocated. Questions about whether the missions are wise, or whether the collateral damage is too high, are above the pay-grade of even the most senior officers.

And since politicians on both sides of the Atlantic insist that everything the NSA and GCHQ are doing and have done is/was done under legal authority and democratic (that is, political) control it follows that the excesses unveiled by Snowden are the consequences of political judgments and misjudgments. Which means that the only way back to more sensible regimes is also a political one. Ultimately, in other words, this is about politics, not technology.

The democratic dilemma

Secrecy impales democracies on the horns of an existential dilemma. On the one hand, democracy abhors secrecy because it makes accountability impossible: citizens cannot consent to what is done in their name if they don't know about it. On the other hand, secrecy is sometimes essential because some things have to be covert – for example activities necessary to ensure the safety of citizens. Societies face a choice between sacrificing accountability; or sacrificing secrecy.

In practice, democracies have fudged the issue by lifting the veil of secrecy just enough to provide a semblance of accountability. In the US, this takes the form of a secret court, with secret hearings and judgments, and a congressional committee, which is pathologically deferential to the intelligence services.

In the UK we have an "oversight" system comprising a deferential Commons committee, together with a couple of retired judges who examine warranted or authorised operations and monitor GCHQ's compliance with the law. This is our semblance of accountability and one of the most important services rendered by Snowden is his exposure of how threadbare it now looks.

To put this point of view to senior British politicians, as some of us have done in the last few months, is to provoke furiously indignant responses. Of course, they insist, they are on top of things. They may not understand the technological details, but they understand perfectly the issues involved and the difficult balances that need to be struck. The boffins and the spooks are on tap, not on top. These ministers (and former ministers) are infuriated by the ignorant questioning of journalists who – unlike those inside the magic circle – don't know what is "really going on". What's more important in explaining the inadequacy of oversight is that most senior politicians in Britain seem remarkably ignorant about IT. A good illustration of this comes from their fond belief that the public would be reassured by the news that GCHQ and the NSA are "only" doing bulk collection of metadata. This was the argument used by William Hague when the Snowden revelations broke and it suggests that our ministers come to the task of regulating digital technology with analogue mindsets.

Metadata are, literally, "data about data". In the case of a mobile phone, they are the numbers called by the handset, the duration of calls, the geographical location of the phone during the call, etc.

In the case of an email, the metadata include the email addresses of sender, addressee, others cc'd, the date and time of dispatch and so on. In web browsing, metadata include a user's clickstream – ie a list of the URLs visited – and the IP address of the computer running the browser. Hague, Sir Malcolm Rifkind et al maintain that collecting metadata is innocuous because it does not involve reading the content of communications. For that a warrant – with the usual supposedly legal procedures and safeguards – is required. So everything is hunky-dory.

This complacency reveals an alarming ignorance of digital technology. In our world of pervasive communications, metadata are incredibly informative. They reveal, for example, everything that one has read online. And everywhere a mobile phone user has been. In a famous case-study some time ago a German politician successfully sued a phone company for his metadata and from it reconstructed an alarmingly accurate, detailed picture of his activities, communications and movements over a period of six months. Hoovering up metadata amounts to invasive, near-comprehensive surveillance.

What's more, it's done without a warrant because of a legal precedent that goes back to the era of analogue telephony – specifically a US supreme court judgment in 1979, which decided that metadata were the property of the telephone company, not of the individual to whose calls the metadata applied. This might have been a reasonable proposition when telephone calls were routed along copper wires, but it's completely inappropriate today. And it is one of the factors that has provided the intelligence agencies with legal immunity.

But the biggest misjudgment of all – the one that legitimised most of the excesses that Snowden has unveiled – was also a political one. It was the decision of the George W Bush administration to declare a "war on terror" in the aftermath of the 9/11 attacks – and the eager adoption by the UK and other allies of the same stance.

As Professor Eben Moglen of Columbia University puts it, the intelligence agencies "presented with a mission by an extraordinarily imprudent national government in the United States, which having failed to prevent a very serious attack on American civilians at home, largely by ignoring warnings, decreed that they were never again to be put in a position where they should have known. This resulted in a military response, which is to get as close to everything as possible. Because if you don't get as close to everything as possible, how can you say that you knew everything that you should have known?" In a real war, one in which the very survival of a state is threatened by a foreign adversary, almost anything is permissible, including the suspension of civil liberties, the right to privacy and all the other things we liberals hold dear. Between 1939 and 1945, Britain was governed by what was effectively a dictatorship wielding unimaginable powers, including comprehensive censorship, the power to requisition private property on demand, and so on. Citizens might not have liked this regime, but they consented to because they understood the need for it.

The "war" on terror is not a war in this sense. It is a rhetorical device aimed at engineering consent for a particular political strategy. But it was enough to provide legislative cover for the acquisition by the US intelligence-gathering agencies of warlike powers, which included the means of surveilling every citizen on earth who had an internet connection, and every owner of a mobile phone in most countries of the world. The war on terror may have succeeded in turbocharging the surveillance capabilities of the US and its allies, but it has also inflicted significant collateral damage on the foreign policy of the US, threatened its dominance of cloud computing and other markets, undermined its major technology companies, infuriated some of its most important allies and superimposed a huge question-mark on the future of the internet as a global system. The war on terror may have made tactical sense in the traumatic months post-9/11. But as a political decision it has had a catastrophic long-term impact.

Surveillance as a business model

Some years ago, when writing a book on understanding the internet, I said that our networked future was bracketed by the dystopian nightmares of two old-Etonian novelists, George Orwell and Aldous Huxley. Orwell thought we would be destroyed by the things we fear, while Huxley thought that we would be controlled by the things that delight us. What Snowden has taught us is that the two extremes have converged: the NSA and its franchises are doing the Orwellian bit, while Google, Facebook and co are attending to the Huxleyean side of things.

In The Master Switch: The Rise and Fall of Information Empires, his magisterial history of the main communications technologies of the 20th century – telephone, radio, movies and television – the legal scholar Timothy Wu discerned a pattern.

Each technology started out as magnificently open, chaotic, collaborative, creative, exuberant and experimental, but in the end all were "captured" by charismatic entrepreneurs who went on to build huge industrial empires on the back of this capture. This is what has become known as the Wu cycle – "a typical progression of information technologies: from somebody's hobby to somebody's industry; from jury-rigged contraption to slick production marvel; from a freely accessible channel to one strictly controlled by a single corporation or cartel – from open to closed system".

The big question, Wu asked, was whether the internet would be any different? Ten years ago, I would have answered: "Yes." Having digested Snowden's revelations, I am less sure, because one of the things he has demonstrated is the extent to which the NSA has suborned the internet companies which have captured the online activities of billions of internet users. It has done this via demands authorised by the secret foreign intelligence surveillance (Fisa) court, but kept secret from the companies' users; and by tapping into the communications that flow between the companies' server farms across the world.

The reason this made sense is because so much of our communications and data are now entrusted to these internet giants. Tapping into them must have seemed a no-brainer to the NSA. After all, Google and Facebook are essentially in the same business as the agency. Its mission – comprehensive surveillance – also happens to be their business model.

The only difference is that whereas the spooks have to jump through some modest legal hoops to inspect our content, the companies get to read it neat. And the great irony is that this has been made possible because of our gullibility. The internet companies offered us shiny new "free" services in return for our acceptance of click-wrap "agreements" which allow them to do anything they damn well please with our data and content. And we fell for it. We built the padded cells in which we now gambol and which the NSA bugs at its leisure.

In our rush for "free" services, we failed to notice how we were being conned. The deal, as presented to us in the End User Licence Agreement, was this: you exchange some of your privacy (in the form of personal information) for the wonderful free services that we (Google, Facebook, Yahoo, Skype, etc) provide in return. The implication is that privacy is a transactional good – something that you own and that can be traded. But, in these contexts, privacy is an environmental good, not a transactional one. Why? Because when I use, say, Gmail, then I'm not only surrendering my privacy to Google, but the privacy of everyone who writes to me at my Gmail address. They may not have consented to this deal, but their email is being read by Google nonetheless. And before any lawyer (or Sir Malcolm Rifkind) pops up to object that having machines read one's communications is not the same thing as having a human being do it, let me gently inquire if they are up to speed on machine-learning algorithms? The fact that Mark Zuckerberg is not sitting there sucking his pencil and reading your status updates doesn't mean that his algorithms aren't making pretty astute inferences from those same updates – which is why Facebook probably knows that two people are going to have an affair before they do; or why one can make interesting inferences about the nature of a couple's marriage from inspection of their network graphs.

And this is where the interests of the NSA and the big internet companies converge. For what they have both managed to do is to abolish the practice of anonymous reading which, in the good old analogue days, we regarded as an essential condition for an open, democratic society. In a networked world, the spooks and the companies know everything you read, and the companies even know how long you spent on a particular page. And if you don't think that's creepy then you haven't been paying attention.

So what is to be done?

All of this we now know – or have belatedly realised – because of Snowden's courage. In less than three decades we have turned a liberating technology into a controlling one. The question now is whether we can extricate ourselves from this mess, or whether we have to resign ourselves to a comprehensively surveilled future?

The answer depends partly on whether we treat the Snowden revelations as evidence of a scandal or a crisis. As the political philosopher David Runciman points out, scandals happen all the time in democracies, but they generate little real change. They pass – and business-as-usual resumes.

Crises, in contrast, do provoke structural change. Our best hope, therefore, is that the Snowden revelations signal a crisis. If they do, then there are some obvious things to be done.

On the technology front, for example, the internet engineering task force needs to reconfigure the network so that every communication on it is encrypted, end-to-end. But, since this crisis has been caused by politics, ultimately only politics can fix it. So we need radically to beef up parliamentary oversight regimes of intelligence services. This requires parliamentarians who are: not awed by spooks or cowed by cant about "national security"; sceptical about the need for surveillance; and advised by truly independent experts. And while some surveillance activities will always need to be covert, activities that are kept secret simply because they would be deemed unacceptable if the public knew about them should be scrapped.

We also need some legislative changes. In particular, the law relating to metadata should be changed. Our metadata belong to us, not to communications providers, and any state monitoring of them should need a warrant. Bulk collection of metadata must end, not only because it threatens democratic values and infringes human rights, but also because it's a wasteful and ineffective way of countering the threats from which it supposedly protects us. And, on the regulatory front, internet companies should be made legally liable for securing any data that we entrust to them.

But finally, and most importantly, President Obama must declare an end to the "war" on terror so that we can all get back to the legitimate and important police work that needs to be done.

John Naughton's book, From Gutenberg to Zuckerberg: What you really need to know about the Internet, is published by Quercus Books. He is currently working on a book about the implications of the Snowden revelations.