"Spies spy! Who knew?" Thus the world-weary shrug from too many people who ought to know better over the revelations deriving from the material leaked by Edward Snowden about what goes on inside the west's major intelligence agencies in 2013. We have all read our Le Carré, they sigh. We spy on them, they spy on us. Except in fiction, it must remain a secret world. The secrecy has to remain near-absolute because our national security depends on it. The best way for the state to ensure such secrecy is to have an armoury of criminal and civil laws − backed by punitive sanctions − to deter any leakages.

This used to work. But the nature of spying has changed: this much we have learned from Mr Snowden. What was once highly targeted has now become virtually universal. The evident ambition is to put entire populations under some form of surveillance. The faceless intelligence masters may say they are still searching for needles, but first they want the entire haystack. And thus countless millions of entirely innocent (in every sense) citizens are potentially being monitored. Their phone calls, web searches, texts and emails are routinely intercepted, collected, stored and subjected to analysis.

Did the governments involved ever stop to think about the notion of consent? Did any engineer, spy chief, minister, congressman or president ever wonder whether such a dramatic change in the contract between state and citizen required some form of debate?

Secrecy and openness

Thanks to Mr Snowden they have now got a debate − one that is rippling around the world. President Barack Obama says he welcomes that debate. That much is encouraging, even if it seems unlikely to be true because it is not going to be a comfortable debate for any government − nor for those in intelligence, nor for anyone running a major technology or telecommunications company. The world was simpler when the law could be used to prevent any meaningful and informed discussion of what was involved. The laws crafted before and during the first world war (the Espionage Act in the US, the Official Secrets Act in the UK) saw to that.

Secrecy and openness must collide. Governments and spies will place the greater emphasis on security: that is inevitable. Individuals who treasure free speech, an unfettered press, the capacity for dissent, or an individual's rights to privacy or protection against the state, will have equal, or greater, concerns.

It is obvious that virtually anyone with a digital life − any user of Google or Verizon or BT or Facebook or Skype − is entitled to know quite how much privacy they can reasonably expect. This is the coming debate.

Who will hold the debate, and how is it to be informed? To date, there has been a vigorous discussion on these matters in the US and European legislatures and media. In the UK, the number of MPs or peers who have said anything at all is tiny. Much legal oversight of intelligence matters happens in closed courts. Parliamentary oversight is a similarly shadowy affair. In the UK, Sir Malcolm Rifkind, who is supposed to be a kind of regulator, too often sounds like a cheerleader. In the US, the same can alas be said of Senator Dianne Feinstein, who heads the Senate intelligence committee.

Responsible reporting

What role does a free press have in assisting and informing this debate? In late May, Mr Snowden gave this newspaper a volume of documents from his role as one of 850,000 intelligence employees cleared to read and analyse top-secret material. It is difficult to imagine any editor in the free world who would have destroyed this material unread, or handed it back, unanalysed, to the spy agencies or the government. The Guardian did what we hope any news organisation would do − patiently analysed and responsibly reported on some of the material we have read in order to inform the necessary public debate.

Some time after our first disclosures we were contacted by the cabinet secretary, who said he spoke on behalf of the prime minister. He acknowledged that we had behaved responsibly, expressed concerns about the security of the material we held and requested the return or destruction of the documents. We explained that complying with the request would destroy our ability to report. At this stage there was no threat of law, but nevertheless we took the precaution of sharing some of the material with news organisations in America, where we consider there to be more robust protections for serious journalism of public importance.

Some weeks later the tone of these and other discussions changed. There was, by mid-July, an explicit threat that the government would, after all, seek to stop the Guardian's work and prevent publication of further material by legal means. To have resisted such action would have involved handing over ultimate control of the material to a judge and could have meant that no stories could have been published for many months, if at all. The first amendment of the American constitution guarantees its press protections of which British editors can only dream. For more than 40 years − since the publication of the so-called Pentagon papers in 1971 − it has been accepted that the state will not succeed in trying to obtain prior restraint of the press. So we will in future report this story from New York. We have shared some material with, and will collaborate with, the New York Times.

It is, we believe, inconceivable that the US government would try to obtain, or the US court grant, an injunction against publication by the NYT. The US attorney general has recently given an assurance that he will not prosecute any journalist "for doing his or her job". So the debate about the mass collection of data on populations, the links between the state, the intelligence services and large corporations, and the uses and limits of oversight can continue.

Meanwhile in the UK, the police − with the apparent knowledge of the government − misused a law designed to combat terrorism to detain a member of the Guardian's team for nine hours and to confiscate his material. The former lord chancellor, Lord Falconer, has confirmed that there was no intention that the 2000 Terrorism Act should be used against people like David Miranda, the partner of the Guardian columnist Glenn Greenwald. "The state may wish that journalists would not publish sensitive material," he wrote in these columns last week, "but it is up to journalists, not the state, to decide where to draw the line."

Civil liberties and security

These are words that should be heeded by the British government official who told us that the Guardian had "had our debate" and that there was no "need" to write any more. It is not the role of politicians or civil servants to determine the limits of public discussion. Nor should the debate be circumscribed by attempting to criminalise the act of journalism − without which, in this instance, there could be no debate.

Citizens of free countries are entitled to protect their privacy against the state. The state has a duty to protect free speech as well as security. Fundamental rights, as we say, collide. Journalists have a duty to inform and facilitate a debate and to help test the consent of people about the nature of any trade-offs between civil liberties and security. A democratic government should seek to protect and nourish that debate, not threaten it or stamp it out.