The age of digital technology, in which we can search and retrieve more information than we could in any previous era, has triggered a debate over whether we have too much information. Is the cure to “unpublish” things we think are wrong or out of date? Ought we have a “right to be forgotten”?

Until recently, this was an argument conducted in Europe and South America and given a powerful push by a decision in 2014 from the European Union’s highest court to provide a legally enforceable right to remove some material from internet searches.

Now the issue has reached American newsrooms. The dilemma is simple to describe and painfully hard to solve. People who have had long-ago brushes with the law or bankruptcy would prefer such information not to be at the top of search results on their name. Foolish pranks immortalised on Facebook may be harming someone’s chances of getting a job.

American editors are now getting so many requests to erase or unlink online material that they’ve been consulting pundits and lawyers for help. American media law, based around the First Amendment guaranteeing press freedom, is very different to European law.

But the development of the EU’s right to be forgotten is a poor precedent for the US or anywhere else. The European version of the right to be forgotten – really a conditional right to be taken out of internet searches – is carelessly written, based on muddled ideas and contains risks for free expression.

The “right to be forgotten” is an emblematic battle at the new frontier between privacy and freedom – both of speech and the right to know. It is a case study of the dilemmas which we will face. Who gets to decide whether free speech or privacy prevails in any given case? And on what criteria?

Gonzales’ gripe

In 2009 a Barcelona resident, Mario Costeja Gonzales, complained to Google that a search for his name produced – at the top of the first page – a newspaper item from 1998 which recorded that some of his property had been sold to pay debts. It was given unfair prominence and was out of date said Sr Gonzales. He asked La Vanguardia, the newspaper, to erase the item. Both search engine and newspaper rejected his complaint.

The case went to court. The court ruled out any action against the paper but referred the question of the search link to the EU’s Court of Justice. In 2014, the court said that Sr Gonzales did indeed have a right to ask Google to de-index items which would be produced by a search on his name – under certain conditions (and there’s a degree of irony that he fought a battle over the roght for this small story to be forgotten only to become a global cause célèbre over the issue).

Eminently forgettable: Mario Costeja Gonzalez.EPA/Cabalar

And the conditions are the heart of the matter. Google routinely de-indexes material from search results: copyright infractions (by the million), revenge porn, details of bank accounts or passport numbers. The court said that search results could be incompatible with the EU’s data protection directive and must be removed if:

… that information appears … to be inadequate, irrelevant or no longer relevant, or excessive in relation to the purposes of the processing at issue carried out by the operator of the search engine.

The judges went on to say that, as a rule, the individual’s “data” or privacy rights outrank the search engine’s commercial interest or the public’s right to know. But that would not be the case if the public had a “preponderant interest” in the information – as would be the case if the individual was in public life.

You might say, what could be more natural than this? The internet has unleashed a flood of stuff: we must have some way of protecting ourselves from the obvious harm it can cause. Carefully, transparently and accountably done, it does not have to amount to “censorship” – the claim from many voices when the judgement first appeared.

Google has taken down 1.72 billion URLs after 566,000 requests. Press freedom and free expression were never absolute – we allow some criminal convictions to be forgotten, we have libel and contempt of court laws. All restrain publication.

The problem lies with much data protection law – principally in the EU – which fails to balance the competing rights. The court judgement’s tests for whether something ought to be de-indexed are vague and opaque. How do we test for the relevance of information? Relevant to whom? When does information go out of date?

The case wasn’t about defamation: no one claimed Sr Gonzales has been libelled. It was not about correcting inaccuracy. It wasn’t private: it had been made public quite legally. The court made clear that a successful claim did not have to show that harm or distress has been caused.

Muddling through

The intellectual origins of data protection law lie in the traumas of 20th-century Europe. The Dutch government in the 1930s recorded with characteristic thoroughness the details of every one of their citizens: name, age, address and so forth. So when Nazi Germany occupied the Netherlands all they had to do to locate the Jewish and gypsy populations was open the filing cabinets. The secret police of communist states in the second half of the century and their carefully filed surveillance reinforced the lesson that secretly stored data can inflict damage.

The “right to be forgotten” is a muddled solution and fails to clarify a specific remedy for a particular problem. Here are a few of the issues which we are going to have to deal with:

Although the Gonzalez case made the compromise of leaving the online newspaper archive untouched while stopping search engines finding it, we have now had two cases – in Italy and Belgium – where courts have ordered news media archives to be altered.

There is a major unsolved problem about how far the right to be forgotten reaches. The French government thinks that it should be global, which is disproportionate as well as unfeasible.

What’s to be done?

The market isn’t providing ways to protect privacy – and individuals often part with their information barely knowing that they have surrendered some privacy. But the history of free expression has surely taught us that we should be very cautious about restrictions. If you want an alternative to the sweeping tests in EU law, have a look at the stiff tests laid out by the free speech organisation Article 19. Judges in several EU countries – notably the Netherlands – have tightened the tests for allowing material to be delinked.

EU law needs to recognise that privacy and free expression are matters of colliding rights which can’t be wished away by pretending that there’s no conflict. Collisions of basic rights can’t be abolished – they can only be managed.

The Gonzales judgement didn’t start the right to be forgotten but it did bring it to the attention of the world. It did some good by correcting thousands of small harms. But because it addressed the rights involved in such a muddled and careless way, it opened up risks to freedom of speech. The judges of the future need to do better.