Some things seem so obviously moral, so unarguable, that years and decades can pass before they are recognised as folly. What could be more self-evident than the rightness of returning works of art stolen by the Nazis in the 1930s and 40s to the heirs of their Jewish owners? Yet nothing in today's art world is more absurd and insidiously destructive. Sir Norman Rosenthal is courageous and correct to speak out against it. The former exhibitions secretary of the Royal Academy, writing in The Art Newspaper, has said that the descendants of Holocaust victims who suddenly discover they are the rightful owners of paintings worth millions of pounds have comparatively remote claims that do not justify weakening public collections.

He is right. Visit Austria's great museums and you can't miss their sad spoliation. One of the most expensive art sales of all time, that of Gustav Klimt's Portrait of Adele Bloch-Bauer I in 2006, came about after a court case that led to this crystalline monument to Freud's and Mahler's modern metropolis being removed from the Belvedere museum in Vienna - where it illuminated Vienna's history, and specifically its Jewish history - and "restituted" to heirs of the original owner, who live in the United States. They sold it virtually instantaneously - for a record sum. In reality, the actual details of how this painting came into the Belvedere collection were complicated, and it seems from documents the museum has published that Adele Bloch-Bauer herself always wanted it to end up there.

Memory is being vandalised in the name of memory. The history of central Europe, of the matrix of Jewish art collectors and Jewish culture that was so rich and central to European life on the eve of the Holocaust, is obscured, not revealed, by this process. At best, restitution so long after the crime is meaningless. The migration of a very important Venetian Renaissance portrait of Cardinal Bessarion, by Gentile Bellini, from Vienna's Kunsthistorisches Museum to London's National Gallery a few years ago raised no eyebrows and stirred little interest - yet it was another case of a work restituted and promptly sold.

And what if a "rightful" owner is found for the National Gallery's stupendous Cupid complaining to Venus by Lucas Cranach, recently discovered to have been in Hitler's art collection, perhaps after being looted? Will London have to lose this masterpiece to some wealthy purchaser? Will that make the world a better place?

The dark side of restitution became very visible last year, when the Royal Academy's exhibition of Russian art (and perhaps this experience made Rosenthal think) was menaced by restitution claims. In this case, it is the descendants of Russian art collectors whose collections were nationalised after the 1917 Revolution who want "their" property back. It was amazing how such claims were reported as if their moral case were self-evident. In reality, anyone who has visited the Hermitage in St Petersburg or the Tretyakov Gallery in Moscow knows how much good these museums do, what beacons of civility and culture they are, how brutish it would be to weaken their collections - and that is true of all public museums, everywhere.

A work of art should never, ever be taken away from a public museum without the strongest of reasons. Making good the crimes of the Nazis may seem just that - but it is meaningless. No horrors are reversed. Instead, historical threads are broken, paintings are taken away from the cities where they have the deepest meaning, and money is made by the art market.