A detail from a work by the Russian artist Alexandra Exter that was among those under dispute during the Wiesbaden trial.

A trial in Germany many hoped would help crack down on a flourishing trade in Russian avant-garde forgeries on the international art market has fallen short of its target after a dispute between two divorced art historians left judges unable to decide whether many disputed works were genuine or fake.

On Thursday afternoon, Wiesbaden regional court sentenced art dealer Itzhak Zarug, 72, and his business partner Moez Ben Hazaz, 45, to 32 months and three years in prison respectively for having knowingly sold forged pictures and invented the provenance of paintings by El Lissitzky and Kazimir Malevich as well as constructivists such Alexander Rodchenko.

For many art experts, however, the ruling was disappointing given the scale of the operation involved. When police arrested the Tel Aviv-born Zarug and German-Tunisian national Ben Hazaz in 2013 after a tip-off from Israeli intelligence, it was hailed as one of the biggest swoops against organised art crime in recent German history.

Via their Wiesbaden-based gallery SNZ Galeries, the pair were accused of running a cartel that made at least €2.53m (£2.2m) through the sale of forged works by leading lights of the Russian avant-garde, including Wassily Kandinsky, Vladimir Tatlin and Natalia Goncharova.

As part of an operation code-named “Malefiz” – an old-fashioned word for “misdeed”, as well as the name of a traditional German board game – police raided more than two dozen properties in Israel, Germany and Switzerland, and seized more than 1,500 works.

In particular the discovery of more than 1,000 suspicious paintings in a furniture store in Wiesbaden raised eyebrows: why were works purportedly by artists who fetched millions on the art market stored in a room without climate control and a special security system? The German news weekly Der Spiegel described it as a “forgery scandal sure to overshadow all previous ones”.

An El Lissitzky, the authenticity of which was under dispute in the trial.

But by the time Wiesbaden’s regional court announced its verdict on Thursday, charges had been slimmed down considerably. Out of all the works seized, 19 were included in the charge sheet when the trial began in 2014.

In the end, the prosecution could only prove the fraudulent sale of three faked pictures, purportedly by Rodchenko and Lissitzky: the paintings contained phthalocyanine blue, a brilliant blue pigment that was only produced years after the works were allegedly made. Zarug and Ben Hazaz were also ordered to pay back about €1m they had made through the sale of the pictures.

Unable to identify a third person required to prove that the accused forgers were acting as a criminal gang, the state prosecutor had found that many of the other alleged misdeeds fell under statues of limitation.

In a trial lasting more than 150 days over three and a half years, the court had called first on conservation scientists to give their verdict on the authenticity of the collection. By dating the pigments and binding agent in a painting, scientists can rule out whether an artwork has been recently forged – though that does not confirm it is genuinely by the artist claimed to be its creator. As a result, the outcome of the trial rested heavily on the aesthetic judgment of a number of art historians.

A painting with the inscription ‘Kazimir Malevich - Supremus’ was among the works seized by police in 2013. Photograph: Fredrik von Erichsen/AP

Andrei Nakov, a Paris-based art historian who has published on Russian avant-garde modernism since the 1960s and compiled Malevich’s catalogue raisonné in 2002, was called on by German federal criminal police in the early stages of the case to give his verdict on a few dozen of the seized works.

“As someone who has studied these artists for over 50 years, [it] took me seconds to realise that these works were obviously fakes,” said Nakov, who has spoken out in the past about the forgery problem in the Russian avant-garde market.

In 2009, an exhibition at the Château Museum in Tours, France, dedicated to Aleksandra Ekster was shut down after Nakov questioned the authenticity of the works on display.

“There is a huge problem with forgeries in the field of the Russian avant-garde,” he told the Guardian. “This is a field of art history which was censored for decades, and where archives were often destroyed. Rebuilding knowledge about this field will take a long time, and the money involved is threatening to destroy that process.”

In the Wiesbaden trial, however, the defence succeeded in getting Nakov downgraded from expert to witness after complaining he had taken only 15 minutes to examine the seized works, and drawing attention to what it called his “War of the Roses” divorce from another art historian called as a witness, Patricia Railing.

Railing, a British national who was married to Nakov between 1972 and 1978, completed a PhD on Malevich at the Sorbonne and presides over the International Chamber of Russian Modernism (InCoRM), a body involved in “expertising works of art”, created around the same time the Wiesbaden gallery started offering Russian avant-garde works for sale.

In court, Railing argued that Malevich “wasn’t just a Sunday painter” and had produced many more works than were included in her former husband’s catalogue raisonné, and that many works by abstract artists would have disappeared into private collections after 1932, when Stalin decreed socialist realism the “official” style of the Soviet Union.

The works in the Wiesbaden collection Nakov thought were fakes were in fact real, she said. “Whenever Railing says a picture is real, Nakov will argue the picture is fake,” said Thomas Scherzberg, defending, in court.

“There are good divorces, and there are bad divorces,” said Nakov when asked about his relationship with Railing. “This was certainly on the all-time list of bad divorces.”

After some of the other eight art historians called as expert witnesses withdrew their verdict or were rejected by the defence, the judge was left only with mutually contradictory statements.

Many experts fear the underwhelming conclusion to such a high-profile trial could now permanently undermine the law’s ability to crack down on fraud within the Russian avant-garde, leaving the market to be flooded with what art dealer James Butterwick called “miracle pictures” by master painters.

Butterwick is among a host of experts and dealers who signed an open letter in January this year that criticised the Museum of Fine Arts in Ghent for displaying questionable works purportedly by Russian avant-garde artists, and has questioned the credibility of some of those connected to InCoRM.

“In the case of Russian avant-garde art, the judgment of many academics is so distrusted that to have their approval on a work is the equivalent of consigning its authenticity to the ‘extremely doubtful category’,” he said.

Reinhard Spieler, the director of Hanover’s Sprengel Museum, which is currently hosting a forgery-themed exhibition called Fake News, said forgeries had become an enormous problem, especially the Russian avant-garde market.

“It is already extremely laborious and expensive for a state to prove that a work has been deliberately faked,” said Spieler, who suggested the establishment of an international online database to help smaller museums and art dealers to spot works of dubious heritage.