It claims to be the biggest and may be one of the oldest food markets in the world, and has been at the heart of Britain's gastronomic boom over recent years. But a storm is gathering at Borough Market in south London, whose trustees have been accused of trying to force out wholesalers based there for generations in the pursuit of greater profits.

Fruit and vegetable merchants are suing the charity that runs the market in a dispute over lease agreements that could end in their eviction.

They accuse the trust's managers of attempting to capitalise on the market's growing international reputation as a foodie destination by bringing in more lucrative high-end retailers.

Separately, some stallholders are furious that the trustees have been subjecting them to anonymous taste-testing, which they claim is biased against traditional and cheaper products.

The last three remaining fruit and vegetable wholesalers based at permanent stalls have launched a High Court case against the market's trustees, claiming they have reneged on an agreement to guarantee them leases until at least 2014.

The trustees of Borough Market (Southwark) insist they will fight the case, which they argue has been provoked unnecessarily by out-of-touch traders who do not want to engage with a changing market.

The wholesalers' case has been backed by a former chairman of the trust. George Nicholson, who oversaw Borough market's expansion for 11 years until 2006, said that the wholesale market was the foundation stone on which its international reputation had been built.

"It is very sad to see the brand tarnished by such wanton stupidity from the current trustees and management. All this was avoidable. The trustees are endangering the market's public purpose and the market has lost its way," he said.

Tens of thousands of food lovers are drawn every weekend to the market, which traces its history at least to the 13th century but may date as far back as Roman times. Originally based at the southern side of London bridge, its 100 or so stalls now occupy a network of railway viaducts near the Thames in Southwark.

Tourists arrive by coach to soak in the atmosphere amid displays of game, British cheese, organic fruits, artisan breads and cold-pressed olive oils, while the market's cobbled streets and covered walkways have been used as the backdrop to films such as Bridget Jones's Diary and Harry Potter and the Prisoner of Azkaban.

The dispute goes to the heart of the market, the wholesalers say. They claim that its success has grown out of the close relationship between wholesalers of staple British fare and high quality "delicatessen" retailers who have moved in over the past decade.

The three claimants argue that their case is a fight to preserve the market as a centre of culinary excellence. Andy Sugarman, 51, the chairman of the market's tenants' association whose company EA Sugarman & Sons is one of the claimants in the case, said the Trust had failed to stick to a signed 2003 agreement to grant a series of leases until 2014.

"They have offered the other two wholesalers reduced premises and they have offered me nothing at all. It's not a viable proposition for any of us," he said.

He claimed that two additional fruit and vegetable wholesalers who were part of the High Court action had dropped out and left because of disputes over their leases. "It is easy to speculate what the trustees want to achieve. If the wholesalers were cleared out of the way, there would be a lot of space left open to be filled with fancy retailers," said Sugarman, whose family have sold fruit and salad to restaurants from a Borough market stall for 50 years.

Martin Bourne, 47, whose family firm has run a vegetable stand in the market for more than a hundred years, said that the loss of the wholesale market would damage the market's reputation.

"I want to stay but they want us out. We have tried to settle this but they won't talk to us. That is why we have ended up in court," he said.

The trust was established in the eighteenth century with the purpose of securing the market's long term future.

Many retailers have been angered by anonymous taste tests that have resulted in some of the longest-established food sellers being put on notice that they could be closed down.

A breakfast from Maria's Market Cafe, a traditional "greasy spoon" run by Maria Morruzi whose bubble and squeak has been praised as the best in Britain by Jamie Oliver, received a mark of zero at the latest taste.

She has been warned that she must now buy bacon from stallholders instead of from outside. She declined to comment last week.

Alex Fraser, who runs the internationally renowned East Teas stall and is a stallholders' representative, said that a "culture of fear" hung over the market.

"Stand and stall holders are worried by the way the market is being managed but do not feel they can speak out because they are worried they will be booted off if they do so," he said.

A spokesman for the trust said that other fruit and vegetable wholesalers would be invited to join the market if the claimants left.

"A small number of individuals have lost sight of what Borough Market's really about," he said.

"To maintain our brand and our obligations, everyone involved with the market needs to live by the highest standards of produce and conduct. In the past that has not always been the case and those people are now squealing when challenged … But no individual is bigger than the Market – whatever their past or present role – and nobody is more important than our customers and our community."