Barclays is facing a fresh blow to its reputation after receiving a lawsuit over its "dark pool", the specialist trading system publicised by the author Michael Lewis in his latest book about Wall Street.

In the latest setback for the bank as it battles to restore its reputation following the Libor-rigging scandal, the New York attorney general accused Barclays of misrepresenting the safety of its dark pool, a trading system under which trades can be conducted in private.

Eric Schneiderman accused Barclays of "a systematic pattern of fraud and deceit" by operating its dark pool to favour high-frequency traders – firms that use complex computer systems to buy and sell huge volumes of stocks in milliseconds to take advantage of often tiny movements in share prices.

Schneiderman is conducting a sweeping investigation into these trading systems at a time when some experts calculate high-frequency trading accounts for the majority of US stock market trading.

Scrutiny of the practice intensified following the March release of Lewis's book Flash Boys: A Wall Street Revolt, which argues that high-frequency traders have rigged the stock market by taking advantage of systems unavailable to others.

Dark pools are a type of private market through which investors can deal without revealing their identities. Barclays' dark pool, LX, is one of the biggest. Recent data shows that in one week in June it traded more than 282m shares.

Barclays said it took the allegations seriously and had been co-operating with the attorney general and other regulators. "The integrity of the markets is a top priority for Barclays," the bank said.

Schneiderman said: "The facts alleged in our complaint show that Barclays demonstrated a disturbing disregard for its investors in a systematic pattern of fraud and deceit." He added: "Barclays grew its dark pool by telling investors they were diving into safe waters. According to the lawsuit, Barclays' dark pool was full of predators – there at Barclays' invitation."

The action by the US attorney general comes as Barclays attempts to clean up its public image after the June 2012 fine for Libor rigging which forced out the then chief executive Bob Diamond and led to Antony Jenkins being promoted to replace him.

But Jenkins's attempts to show that the culture of the bank is changing have been impeded by a number of issues, including a £26m fine last month for failings relating to the way the price of gold is "fixed".

Like others, Barclays is part of a global investigation into the way currencies are priced. It also fighting a £50m fine from the City regulator, the Financial Conduct Authority, over information it disclosed during a fundraising in 2008 that allowed it to avoid a taxpayer bailout.

The FCA says the bank "acted recklessly" but Barclays is contesting the findings, which are also subject to investigations by the Serious Fraud Office.

Analysts at Credit Suisse said: "Litigation costs are a challenge to Barclays executing its strategic plan, as they put pressure on the capital base, which based on our estimates pro-forma for litigation, is the weakest of the peer group. Having underperformed by 16% year to date relative to the sector this could further weigh on the stock". They warned other banks could face action.

Schneiderman alleged Barclays made false claims about the extent and type of high-frequency trading in its dark pool. In one example he said the bank removed the name of a high-frequency trading firm, then the fund's largest participant, from its marketing material. The attorney general claimed the bank knew the trader engaged in predatory behaviour. In response to the change, Schneiderman claimed, one employee stated: "I had always liked the idea that we were being transparent, but happy to take liberties if we can all agree."

Schneiderman's office said in a statement: "Barclays heavily promoted a service called Liquidity Profiling, which Barclays claimed was a 'surveillance' system that tracked every trade in Barclays' dark pool in order to identify predatory traders, rate them based on the objective characteristics of their trading behaviour, and hold them accountable for engaging in predatory practices."

The latest accusations by the attorney general allege that: • Barclays never prohibited any trader from participating in its dark pool, no matter how predatory its activity was determined to be.

• Barclays did not regularly update the ratings of high-frequency trading firms monitored by Liquidity Profiling.

• Barclays "overrode" certain Liquidity Profiling ratings – including for some of its own internal trading desks that engaged in high-frequency trading – by assigning safe ratings to traders that were otherwise determined to be toxic.

Schneiderman alleged Barclays operated its dark pool to favour high-frequency traders and gave them systematic advantages over others trading in the pool in order to attract them.

He said the investigation was aided "significantly" by a number of former Barclays employees.