The European commission said three banks and one broker had refused to settle on other claims. Photograph: Olivier Hoslet/EPA

The already battered reputation of banks took an expensive new hit yesterday when the European commission slapped a record €1.7bn (£1.4bn) fine on six firms – including the bailed-out Royal Bank of Scotland – for colluding to fix key interest rate benchmarks.

Joaquín Almunia, the European competition commissioner, warned that further fines were on the cards as three more banks, including HSBC, and one broker have refused to settle the long-running investigation by Brussels.

He revealed that the authorities were in the process of unmasking a fresh scandal in the currency markets that could further damage the industry and add to the vast sums banks have had to pay for their mistakes since the financial meltdown.

"This will not be the end of the story, neither for interest-rate derivatives nor for the manipulation of benchmarks," Almunia said. "One of the areas where we have received information that we are looking at very, very carefully is forex [foreign exchange]."

The latest fines – the first levied on a financial cartel by Europe since the banking crisis began – take the total penalties for rigging Libor and other key interest-rate benchmarks to £3.5bn.

Research by the London School of Economics (LSE) to be published next week will put a total of £100bn on the costs of misconduct for 10 major banks, including RBS, Barclays and Lloyds Banking Group, in the five years to the end of 2012. That total has now risen by £30bn, according to an analysis for the Guardian by MSCI ESG Research.

Roger McCormick, the LSE professor who led the research team, said: "To put that into context, the one year's budget of the 24 richest nations for international aid is £80bn, and the national health service budget is £100bn to £110bn every year."

Yesterday's announcement saw two US banks, Citigroup and JP Morgan, receive their first fines over the interest-rate rigging scandal, while the other banks received fresh penalties on top of those already imposed by regulators. Barclays, the first bank penalised for rigging Libor when it was fined £290m in June last year, was also part of the Euribor cartel, the European equivalent of Libor. However, this time the UK bank blew the whistle. In return for that help Barclays was let off another £570m fine.

Similarly, the Swiss bank UBS escaped a £2.1bn fine by telling regulators about the cartel that had been rigging yen Libor. It had already paid £940m in fines related to manipulating the London Libor rate.

Almunia said: "What is shocking about the Libor and Euribor scandals is not only the manipulation of benchmarks, which is being tackled by financial regulators worldwide, but also the collusion between banks who are supposed to be competing with each other."

The biggest fine – £600m – was on Deutsche Bank, putting pressure on the Treasury minister Sajid Javid, who was previously head of global credit trading for Asia and had seat on the board of Deutsche's international operations division until he left in 2009.

Javid said last night that while at the German bank he was not responsible for the departments hit by the fine and as a minister he had not held any talks with Brussels or any of the firms involved.

The commission also inflicted more pain on RBS, which has suspended four traders and will now pay another £320m on top of £390m already handed over to US and UK regulators. The new fine adds to its public image problems following a systems meltdown this week and recent allegations, which it denies, that it has abused its small business customers.

Its chairman, Sir Philip Hampton, said: "Today is another sobering reminder of those past failings and nobody should be in any doubt about how seriously we have taken this issue. The RBS board and new management team condemn the behaviour of the individuals who were involved in these activities."

The Euribor investigation focused on the period between September 2005 and May 2008 and the settlement involved Barclays, Deutsche Bank, RBS and Société Générale. In yen Libor, the banks involved were UBS, RBS, Deutsche Bank, Citigroup and JP Morgan. The broker RP Martin was also involved, and fined for using its contacts with banks involved in settling Libor.

Other banks yet to agree fines for fixing Euribor rates – apart from HSBC – are Crédit Agricole and JP Morgan. The money broker Icap, run by former Conservative party treasurer Michael Spencer, is still facing penalties for rigging yen Libor.

Almunia said: "Today's decision sends a clear message that the commission is determined to fight and sanction these cartels in the financial sector."

Matt Moscardi, senior analyst at MSCI ESG Research, questioned whether the penalties were as hefty as they first appeared. "If you look at RBS it is the equivalent to its revenues for five weeks. That's not much. If you think about the manipulation of a global interest rate where several trillion dollars are priced to it, you'd think they'd be more punitive."