A British banking trade group sets the Libor every morning after international banks submit estimates of what it costs them to borrow money. The rate affects interest on many loans.

The report's warning comes three weeks after Britain's Barclays bank admitted it had submitted false information to keep the rate low. The bank agreed to pay a $453 million fine.

A number of major banks, including Citigroup and JPMorgan Chase, are also being investigated.

"An opaque and closed process" for setting the Libor allows the rate to be manipulated, the report notes. That puts some market players at a disadvantage and erodes investors' trust, it says.

The Office of Financial Research was created by the 2010 financial overhaul law. Under the law, the agency can collect and analyze financial data so it can give early warnings to regulators of potential problems.

The report cites other risks to financial stability: roughly 12 million U.S. homeowners who owe more on their mortgages than the value of homes; continued risk-taking by big financial institutions; and the European debt crisis.

The report also said the agency may require financial companies to submit data on transactions and trading positions. The office says its needs to review that data to better understand what's happening in financial markets.

Some Republican lawmakers have objected to creation of the office and its mandate. They say its power to collect confidential information from companies is too broad.

A congressional panel is investigating the alleged manipulation of the Libor.

The Federal Reserve Bank of New York released documents last week requested by the congressional panel. The documents show the New York Fed learned five years ago of big banks understating their borrowing costs to manipulate the key interest rate. They also show Treasury Secretary Timothy Geithner, who was then president of the New York Fed, raised concerns about the Libor process in 2008.

Geithner's concerns were not made public until last week, with the release of the documents.

In an interview with CNBC Wednesday, Geithner said he sent a memo in 2008 to British banking authorities outlining his concerns and alerted U.S. regulators. He said he acted quickly and appropriately to deal with the problems once he realized the rate-setting process was flawed.