The ousted chief executive of the Financial Conduct Authority has expressed disappointment about his departure, saying he had “unfinished business” as the City’s top regulator.

Martin Wheatley was speaking in public for the first time since his surprise departure was announced last Friday. Wheatley is to step down in September after the chancellor, George Osborne, decided not to renew his five-year contract. That contract was due to end next March, but in a statement Osborne said the City’s top watchdog needed “different leadership”.

Wheatley told the FCA’s annual public meeting– which was peppered with angry exchanges about the regulator’s handling of compensation from banks for interest rates swap mis-selling - that he still had work to do.

Addressing the meeting at the Queen Elizabeth II conference centre in London, he said: “I am disappointed to be moving on and I do with a sense of unfinished businesses.”

He later listed the ongoing work to clean up markets through the Fair and Effective Markets review and the implementation of the Senior Managers Regime, which is intended to hold top bosses to account when things go wrong. The cleanup was prompted by the Libor rigging scandal.

Wheatley, whose role as chief executive will be temporarily filled by Tracey McDermott, a senior official at the regulator, was brought to London from Hong Kong, where he had also been running the regulator, by Osborne when the coalition government was overhauling City governance after the 2008 financial crisis. Wheatley became the first chief executive of the FCA when the Financial Services Authority was broken up.

Following speculation that the City is facing a more lenient approach from regulators, he said there was a risk “people will forget” the problems the industry faced in 2008.

John Griffith-Jones, the FCA chairman, insisted there would not be a change in attitude and paid tribute to Wheatley, saying: “I can tell you that establishing a culture at a new organisation is a pretty lonely top-down process.”

Griffith-Jones has also faced questions over his position as chairman. But when asked on Wednesday about his future, said: “I have every intention of being at this meeting next year.”

The FCA has levied a record level of fines on financial firms since Wheatley took the helm.

Fines were handed down for foreign exchange rigging earlier this year and Wheatley said conduct issues were now at the forefront of the minds of the bosses of leading firms. “I’m more convinced than ever that conduct is the top of firms’ agendas. It is no longer an afterthought,” he said.

Wholesale markets such as forex, however, were an “area that has much further to go” in improving culture, he added.

Michael Mason-Mahon, a member of the public speaking from the floor, was critical of the FCA and Wheatley. “Are you the worst joke that ever happened to the financial services industry?” he said, after claiming the regulator was not scrutinising information that it had received. He also questioned why members of the board of HSBC had not been removed despite being accused of laundering cash by US regulators.

Wheatley and other members of the FCA board faced sustained questioning over the handling of the interest rates swap mis-selling scandal. He was forced to admit that no fines have been issued and no individuals have had their licences revoked as a result.

The only course of action for individuals who feel they have been wrongly turned down for compensation is the courts, McDermott said. One campaigner said evidence had been found of criminality among bankers who sold interest rate swaps.

For the fourth year, investor activist Philip Meadowcroft asked when the report into the near-collapse of HBOS seven years ago would be published. Griffith-Jones, who is recused from discussions about the report because he chaired the bank’s auditor, KPMG, said there had been no political pressure to withhold the document, adding: “No such wink, nudge, email has existed.”

On Tuesday, the deputy governor of the Bank of England, Andrew Bailey, wrote to Andrew Tyrie, the chairman of the Treasury select committee, to warned that further delays were occurring to the HBOS report.