The BBC has admitted to a "fundamental failure of control" after a damning investigation by the public spending watchdog revealed it had repeatedly broken its own rules with a string of controversial six-figure payoffs to top management.

The National Audit Office report, published on Monday, said the corporation had paid out a total of £60m to senior staff, including £949,000 to former deputy director general Mark Byford, and £486,500 to George Entwistle who resigned in November last year after just 54 days as director general.

Another former senior BBC executive, Roly Keating, took the unprecedented step of returning his £375,000 payoff after the NAO report branded the decision to award it "seriously deficient".

The BBC had paid departing management more than they were entitled to in nearly a quarter of the 60 cases reviewed by the NAO, costing licence fee payers at least £1m.

The BBC Trust described the NAO's conclusions as "deeply worrying" and said the failure of management to stick by its own rules on payoffs was "entirely unacceptable".

Hall, who recently announced a £150,000 cap on future redundancy payments, admitted the BBC had "lost its way on payments in recent years".

He said the redundancy programme had helped the BBC to cut its number of senior staff by a third in recent years, to 445. "But we have to accept that we achieved our objectives in the wrong way," said Hall. "The level of some of these payments was wrong – I said so in my first week in the job."

Tory MP Rob Wilson described it as an "absolutely shocking report" and said it posed further questions for former director general Mark Thompson, who has already faced calls to return to give evidence to parliament over the Digital Media Initiative fiasco.

Anthony Fry, BBC Trustee and chair of the trust's finance committee, said: "Although the BBC has achieved significant savings in its senior manager pay bill, some of the NAO's conclusions are deeply worrying, particularly the failure to follow agreed severance policies in a number of cases as a result of weak governance from the BBC executive in the past. Such practices are unacceptable, and I have no doubt that they will, quite rightly, be met with considerable dismay by licence-fee payers and staff alike."

The NAO report said the BBC had "breached its own policies on severance too often without good reason" and "put public trust at risk". It found instances in which former senior managers were given big payoffs even after they had found jobs elsewhere, and examples of executives who were given guaranteed consultancy and other work at the time of their departure.

A total of £60m was paid out to around two-thirds of 401 senior staff who left between 2005 and March this year. The payoffs had helped trim the bill for senior management, saving £188m over the same eight year period.

Amyas Morse, head of the NAO, said: "The BBC has too often breached its own already generous policies on severance payments. Weak governance arrangements have led to payments that exceeded contractual requirements and put public trust at risk."

Culture secretary Maria Miller said the NAO had revealed a "culture of payoffs that simply cannot be justified" at the BBC.

Roly Keating, the BBC's former director of archive content who left last year to become chief executive of the British Library, returned his £375,000 payoff after it was criticised by the NAO.

The watchdog said the payoff had not been agreed by both the then director general Thompson and the BBC's director of human resources Lucy Adams, despite previous claims to the contrary.

Explaining his decision to return the money, Keating told Hall: "You will understand that as a matter of principle I would never wish to benefit from a payment that could not be demonstrated to have been fully and appropriately authorised."

The BBC also published on Monday the amounts paid to four of its most senior former executives in their last year in the job. Former chief operating officer Caroline Thomson received a total of £860,000, including £683,000 for loss of office, in 2012/2013.

John Smith, the former chief executive of BBC Worldwide who is paid from commercial rather than licence fee income, picked up £1.4 million after he decided to pay back six months salary in lieu when he took up a new job at fashion house Burberry.

Both Entwistle and Thompson were given considerable financial assistance for legal costs related to the Pollard inquiry into the Savile scandal, receiving £107,000 and £86,000 respectively. Entwistle received a total of £802,000 in 2012/13, including £470,000 for loss of office and an additional three weeks' salary of £25,000 after he resigned, with Thompson picking up a total of £503,000.