Rushed testing, poor communication played role in British TSB outage - contractors

LONDON (Reuters) - A computer systems migration at Britain’s TSB bank that left up to 1.9 million customers unable to access their accounts was hindered by rushed and inadequate testing and poor internal communication, two contractors who worked on the project said.

FILE PHOTO: A sign is displayed outside a branch of the TSB bank in central London March 12, 2015. REUTERS/Neil Hall/File Photo

TSB, bought in 2015 by Spain’s Sabadell Bank (SABE.MC) from Lloyds Banking Group, was migrating to Sabadell’s in-house developed system, Proteo, from legacy systems for which it had been paying Lloyds around a hundred million pounds a year.

TSB customers started reporting problems making bill and mortgage payments within hours of the migration over the weekend of April 21-22. On Tuesday, clients were still complaining on social media.

Two IT contractors who managed employees involved in the migration up until late last year told Reuters that testing of the different systems was not as thorough as it could have been at that time because TSB rushed as it neared a self-imposed deadline of November 2017, later extended to April 2018. TSB rejected that view, telling Reuters testing was “extensive”.

“There wasn’t time to test everything, digital and mobile payment testing weren’t properly scoped, so it wasn’t a surprise to me when it went live last week and those parts didn’t work,” said one source, a project manager who oversaw people working on the upgrade, referring to the testing up to the first deadline.

A second source, a software tester, said tests were in some cases poorly designed or rushed in order to meet the initial project launch date. He also cited a lack of communication between IT and the business about who was managing the testing.

“There were multiple daily failures in Proteo that would go on sometimes for five days and a lack of adequate training on Proteo, meaning testing was conducted based on knowledge of the legacy systems,” the tester said.

The project manager stopped working for the company when his contract ended in December. The tester left around the same time. Both said they have kept in touch with colleagues still working at the bank and were told the situation had not changed.

Reuters contacted several contractors and TSB staff currently working on the project but none were willing to talk.

TSB, which has apologised to customers and pledged an investigation, declined to comment on the specific allegations but disputed any shortcomings in its testing.

“We are working round the clock to put things right and to keep our customers informed about the latest position. This is our priority at the moment,” it said in a statement to Reuters.

“There was extensive testing that was completed before migration. There will of course be an investigation into why the migration did not go as expected.”

TSB chief executive Paul Pester and chairman Richard Meddings are due to appear at a parliamentary hearing on Wednesday to explain how the problems occurred and what they are doing to fix them.

SOFTWARE UPGRADES

Banks around the world face the potential for similar crises as they upgrade aging computer systems after decades of under-investment and stitch together different platforms from a wave of mergers.

The sources described the Lloyds system as complicated because it was created by amalgamating many systems as Lloyds acquired rivals leading up to the 2008 financial crisis.

Banks have been fined in the past for technical problems. The Royal Bank of Scotland (RBS.L) was fined 56 million pounds by regulators in 2014 over a botched software upgrade in June 2012 that left millions of customers unable to access accounts.

TSB said it will cancel overdraft fees for the month of April and increase interest payments to savers as it tries to prevent a customer exodus in the wake of the outage. Sabadell said it is too early to estimate the costs of the incident.

It has had to bring in remedial teams from IBM (IBM.N) to help try and stem the crisis, while Britain’s Financial Conduct Authority has sent in its own team to investigate.

An FCA spokeswoman said: “We are working with the firm to ensure customers are properly communicated with and are not left out of pocket. We’ll be talking to the firm to understand exactly what went wrong.”

On a call to discuss the bank’s annual results on Thursday, Pester told Reuters he did not know whether the problem was in infrastructure, servers or the software communications layer.

Asked whether the issue could be poor testing and communications, he said he was not aware of problems in those areas.

“If he (your source) is so knowledgeable, he can come and help IBM fix it,” he said.