As IT Fiasco Drags On, UK Bank’s Next Big Problem Could Be Internal Revolt

It’s time for executive bonuses.

In my last article on UK lender TSB’s escalating IT nightmare, posted last Saturday, I signed off with a sarcastic remark about how, in the absolute worst case scenario, TSB’s senior executives may even lose their bonuses. It was meant as a joke. Surely TSB’s management would realize that awarding themselves massive bonuses as the chaos and financial pain caused to customers by their own botched IT upgrade continues to mushroom would be an unmitigated PR disaster. Wouldn’t they?

During questioning on Wednesday TSB’s chairman Richard Meddings did tell UK’s Members of Parliament that TSB’s CEO, Paul Pester, would be missing out on a £2 million bonus related to the IT project, which had been deferred last year. But that was in reference only to bonuses directly tied to the IT project. He did not rule out more general performance bonuses for the bank’s leadership.

On Friday the Financial Times reported that a union representing thousands of TSB employees has threatened legal action against the bank if it pays out bonuses to executives this year. Mark Brown, general secretary of the union, TBU, wrote in a letter to the chair of TSB’s remuneration committee that “given the current shambles, neither Mr Pester nor any member of the bank’s executive committee should get any performance bonuses whatsoever for 2018.”

The letter, dated May 3, includes what sounds a lot like an ultimatum: “If the remuneration committee pay any performance bonuses to any member of the bank’s executive committee then, as a shareholder, the union will take legal action and your justifications for making such awards in the face of this botched IT migration will be subject to the scrutiny of the court.”

That the union, which represents some 4,000 TSB employees, is issuing a threat as stark as this would suggest that many branch and other front-line workers, many of whom have had to bear the brunt of customer ire over the past two weeks, are close to the end of their tether. They also appear to have reason to suspect that executive staff still intend to award themselves bonuses, despite the fact that:

Pre-tax profits at the bank fell 39% in the first quarter of 2018. In other words, even before the IT problems began, the bank was hardly performing at its best.

Now, TSB faces its worst crisis since former parent company Lloyds Banking Group requested a bailout in 2009. In large part the current problems are due to management’s own incompetence, including its reckless determination, largely at the behest of senior mangers at TSB’s new parent company, Banco Sabadell, to complete a highly risky project as quickly as possible.

Management at both TSB and the new parent company Banco Sabadell have denied charges that the data migration was rushed. But as we revealed last week, an insider at the bank has alleged that the disastrous roll-out of Sabadell’s Proteo4UK system at TSB was foreseeable a long time ago, at least to many of the tech workers involved in the project, and yet the management went ahead anyway, adopting a hope-and-pray attitude.

The TBU appears to confirm the insider’s claims. In a letter to the remuneration committee sent in February, the union pointed out that tests had shown “the system was not stable enough for customer rollout.” The TBU said this week that: “if you had taken due notice of our concerns … which were coming directly from members working on the project and involved in the testing, TSB’s IT fiasco could have been avoided.”

That fiasco, now on its 16th day, shows no sign of letting up. Thousands of customers are still locked out of their online accounts. Countless standing orders, payrolls, mortgage installments and other payments and transfers have failed. There are still signs that some data have been lost or corrupted. By Wednesday the bank had received 40,000 customer complaints.

It’s far from clear how long it will take before the system is fully operational, if indeed it ever will be. Miquel Montes, Banco Sabadell’s managing director, told MPs that it should be up in a matter of days. But his was a lone voice of mindless optimism. According to The Times, a team from the Financial Conduct Authority has established a temporary operation base in TSB’s head office to monitor the bank’s efforts to restore its systems. One source told the newspaper that officials were working on the assumption that fully fixing the problems could take several months.

“The fact you can see other people’s accounts suggests this is way beyond a capacity issue. I think this is genuinely catastrophic for TSB,” said the expert, who works for a UK bank and asked not to be named.

If he is right, TSB and Sabadell will need to launch the mother of all charm offensives just to prevent a mass exodus of customers to other banks. To do that, they will need the bank’s overworked, over-stressed, under-appreciated front-line staff fully on board for the entire duration. And the best way of ensuring that doesn’t happen would be to award themselves handsome bonuses for work they haven’t done.

That’s what senior bankers do all the time these days. As the British journalist Peter Oborne laments, “over the past 30 years all the bank managers who put customers first have retired or been sacked. They’ve been replaced by sharp-suited, amoral cheapskates who are paid outrageous sums to rip us off.”

Bonuses on Wall Street are now back to pre-crisis levels while in the City of London they jumped 10% last year. Even at Deutsche Bank, which just posted its third annual loss in a row bonuses have quadrupled. As such, it would be entirely in keeping with industry practice if TSB executives were given wholly undeserved bonuses. But if that happens, the next big problem TSB faces could be internal revolt, at the worst imaginable time. Just as with the IT crisis, it would be entirely self-inflicted. By Don Quijones.

I still do not understand why they did not immediately revert back to the old system when they encountered problems (this is the IT standard fix). It is painfully amazing to see this kind of failure and what appears to be complete executive ignorance about how to react.

Depending on how poorly the “new” system handles data, they may be corrupting accounts so badly that system fall-back has become impossible.

I guess management’s plan to bring in IBM to magically fix things didn’t work so well.

Covey

May 7, 2018 at 1:39 am

Within a company IT system you could always roll back, but TSB was migrating from the IT systems of Lloyds Bank, which is now a competitor bank.

One of the problems with this migration is that the “host” system belongs to another bank who will not have let the TSB transition team unfettered access to the Lloyds system.

As I pointed out here last Saturday, the major issue is that data is being corrupted/changed. A migration can cope with routing issues where the data stream cannot find a server/SAN, but where the data is being changed and you cannot trust the numbers being displayed, then you have very few options but to start again, assuming that your last known good backups contain the correct data.

The Sabadell software is designed to run in Spain and the character/symbol sets will be for the Spanish version. If they are importing data from a UK character set system at Lloyds and pushing it in to a different set, you will get issues as I found some 9 years ago when importing Spanish data in to a UK system.

fajensen

May 7, 2018 at 2:55 am

I still do not understand why they did not immediately revert back to the old system when they encountered problems
Well, if the main application is crap then the un-install is not likely to work much better :)

From what one hears, the backend is fried too (it is not a healthy sign when someone can see other peoples accounts and when the branch office staff cannot access accounts).

They probably could revert to a known state, but one suspects, only by “dumping” all transactions since “Day Zero”.

Apart from the numerous technical aspects, Lloyds probably does not want to taint their auditing by having millions of incorrect, potentially fraudulent, transaction records being bulk-imported into their working systems.

TSB is a goner (The UK Government will bail it out somehow). And all the excec’s will get their bonuses for all of the stress they suffered (rats will be rats: when the titanic goes down, why not loot the bar on the way to the reserved lifeboats).

Covey

May 7, 2018 at 4:44 am

I have just been talking to my next door neighbour who is a TSB customer. He managed to get through to his internet banking account last night to discover that his private pension had not been credited to his account, 4 Direct Debits had failed to be paid and his balance was £64 instead of the £3000 his local TSB branch told him was in his account on Thursday.

The branch balance was what he expected, the £64 was not!! He is not a happy bunny and he was told by the branch that TSB were not processing account transfers to another bank at present.

Kent

May 7, 2018 at 5:14 am

In order to rollback to the previous system, you would have to perfectly record each transaction that has come in since the time of the cutover (called journaling in the IT world).

In essence you rollback to the old system, then, in an automated fashion, push those transactions back to the old database.

My guess is the IT team never actually journaled the transactions, did journal them but didn’t back them up often enough and has lost some, or have been too optimistic that they will fix the issues “any day now”. I would place my bets on the first option, because it would require that they hand code the journal rollback software for the structure of the old databases. That would be a sign of intellectual prowess rarely found in the banking industry.

Javert Chip

May 7, 2018 at 4:24 pm

Certainly standard practice would have been to save the incoming transactions for an appropriate period of time (30+ days or more). Reverting back to the old system should have been done immediately after detecting the new system’s problems.

I understand fajensen’s comments (above) about Lloyds wanting TSB off their IT system, but the problems relating to the new system had to be apparent to TSB technical & executive management prior to the cut-over. A problem of this magnitude just does not jump up and bite a bank in the ass overnight. Successful systems-testing for a system of this magnitude probably consumes 3+ months.

Proceeding with an obviously failed financial system is pure negligence.

fajensen

May 8, 2018 at 12:48 am

A problem of this magnitude just does not jump up and bite a bank in the ass overnight. :)

Lloyds management must have known too, maybe been on too many meeting with Pestilent Pester, that could be why they cut the lanyards and wished them “GodSpeed” (Good riddance to bad rubbish, the trash taking itself out, et cetera).

We have the same scummy management attitude in Denmark, even in government: Our IRS has so far lost somewhere between 12 Billion DKK (the number they admit) and 100 Billion DKK (that the auditors estimate when including possible VAT-fraud too) on pure fraud enabled by a ludicrously ill-conceived and woefully poorly executed automation project.

Basically, the DK IRS didn’t even check if a stock paid dividend or that taxes were paid, before one can claim DK taxes back on account of being taxed somewhere else.

Same principle applies to VAT refunds!

If they do check, they have like 12 people doing it manually because the IT system is still borked – the brainiac leadership filth fired everyone before the system was operational.

In my opinion, the whole thing is a clear control fraud if there ever was one. Somebody did this on purpose. Problem is that legally, fraud is a “thought-crime” and therefore very hard to prove.

MF

May 6, 2018 at 7:49 pm

Normally, I’d call this as a watershed moment for the Eurozone banking system, in the form of loss of credibility.

However, given the impunity with which C-Suites operate these days, I have my doubts. I assume the ECB could step in with unlimited free money until people forget, no? It doesn’t take long these days. A few months at the most.

Frederick

May 7, 2018 at 12:21 am

MF more like a few hours in most cases

Realist

May 7, 2018 at 12:53 am

TSB is a British bank operating under rules set by thd BoE, thus not belonging to the Euroland although it is owned by a Spanish bank.

Thanks to DQ for a great article once more! Intersting to see how well British press is covering this conniption ….

2banana

May 6, 2018 at 8:01 pm

Testing? Milestones? Pilot programs? Backups? A way to revert to the old system?

Nah. Throw the switch.

What could go wrong?

Old Engineer

May 6, 2018 at 8:47 pm

If “…some data have been lost or corrupted…” I’d say the bank has a far more serious problem than just their online accounts access. If they have actually “lost” some depositors money you’d have to question whether they are qualified to be a bank. And the courts might actually view it as theft. Kind of like Mt. Gox. Are you sure this bank isn’t owned by Wells Fargo?

Maximus Minimus

May 6, 2018 at 9:22 pm

The customer relations fiasco can be can easily be fixed by an advertising campaign which would say something like this: The only bank in the world that pays interest on savings account over the real inflation rate. That would surely make an impression, not to count the huge influx of deposits. Just one innovative idea. /sarc

RepubAnon

May 6, 2018 at 9:40 pm

Another reason to just say NO to paperless bank and brokerage statements…

You’re supposed to download copies (PDFs) and keep them on file; and keep a backup of them along with copies or originals of all your other cannot-lose stuff at an off-site location, such as safe deposit box.

William Smith

May 7, 2018 at 12:24 am

The problem with that is that PDFs (or any digital file) can be modified with the right software. Someone could then argue that it was forged: such as a bank trying to steal your money due to its own incompetence. Paper is still superior from a “legal proof” point of view as it is much harder for the bank to say it was forged. It is VERY hard to counter the “the computer says” argument without a physical paper trail. The paperless voting machine scandals of the past prove this. I have worked in IT a very long time and still trust only paper for important stuff such as bank statements because “bits and bytes” are just too easy to change. Sorry to be so old fashioned but I don’t trust computers, especially when programmed by el-cheapo incompetent (IBM) Indian offshore programmers.

Cameron S

May 7, 2018 at 1:32 am

Absolutely correct William. Any banking data generated from a home computer and then printed out can be messed with and when push comes to shove it’s validity can be challenged.

When it comes to important bank records you need to have the original paper statements generated by the bank and sent to you. It is very much more difficult to challenge the validity of these because they can be authenticated but the banks increasingly want to do away with customers having bank generated paper statements. Even if they charge you for them, always ensure you have the bank generated paper statements for every type of account of any significance.

L Lavery

May 7, 2018 at 3:24 am

“The problem with that is that PDFs (or any digital file) can be modified with the right software. ”

Not if it’s cryptographically signed it can’t, as any alteration will invalidate the signature.

PNC will not let me go back to bank-printed monthly statements after having switched to electronic — I checked last week after reading here about TSB’s issues. I sure don’t recall having agreed to give up printed statements *forever* when “going green”! I’m going to call a human and check to make sure, and will refer to TSB’s issues if asked.

anon

May 7, 2018 at 4:03 am

I spent the last 10 years of my 40-year IT career working for a major bank located near Newcastle. I was shocked to the core at the general lack of understanding of basic IT safeguards when releasing important software. The culture was one of “fix on fail” with no thought given to backups. On more than one occasion I was castigated for developing a rollback plan. My colleagues “got the hand” from (non-IT educated) management when concerns were raised. It was only through luck and the dedication of experienced IT staff that major disasters were prevented. I think two things hit me hardest – that senior management allowed the IT infrastructure to run out of control and there was a significant lack of IT-related education in the managers who were responsible for the systems. To be fair I think there is the assumption that managing IT follows the same principle as managing any other department – I disagree with this.

Covey

May 7, 2018 at 9:08 am

The bane of my life used to be non-tekkie Project Managers running IT projects. I actually once had to explain to the PM how Exchange email software worked when I arrived on site to do an 18,000 user Groupwise to Exchange migration. The PM had a sheet from MS Project saying that it was going to be finished in 10 working days, no testing and she was taking more Vallium than was good for her.

sierra7

May 8, 2018 at 1:43 pm

Decades ago when the grocery business was converting to “scanners” those of us who worked the systems on the front line (the stores) were given a piece of advise from our IT (help desk) department:
“If the system malfunctions, goes down, etc., FIRST thing you do is lock the manager in his office so he can’t touch anything!” I used that advice many times sans the lock him in the office…just “Hands OFF!”

desmond

May 7, 2018 at 4:10 am

is this the wimper which will end the banks?

mark

May 7, 2018 at 8:16 am

I’m sure that TSB is no different than The Fed.

Like all “too big to jail” banks-

they certainly have only our best interests at heart …….

Petunia

May 7, 2018 at 8:17 am

When you posted the original story I opined that this was a regulatory failure. It is still a regulatory failure and cannot be fixed until the regulators shut the bank down to fix it.

It should go something like this, reinstall software and check to see if balances still correlate, then replay each and every transaction – day by day. If and when something breaks, fix it, go back and replay the day. This needs to be done until every transaction is accepted by the system without failure and they balance the books. Until then, they need to shut down this bank because there is systemic risk that is entering the system, whether they know it or not.

Javert Chip

May 7, 2018 at 4:35 pm

Petunia

The problem is the system is in-flight (ie in production) – every day there are millions of required transactions – debit & credit – that need to post and every day the system is off-line to “fix” the software just makes things geometrically worse.

I spent my career in technology, audit & finance, but I have never seen anything as negligent as this TSB fiasco. It’s hard to even come up with a story about how this will end.

unit472

May 7, 2018 at 8:24 am

Back in the stone age (1972), Bank of America would, every night, print out a huge stack paper with every account holders balance. Of course it was just a California bank back then but still a giant bank.

Couriers would deliver to each branch their customer’s balance and that is what tellers would check when a depositor came in to withdraw money. I worked at BofA’s data processing center back then and used to peruse the data as it showed foreign depositors as well. I thought at the time I wonder what the Mexican government, e.g., would pay to see list of account holders but then realized the biggest account holders were probably Mexican government officials!

I don’t suppose it ever occurred to TSB officials to do such a printout before switching over to their new system so that they would have an incorruptible starting point and do manual ledger entries until they got their system running ( if they ever do)

raxadian

May 7, 2018 at 12:30 pm

Never attribute to foolishness and ignorance what can be perfectly explained by excessive greed.

SnotFroth

May 7, 2018 at 6:15 pm

I bet these IT problems haven’t gone unnoticed to the criminal hacker community. They’re probably descending like vultures on this opportunity to steal money and account numbers, compounding the problem.

If it’s true that customers can see each other’s accounts, the hackers are surely perfused with glee.

Swamp metro area

May 8, 2018 at 1:17 am

Talk about debt destruction. Another hit for the UK.

sierra7

May 8, 2018 at 1:47 pm

“Hi! I’m your banker and I’m here to help!!”

JustMeh

May 9, 2018 at 7:59 am

Hmm, betting against a c-suite level banker getting their bonuses despite horrific performances? Sadly that’s a likely losing bet these days. Hope I’m wrong, but I can still see them collecting multi-million dollar paydays.

Sam Trimble

May 9, 2018 at 12:43 pm

Typical of management and tech issues. They don’t know anything. But they put out optimistic schedules and demand they be followed.

I used to work for a tech company. I saw a division manager, from the Sales side of the company, just make a blanket assertion that all R&D projects had to be done twice as fast. As could be expected, he did not provide any suggestions to the engineers on how they could accomplish this.

And the same executive pushed for an early release of a key product line when the testers and engineers knew there were problems with that. 3 years later the problems were still unresolved, customers were complaining loudly, this key product line wasn’t selling, and revenue was falling. All to avoid a few months delay in a release date when the problems could have been solved. So of course I was laid-off to cut costs while the same executive was promoted.