RBS IT failure: A result of creaking mainframes, offshoring or poor governance?

It has been two weeks since Royal Bank of Scotland (RBS), and its subsidiaries’, started reporting catastrophic IT problems that resulted in some 17 million customers not having access to their finances for days on end.

Derek du Preez
July 2, 2012

Share

Twitter

Facebook

LinkedIn

Google Plus

It has been two weeks since Royal Bank of Scotland (RBS), and its subsidiaries’, started reporting catastrophic IT problems that resulted in some 17 million customers not having access to their finances for days on end.

Although the bank is giving away little until it has completed a root casue analysis, the initial consensus pointed to an upgrade to RBS’ batch processing software, CA 7 from CA Technologies, as the cause of the problems, with a junior RBS IT worker based in India make a major mistake by deleting all the scheduling that was in place. This meant that the information had to be re-inputted into the system, causing a significant backlog.

RBS subsequently said in a letter to the House of Commons Treasury Select Committee that the error was made by Edinburgh-based staff.

Where ever the error occurred, RBS’ chief executives while considering the reputational damage and compensation that will inevitably cost the group millions of pounds, should also listen to the IT industry which has begun to debate the broader issues and ask how such a mistake could occur in the first place? Opinion is divided.

Alex Kwiatkowski, research manager for IDC Financial Insights, argues that the banking giant’s creaking mainframe infrastructure has a role to play, which is easily disrupted when coupled with inadequate IT skills.

“The complexity of banks’ mainframes is made up of a combination of in-house developed software, vendor supplied software that has been heavily customised and is running processes in ways and means that it was never originally designed for,” says Kwiatkowski.

“However, this is banking IT infrastructure the world over – it is by no means a problem isolated to RBS. They are now reaching a point where it doesn’t take much to have a disruption to the system, it is starting to creak.”

He adds: “It is effectively a ticking time bomb, which doesn’t take very much for somebody to short circuit the wiring and for it to all to blow up in their face.”

Kwiatkowski’s point about the state of banking IT infrastruture applies to many more organisations than RBS. It also leads to a wider discussion around skills. He describes the upgrade to CA’s software as ‘simply routine’, which suggests that if something was to go wrong, there should have been staff in place, with the correct knowledge, to manage the fallout.

Where is accountability going to lie?

Ovum’s financial services technology practice leader, Daniel Mayo, points to the staff member that was managing the ‘routine upgrade’ as a significant contributing factor. He states that although the upgrade was in fact routine, reprogramming the scheduling order is complex, and he believes required skills ‘well above the pay grade of the guy who was actually working’.

“The actual core system is reliable, the issue is that it’s based on quite old technologies. Because there is pressure to cut costs, there are lots of senior staff retiring or taking redundancies, and the new IT staff coming in don’t have the experience of these old IT systems to manage problems,” he said.

“When something goes wrong like this, there is probably a relatively small pool of people in the organisation that knows what to do. The people working would have understood how to do the upgrade itself, but wouldn’t have known about the scheduling order.”

In the past 12 months RBS has offshored hundreds of jobs to India in a bid to cut costs, which has led to speculation that if it had maintained a highly skilled workforce in the UK, albeit at a higher cost, it would not have suffered such a share price hammering, brand-damaging and costly failure. RBS raised a few critical eyebrows when the move too place, and has now been crucified for the decision, as it is likely that compensation and peripheral financial damage is likely to be far greater than the money saved from offshoring.

Financial services think-tank Balatro’s chief executive, Chris Skinner, however, is adamant that offshoring as a principle cannot be blamed. He argues that if you outsource or offshore functions, these operations need to be considered as if they were a UK based employee of your company’s, where effective management, governance and SLAs should limit any widespread fallout from IT mishaps.

“Where is the accountability going to lie? If you have an outsourced partner, it is the same as having an employee. You should have a vetting process that is effective, so you can’t suddenly turn to the partner and say you screwed up, because even if they did screw up, you should have internal processes to manage that,” says Skinner.

He adds: “You should have disaster recovery and contingency plans in place for these eventualities, and that is what all of us are stunned by. RBS didn’t have the competent processes in place to roll back the migration.”

Skinner questions why RBS was carrying out the upgrade on a Tuesday, rather than, for example, on the Diamond Jubilee weekend, where it would have had four days. He says: “Why did they end up with a corrupted core file without a contingency plan to roll back to a mirror parallel processing system that’s still operational?”

“I don’t think offshoring is a problem. When you outsource to a partner you need to ask yourself – do you have the right disciplines, SLAs and internal due diligence in place to ensure that it will work. If you don’t manage and govern it correctly you will have issues.”

First tremor of a technological earthquake

IDC’s Kwiatkowski is keen to highlight that although inadequate governance and skills played a part, there is also a wider issue within the global industry that these banks are operating on systems that were built decades ago and aren’t sufficient for a modern world. He describes the banks’ systems as their ‘major organs’ that need overhauling, but projects to do so have been side-lined by turmoil in the macro-economic environment.

Kwiatkowski says: “There will be a conversation that has to take place in the next 18 months, where banks and CIOs will be saying – if we need to be modern banks, to be more transparent, we need systems that weren’t built decades ago. It’s like trying to do digital banking on an analogue system.”

“But how do you make this happen? You have to examine the systems that are at the very core of your business. Replacing them will cost you tens, if not hundreds of millions of pounds. Unfortunately, doing nothing with these systems is no longer a strategy – every major UK bank has an enormous obstacle to overcome.”

Kwiatkowski believes that this overhaul will be difficult for the banks and will require outspoken executives to make it happen. However, he argues that if it doesn’t happen, this won’t be the last IT banking failure we see in coming months.

“The blame has to ultimately lie with the CEO, but it’s also how he is being advised. It takes a brave person that puts their head on the line and says – we have got a problem and it we don’t do something about it, it could be a catastrophic failure,” he says.

“RBS’ failure is the first tremor of a technological earthquake that will start to hit some of the major banks. Keeping the lid on it difficult and it doesn’t take very much for there to be a major problem.”