Software testing in banking industry: an uncommon story?

I’m confused when I see people from other industries taking banking industry as example of high quality software caused by high stake/risk. Banking software is far from perfect. Perfect software cost a lot and is illusion anyway. If internet banking sites are more or less user friendly, then back-office and teller software is not much better than green screens form 80ties. There are a lot of back office people doing workarounds, manual fixes, etc. Why so? I think the reason is high complexity of the software and low return on investment, because humans are still so good at doing their job. And by complexity I don’t just mean code complexity or pages of specifications. The true complexity is behind communication (both one-way, i.e. manuals and two-way such a emails) between people who wrote some (legacy code, 3rd party libraries, development tools, etc.) software in past, people who try to use that software to create and integrate new software and people who are supposed to use that software in future, but are using (and biased by) a different software today. And by the way - add the regulators/auditors to the picture in this industry.

The disliked part of the job

In last 5 or more years I’ve been frequently doing and managing testing of the software the banks pay for. Yes, there is a lot of „ceremonial testing” as James Bach have been calling it or „testing theatre” as I’ve red Ben Kelly calling it, but I don’t want to talk about it. I admit part of our job is what I don’t call a testing, but an evidence generation. Here in this industry software testing is not just an investigation conducted to provide stakeholders with information about the quality. Because our stakeholders are just as interested in the investigation protocol details and the ability to overtake that investigation and make their own conclusions.

Nevertheless there are a lot of interesting testing challenges that you will be missing in most of the other industries. Those unique challenges is want I want to discuss today.

But first of all – the context

Why banking industry is so different? Is it because the stake of a money to be lost? Probably. However as a technical person I will tell you that the biggest difference is the complexity of the IT solution. It changes depending on sub-industry and bank, but a worst case picture is like this: you have more than 20 different systems developed independently and integrated later. They are created by at least 10 different vendors over last 20 or more years (but you will not find a single developer in any of those companies who would be coding or maintaining this software more than 5 years). The business supported by all those 20 systems is so wide that there is not a single person in a bank to understand it all. Actually, most of the bank people rely on existing software as the software knows and help them to understand and manage the business. They don’t know the business logic in details themselves – computer made this knowledge obsolete. When you ask a subject matter expert – how this SWIFT should be processed, they will just run the SWIFT through the system to see how the system process it – and that’s how it should they will say.

High stake, high quality?

Yes, this software help banks to organize huge money transfers – millions or even milliards. But how about the quality? As I’ve wrote – the software in use is 10 or even 20 years old. It’s not uncommon for bank people to work with mainframe green screens and having to retype IDs, codes, etc from one screen to another rather than just click on a hyperlink or do a drag and drop with a mouse. You can’t compare the their experience with what we experience using free software such as google maps. Depending on how you define quality bank’s people are probably working with most terrible software in this world. That’s because software is used by their internal stuff – not by their customers. Internet banking applications are typically the most user friendly piece of software there.

Anyway, let’s get back to the main topic – how do we test this software that are designed to be not user friendly, but still reliable.

Testing step 1: get it working on my machine, or system testing

The first step is obvious and just like in any industry: software works on developer’s machine, executed from their IDE, with stubs, etc. The only difference is the complexity and as a result the number of stubs, fake data used. In other words – the difference between the developer’s machine configured environment and the final system environment is enormous. The ultimate goal is to get it working deployed on web server, connected to real database, filled with customer data. But otherwise it’s in no way interesting topic for me – so I would like to skip this step and move on to first real tester’s task

Testing step 2: smoke test automation

We want to make sure it still works when new builds are deployed, or any changes to environment done. Anything could break there – it’s just too complicated and complex to predict. And it does break more often that don’t. Even of you deploy every day or several times a day. So defining very limited (no more than 30-120 person minutes big) “smoke tests” is almost a must and automating them is a very good idea because

Testing step 3: testing different cases

So, developers have developed and unit tested code. But they are too much “unit” and it don’t help you. Let’s take an example unit tests process different values of a single SWIFT field. The only issue is – they are changing value of one field without adjust the other fields, so they are testing with swifts that are: 1. Invalid 2. Will be correctly processed by other modules/components processing swift before this component. It’s as simple as this – tester should design a SWIFT payment that 1. Is valid 2. Will pass all components, taking into account know issues, limitation and yet unfinished functionality.

So you challenge is not to do the equivalence partitioning of each value, but to discover a valid business data that would fulfill the condition from the requirements document or user story. If you haven’t tried doing it you will never believe how difficult it is.

Testing step 4: understanding the regression

The next challenge is during your regression testing. If your team is doing incremental development then your test data that we valid on last time could be invalid today, or could trigger completely different business case. So you can’t just register regression bug. You need revisit and understand once again the details of whole flow of data processing.

Testing step 5: Explaining what you have done

Your specifications, your user stories typically identify a single feature of the whole system. How does this feature translate and map into business cases that your customers operate with? That’s not uncommon that you will talk in completely different languages. They would talk about different markets (i.e have you tested the Japan market specifics), while you would talk about special business-rules, configurations, default values, etc. So how do you make sure:

They understand what do you test, what do you choose not to test and why

They review your tests and tell where test is wrong

They understand your decisions and help to improve your strategy (selecting the right tests)

You need to be a member of Software Testing Club - An Online Software Testing Community to add comments!

Yeah – that’s the problem : whenever you hear about a bug in baking software you think about stealing money and not about hundreds of back office clerks doing nothing but waiting a critical bug-fix. Bank would tell us! Not even about simple people who need to fill in extra papers or visit bank in person just because software can’t process his special case correctly.

A security consultant who quietly tipped off First State Superannuation about a web vulne... that potentially put millions of customers at risk has been slapped with a legal threat demanding he allow the company access to his computer, and warned he may be forced to pay the cost of fixing the flaw. Webster, a customer of First State Superannuation, found a direct object reference vulnerability in a statement emailed on 24 September.

He had increased a numerical value in a URL used to access his statement by one digit and was granted access to a former colleagues' account.

That account contained information including name, address, date of birth and financial payments.

Webster notified his colleague and contacted Adam Jarrett of Pillar hours later and informed them of the vulnerability and that he had not accessed other accounts or retained customer data.

He said the company thanked him for reporting the issue and fixed the flaw within 24 hours.

But the company, which had not returned requests for comment by SC, had begun legal action and informed NSW Police of what it considered a breach of the NSW Computer Crimes Act.

Burwood police detectives questioned Webster

Amazingly the letter warned that the “Trustee has the right to seek recovery from you for the costs incurred in accordance with those terms”.

The same bug, one number changed in a URL and any citigroup customer has access to the details of another customer. It was just discovered 3 months earlier. The information was out there. They just choose not to look for it.