Financial services firms are investing more to remedy post-event quality failure, but not enough to pre-empt failures at the source, writes Steve O'Reilly of SunGard Consulting Services.

As system interdependency continues to grow and IT infrastructures become ever more complex, financial services firms are spending more to remedy post-event quality failures rather than properly investing in failure elimination at the outset. For financial services organizations today, software quality is the lifeblood of efficient operations, but still, many do not pay enough pre-emptive attention to ensure systems are fit-for-purpose to support increasing usage and regulatory demands.

Stephen O'Reilly, SunGard Consulting Services

SunGard’s Meeting the Testing Challenge reportexplores the current state of system testing in the financial systems industry throughout Ireland, the UK, U.S. and Middle East. The survey found that while there is some intent in improving testing practices, there is still a distinct lack of sufficient investment required to pre-empt or mitigate failures at source.

Despite intent toward improvement, notable system failures still occur as we have seen over the past few years. The costs involved in such quality failures can range from millions to billions of dollars in after-release fixing, maintenance and reputational damage – still, the problem continues to prevail in a big way.

If these were one-off incidents we could surmise that the industry is learning its lessons. However, we only have to look at examples like the U.S. healthcare.gov website becoming mired in technical glitches and bugs or when a German regional bank transferred money twice to thousands of customers in error to see that quality failures still prevail – and there are many, many more recent examples of quality failures across the globe that can be used to support this viewpoint.

While some of the findings uncovered in the survey were expected, others generated surprise –considering the type of failures mentioned, the turmoil experienced in financial markets in recent years and the subsequent shift to reduce the costs associated with systems delivery to support key financial activities.

In the Meeting the Testing Challenge survey, respondents were asked to highlight the most important factors that restricted the potential to improve and introduce efficiencies into testing processes.

• Fifty-nine percent stated that budget restrictions were their most prevalent issue with many respondents citing testing cost in overall project spend as another major factor. Testing is not typically the most expensive part of system development, but it is by no means an insignificant cost.

• While nearly half of respondents (49%) said testing accounts for 25% or less of project costs, 23% said that their testing expense exceeds that percentage.

• Nine percent of respondents said testing accounts for 50% or more of project costs, and while 59% said there was an appetite to improve testing, a whopping 41% selected, “no, I do not have any impetus to improve at this time.”

Given these findings and the quality failures we continue to see, it seems logical then that improving the testing process is the first step to improve quality and reverse the defect cost curve and overall project spend. However, the study puzzlingly found that only two-thirds of firms are willing to commit 25% more budget than their previous year spend to improve testing practices while only one in 20 stated that they plan to increase their testing budget by 50% or more in order to introduce such efficiencies.

In order to truly reduce costs associated with testing, the testing process has to change, and that change must relate to an increase in the quality of software put forward. Cutting the test team to reduce costs may reduce your wage bill, but this strategy does not naturally lead to an increase in quality and an elimination or mitigation of the problems we’re seeing. Quite simply, the opposite is likely to occur and quality will suffer.

If the “status quo” continues to reign, the rising cost of failure will continue its path upward. Fear of failure, however, should not be the only reason for improving quality. Besides reducing costs, changing testing practices to embed quality from the outset can enable firms to generate increased sales opportunities through longer-term improved reputation and customer perception.

The simple message is: all firms have the power to make the necessary changes required to move towards the path of improvement, and it’s worth noting that the price of simple change is not prohibitive. The focus, therefore, needs to move from meeting deadlines just for the sake of meeting deadlines to becoming more quality-oriented, more efficient, and thus more effective in IT software testing practices. In doing so, quality should improve, costs should reduce, and reputational and organizational value should be enhanced.

It is indeed a consequence of our times that we all expect an immediate reward from every investment. Investments in testing only generate long-term (non-immediate) returns and in fact the outcome itself won't be visible, i.e. the lack of catastrophic system failure. Instead, investment in more system functionality alone leads to an immediate result: "look boss, we just delivered all this new functionality". It takes a visionary leader within the department, or a firm with the right culture to plan and execute for the long-term to strike the right balance across the software delivery pipeline, which means a budget that include adequate testing, test automation platforms, test automation specialists, and so on.

Part of the issue (prevalent for how many decades now?) I would attribute to "Happy Path Requirements". It is relatively easy to state "it must work like this" and derive the associated positive testing. But putting down all the operational experience around "it must not work like this" or "it should handle this error gracefully" is a daunting task most shy away from. Those would give you the grounds to start meaningful negative testing from.

Thanks, Ivy & Nathan for your comments. We unfortunately find the issues mentioned in the article are all too common - and again, unfortunately, common sense as far as putting in place measures in pre-empting or mitigating such failures, is not that common, in our experience.

Budget constraints (59%) seem to be the main reason that financial firms worldwide are not committing to spending more on testing. Yet, inadequate or rushed testing of financial systems to meet a deadline can have disastrous consequences if a failure results. In automated trading, we've seen this happen. As you point out, changing testing procedures to embed quality from the outset, can have positive results such as increased sales and better reputation.

Advanced Trading takes you on an exclusive tour of Abel Noser's New York trading floor, where the agency broker known for transaction cost analysis, is customizing algorithms for the buy side, while growing its fixed income trading and transitions business.