Because Good Software Is Good Business

Tag Archives: software failure

Post navigation

Software risks to the business, specifically Application Resiliency, headline a recent executive roundtable hosted by CAST and sponsored by IBM Italy, ZeroUno and the Boston Consulting Group. European IT executives from the financial services industry assembled to debate the importance of mitigating software risks to their business.

If you’ve read the news lately, you’ve seen headline after headline (some, even on our blog) about computer glitches, technical failures, software risk, and hacks. The health of applications is now under more microscopic attention than ever before – because no matter whether internal or external causes prompt a software outage, the security and stability of your applications are paramount.

Last Wednesday the Royal Bank of Scotland (RBS) underwent an IT failure which withheld 600,000 payments from customer accounts. This occurs seven months after RBS was fined ₤56 million due to an IT Crash in 2012 that impeded customers from accessing their online accounts. The poor system performance has caused difficulties for customers and shock from the banking community.

There’s a very old mantra around project quality that says, “If you want something done right, do it yourself.”
I disagree.We recently remodeled the bathroom in our master bedroom. Rather than taking my own sledgehammer to the walls, tub and toilet and then hanging my own sheet rock, my wife and I hired a local contractor who came in, did the demolition and reconstruction, and in the end we wound up with a room with which we’re very happy.
I can tell you without reservation that had I done it myself the project would have turned out disastrous because I confess to a certain measure of incompetence when it comes to carpentry…and plumbing…and electrical systems…and just about every other discipline that goes into rebuilding a bathroom.
I guess you could say we had “great expectations” and knew that to achieve them we needed to find someone else to do the job.
Loosing Control
I suspect that this lack of capability in terms of doing something yourself does not always extend to companies when they choose to outsource software builds, but there is some measure of it. The decision to outsource usually comes down to one of two reasons – a company doesn’t have the time to do it or feels an outside group can do it better.
This decision to outsource is being made by an increasingly large segment of the business community. As was recently noted on The Outsourcing Blog, “the public and private sectors alike are becoming increasingly reliant on third-party suppliers to effectively operate.”What is a bit off-putting, however, is the claim made in that post that, “that some 64% of third-parties fail to meet stakeholder expectations and contractual commitments, according to recent research we have undertaken.”
The fact of the matter is, regardless of where a company chooses to outsource, there is a certain relinquishment of control. It is simply neither possible, nor desirable to hold tightly to the reins of all aspects of an outsourced project. When the outsourced project has an offshored element, the potential increase in benefits is met with an equivalent set of risks. Cultural differences and distance alone significantly contribute to increasing both the risks and management costs.
Much of this can be attributed to the fact that organizations have not previously had the means to assess application software quality in real-time when its development has been outsourced. QA plan compliance checks, while useful in some capacities, are normally performed via random manual code reviews and inspections by QA staff. For a typical one million-line-of-code J2EE distributed application, there is significant risk that key issues will go overlooked. Furthermore, standard functional and technical acceptance testing is simply insufficient at detecting severe coding defects that may have impact on the reliability and maintainability of an application. Finally, in the current geopolitical context, programming vulnerabilities, or even hazardous code in a mission-critical application, could easily produce disasters in production – data corruption or losses, system downtime at crucial moments – all of which negatively affect the business operations.
Unfortunately, most IT organizations have chosen to leave the technical compliance issues aside, due to either limited resources are scarce or a lack of the required skills. Instead, they all too frequently assume that tersely worded SLAs will be enough to protect them over time. In reality, while today’s SLAs routinely include financial penalty clauses, fines and legal battles, they are not all that effective in preventing system failures.
Get it Right
In order to be successful, companies need to acquire and deploy software solutions that help manage these global partnerships by providing greater insight into the build process through real-time access to objective data. Employing a platform of automated analysis and measurement to assess the application as it is being built, for instance, affords transparency into the outsourced work, grants discipline into how information is handled and yields metrics to evaluate results.
With that kind of real-time access and information into how a company’s software is being built or customized, it won’t matter if the outsourcer is across the hall, across the street or across the ocean. You will always know just where your software is and if the outsourcer is building it efficiently and up to your high application software quality standards. Not having that kind of insight could lead to software issues that would scare the Dickens out of you!

Isn’t it ironic?
Facebook, the galactically popular social networking site that for so long has weathered friction regarding weaknesses in its software – particularly around security and privacy issues – may have seen its own IPO effort submarined by a software glitch in the NASDAQ stock exchange.
In reporting on NASDAQ’s response to the technical difficulties it encountered on Facebook’s IPO day, Bloomberg’s Nina Mehta writes:
Computer systems used to establish the opening price were overwhelmed by order cancellations and updates during the “biggest IPO cross in the history of mankind,” Nasdaq Chief Executive Officer Robert Greifeld, 54, said yesterday in a conference call with reporters. Nasdaq’s systems fell into a ‘loop’ that kept the second-largest U.S. stock venue operator from opening the shares on time following the $16 billion deal.
According to Mehta, the reason Greifeld gave for the issues with the IPO was “poor design in the software it uses for driving auctions in initial public offerings.”
One would think that if any exchange out there were to be free of poorly designed software it would be the Tech-heavy NASDAQ exchange. Apparently offering the top Tech companies in the industry, though, does not necessarily mean you run the best software the Tech industry has to offer.
Profile of a Failure
Truth is, software failures like the one experienced by NASDAQ have become quite commonplace lately; so much so that they’re practically met with a shoulder shrug and an “oh well.” We treat news of software failures as though they were inevitable and almost expected. Only when it affects finance – particularly the financial status of a marquis brand name like Facebook – do we step back and even offer so much as a “tsk, tsk, tsk” for the failure.

But why? When exactly did we decide that software failure was an unavoidable part of business, an acceptable excuse for possibly undermining the value of a highly touted IPO?
Facebook reached a high of $45 per share before it dropped back below its initial offering price of $38 per share. Whether the glitches at NASDAQ caused the per performance or whether you agree with Henry Blodget at Business Insider that they were just a convenient excuse for a poor showing, there is still no excuse for application software failure, especially since we know what causes it:

Business Blindspot: Regardless of the industry, most developers are not experts in their particular domain when they begin working for a company. It takes time to learn about the business, but most of the learning, unfortunately, comes only by correcting mistakes after the software has malfunctioned.
Inexperience with Technology: Mission critical business applications are a complex array of multiple computer languages and software platforms. Rather than being built on a single platform or in a single language, they tend to be mash ups of platforms, interfaces, business logic and data management that interact through middleware with enterprise resource systems and legacy applications. Additionally, in the case of some long-standing systems, developers often find themselves programming on top of archaic languages. It is rare to find a developer who knows “everything” when it comes to programming languages and those who don’t may make assumptions that result in software errors that lead to system outages, data corruption and security breaches.
Speed Kills: The pace of business over the past decade has increased exponentially. Things move so fast that software is practically obsolete by the time it’s installed. The break-neck speeds at which developers are asked to ply their craft often means software quality becomes a sacrificial lamb.
Old Code Complexities: A significant majority of software development builds upon existing code. Studies show that developers spend half their time or more trying to figure out what the “old code” did and how it can be modified for use in the current project. The more complex the code, the more time spent trying to unravel it…or not. In the interest of time (see “Speed Kills” above) complexity can also lead to “work arounds” leaving a high potential for mistakes.
Buyer Beware: Mergers and acquisitions are a fact of life in today’s business climate and most large applications from large “acquirers” are built using code from acquired companies. Unfortunately, the acquiring organization can’t control the quality of the software they are receiving and poor structural quality is not immediately visible to them.

A Comment on Facebook’s StatusNASDAQ may need to pay back $13 million to investors who should have received transaction executions but did not because of its software failures. Meanwhile, brokers around the world may lose $100 million repaying investors for mishandled orders. A quick, pre-deployment application of automated analysis and measurement to diagnose the structural quality and health issues within the application software used by NASDAQ or any company would have been a much better investment of time and money.
I guess this is one more reason to lobby for a “DISLIKE” button.

After a very mild winter this year, the Northeast part of the country found itself stuck in a prolonged “early spring” where it seemed like but for a couple of days temperatures refused to warm up from the 40’s and 50’s. We seemed to be stuck in the ether between “actual cold” and “comfy warm” for quite a while until the past week or so.
When finally the temperatures turned upwards into the 60’s and 70’s, I happily threw open all my windows in the house to “air the place out.” Apparently, though, the insect population of my neighborhood seemed to be waiting for this moment as well and took my open windows as an invitation to breach the many holes that have somehow sprung up in my screens over the years. On the bright side, I got a good bit of exercise chasing after flying critters with my fly swatter (oh, how I long for the days when environmentalists didn’t guilt us out of using that magical red can of bug spray).
As I went about my business swatting flying insects that had infiltrated my house, I noted a sort of irony in what I was doing. After many posts about fixing issues (i.e., holes) with application software to avoid bugs breaching a company’s infrastructure, I realized that I had failed to heed my own advice. So I did to my screens what any good software developer should do when he realizes there are holes in his software – I fixed them. I haven’t seen a flying critter in the last three days despite my windows all being open.
There’s a Hole in the Bucket
Fixing holes might seem like a logical first line of preventative measures to prevent an outside entity from breaching a company’s software portfolio with bugs…or viruses or other manner of malware. Why is it, then, that many who claim to be “in the know” immediately jump to measures that either help identify a breach but not prevent it, or take a shotgun rather than flyswatter the problem?
In a recent post over at Dark Reading, John Sawyer writes yet another column about what companies need to do to prevent data leaks in their organization. Most of his solutions are necessary, although predictable – encryption, locking down the network and employee education. All of these are necessary elements to any good security system, but as so many security discussions do they leave out one significant element – eliminate the point of infiltration.Encryption is not infallible. As for locking down a network, it’s not only pretty drastic, but if the attack comes from within the organization, a locked down network is about as effective as the infamous Maginot Line that was supposed to have protected France from the Germans in World War II. (For those not into History like I am, the Germans just went around the thing).
Employee education is extremely important, though, and I do credit Sawyer for bringing it up. So much of what trouble’s an organization’s IT portfolio is introduced by its own employees, both knowingly and unknowingly. Nevertheless, if there were no issues with the application software to be breached, data would not be leaked.
So Fix It
Like so many issues with the IT portfolios of today’s companies, problems with application performance – including security breaches – can be traced back to the structural quality of the software. What needs to happen, therefore, is for companies to make themselves more aware of the issues that exist within their portfolios.
To ensure that they are protected at the very core of their IT portfolios, companies need to perform thorough assessments of the applications in them. This base protection platform should start with automated analysis and measurement when software is being either built or customized to ensure that at each stage of the build issues are caught and dealt with before they become problems. As studies have shown, with each successive stage of the build process, these issues become 10 times more difficult to mitigate than if caught in the previous stage.
However, just assessing applications during builds ignores the fact that structural quality standards are constantly increasing. What was sufficient for keeping data secure just a few years ago may no longer be good enough to keep it locked down. This is why companies also should perform periodic analysis of the entire portfolio to identify those applications that no longer measure up to the current standards for optimal structural quality.
So when companies are starting to consider all the things they need to do to secure their IT portfolios, they should think of their homes and remember, the fewer holes there are, the fewer bugs there will be.

This blog has long professed the need for businesses to analyze, measure and assess their IT application portfolios to identify those issues with application software that cause a whole spate of headaches, from application failure, to business risk to increased technical debt.
To this end, Large enterprises have been able to apply an automated solution to perform this assessment has meant installing a comprehensive platform that analyzes and measures existing applications and can also scrutinize application software during the build or customization processes to catch issues as they happen. However, smaller businesses and individual developers generally do not have access to these large platforms and even some larger enterprises have not yet availed themselves of the technology available to them. As a result, those who have yet to adopt automated analysis and measurement platforms find themselves having to do the best they can with manual assessment of their application software, which is grossly inefficient.
Today, however, CAST released a solution for those who do not have access to comprehensive application analysis platforms with the launch of Rapid Portfolio Analysis (RPA).
RPA is a cloud-based SaaS offering that proactively identifies poorly performing applications. It provides an affordable, on-demand alternative to expensive enterprise application portfolio management or project portfolio management solutions, while automatically providing feedback on software health. The information provided by RPA arms executives with the information to determine which applications need further investigation or closer ongoing monitoring for structural issues.With companies spending as much as 70% of their IT budgets on maintenance and support costs, RPA can provide a quick overview of the things that could be ailing a company’s IT portfolio. By identifying those problem areas and then fixing them, CIOs could save their companies significantly by reducing or even eliminating business risk (i.e., security breaches or outages and downtime due to application failure) and excess technical debt, which the CAST Report on Application Software Health (CRASH) identified back in December as reaching an average of over $3.6 million.
At a time where world economics dictate that companies do all they can on trimmed budgets, being able to identify problem areas quickly and relatively inexpensively would certainly allow CIOs to focus more of their budgets on innovation rather than remediation.
So when it comes to identifying problems in IT applications, it seems like “clouding” the issue might actually clear it up.