Considering modernization of your legacy code? Let me know if you would like assistance in monetizing your technical debt, devising plans to reduce it and governing the debt reduction process. Click Services for details.

Some of the business ramifications of the $1 trillion in IT debt have been explored in the first post of this two-part analysis. This second post focuses on “an ounce of prevention is worth a pound of cure” aspects of IT debt. In particular, it proposes an explanation why prevention was often neglected in the US over the past decade and very possibly longer. This explanation is not meant to dwell on the past. Rather, it studies the patterns of the past in order to provide guidance for what you could do and should do in the future to rein in technical debt.

The prevention vis-a-vis cure trade-off in software was illustrated by colleague and friend Jim Highsmith in the following figure:

Figure 1: The Technical Debt Curve

As Jim astutely points out, “once on far right of curve all choices are hard.” My experience as well as those of various Cutter colleagues have shown it is actually very hard. The reason is simple: on the far right the software controls you more than you control it. The manifestations of technical debt [1] in the form of pressing customer problems in the production environment force you into a largely reactive mode of operation. This reactive mode of operation is prone to a high error injection rate – you introduce new bugs while you fix old ones. Consequently, progress is agonizingly slow and painful. It is often characterized by “never-ending” testing periods.

In Measure and Manage Your IT Debt, Gartner’s Andrew Kyte put his finger on the mechanics that lead to the accumulation of technical debt – “when budget are tight, maintenance gets cut.” While I do not doubt Andrew’s observation, it does not answer a deeper question: why would maintenance get cut in the face of the consequences depicted in Figure 1? Most CFOs and CEOs I know would get quite alarmed by Figure 1. They do not need to be experts in object-oriented programming in order to take steps to mitigate the risks associated with slipping to the far right of the curve.

The return on assets (ROA) for U.S. firms has steadily fallen to almost one-quarter of 1965 levels.

Similarly, the ROA performance gap between corporate winners and losers has increased over time, with the “winners” barely maintaining previous performance levels while the losers experience rapid performance deterioration.

U.S. competitive intensity has more than doubled during that same time [i.e. the US has become twice as competitive – IG].

Average Lifetime of S&P 500 companies [declined steadily over this period].

However, in those same 40 years, labor productivity has doubled – largely due to advances in technology and business innovation.

Discussion of the full-fledged analysis that Brown derives based on these five facts is beyond the scope of this blog post [2]. However, one of the phenomena he highlights – “The performance paradox: ROA has dropped in the face of increasing labor productivity” – is IMHO at the roots of the staggering IT debt we are staring at.

Put yourself in the shoes of your CFO or your CEO, weighing the five facts highlighted by Brown in the context of Highsmith’s technical debt curve. Unless you are one of the precious few winner companies, the only viable financial strategy you can follow is a margin strategy. You are very competitive (#3 above). You have already ridden the productivity curve (#5 above). However, growth is not demonstrable or not economically feasible given the investment it takes (#1 & #2 above). Needless to say, just thinking about being dropped out of the S&P 500 index sends cold sweat down your spine. The only way left to you to satisfy the quarterly expectations of Wall Street is to cut, cut and cut again anything that does not immediately contribute to your cashflow. You cut on-going refactoring of code even if your CTO and CIO have explained the technical debt curve to you in no uncertain terms. You are not happy to do so but you are willing to pay the price down the road. You are basically following a “survive to fight another day” strategy.

If you accept this explanation for the level of debt we are staring at, the core issue with respect to IT debt at the individual company level [3] is how “patient” (or “impatient”) investment capital is. Studies by Carlota Perez seem to indicate we are entering a phase of the techno-economic cycle in which investment capital will shift from financial speculation toward (the more “patient”) production capital. While this shift is starting to happens, you have the opportunity to apply “an ounce of prevention is worth a pound of cure” strategy with respect to the new code you will be developing.

My recommendation would be to combine technical debt measurements with software process change. The ability to measure technical debt through code analysis is a necessary but not sufficient condition for changing deep-rooted patterns. Once you institute a process policy like “stop the line whenever the level of technical debt rose,” you combine the “necessary” with the “sufficient” by tying the measurement to human behavior. A possible way to do so through a modified Agile/Scrum process is illustrated in Figure 2:

Figure 2: Process Control Model for Controlling Technical Debt

As you can see in Figure 2, you stop the line and convene an event-driven Agile meeting whenever the technical debt of a certain build exceeds that of the previous build. If ‘stopping the line’ with every such build is “too much of a good thing” for your environment, you can adopt statistical process control methods to gauge when the line should be stopped. (See Using 3σ Control Limits in Software Engineering for a discussion of the settings appropriate for your environment.)

An absolutely critical question this analysis does not cover is “Buthow do we pay back our $1 trillion debt?!” I will address this most important question in a forthcoming post which draws upon the threads of this post plus those in the preceding Part I.

Footnotes:

[1] Kyte/Gartner define IT Debt as “the costs for bringing all the elements [i.e. business applications] in the [IT] portfolio up to a reasonable standard of engineering integrity, or replace them.” In essence, IT Debt differs from the definition of Technical Debt used in The Agile Executive in that it accounts for the possible costs associated with replacing an application. For example, the technical debt calculated through doing code analysis on a certain application might amount to $500K. In contrast, the cost of replacement might be $250K, $1M or some other figure that is not necessarily related to intrinsic quality defects in the current code base.

In a recent research note and a corresponding press release, Gartner’s Andrew Kyte assessed the current level of world-wide IT Debt [1] at about $500 billion. Andrew actually considers $500 billion a conservative estimate, expecting it to grow to $1 trillion by 2015. As was pointed out by David Nagel, $1 trillion is more than four times total worldwide enterprise software expenditures this year.

I am publishing two posts in order to put these staggering figures in perspective. This post primarily addresses the business design ramifications of the numbers quoted above. A follow-on post in the coming week will explore the dire repercussions of disregarding the proven practice “an ounce of prevention is worth a pound of cure.” It will also offer an explanation rooted in our business context why this tried and true wisdom has so often been disregarded over the past decade.

The first thing to point out about the Kyte/Gartner figures is that these figures are the cost of fixing, not the value that could be lost due to the myriad malfunctions that a $1 trillion worth of software quality deficits can cause. It is like the loss incurred through fixing the truck in Figure 1 below. The cost of fixing might be $10,000. The corresponding loss of value due to the time it took to carry out the fixing of the truck is vividly captured in the quip written on the back of the sleeper: “Day late $100,000 short.”

If you accept this premise, one real risk of a high level of IT debt is the deterioration of services provided through the software. An even bigger risk, however, is obsolescence of business designs due to the software systems decaying to the point that adding critical services is next to impossible. For example, consider the following B2B eCommerce services for retailers (taken from an unrelated exchange I recently had with my friend Erik Huddleston):

Vendor drop ship

Catalog/data sync

Vendor management

Compliance

It is unlikely a 10-year-old eCommerce software system whose upkeep was neglected for the past decade would have enough changeability left in it to enable providing such services. Lacking these services, the business is likely to revert to outdated designs for generating and recapturing value.

The B2B eCommerce situation discussed above is not really different from the classical dynamics of regression in the development of a child. It is, of course, poignant when a child suffers during one phase or another in his/her development. The bigger poignancy, however, is that the struggling child gets stuck. He/she is unable to move on to the next developmental phase(s). Other children surpass him/her.

What it means in less metaphorical terms is that an incumbent with a significant IT debt might fall behind new entrants who are not (yet?) saddled with such debt. The new entrants can utilize the flexibility of their software to satisfy customer needs in ways that the incumbent’s legacy software will be hard pressed to meet. Moreover, the new entrants can modify their software in response to actual customer feedback in a much faster manner than the ‘neglectful incumbent’ can.

As an incumbent, you need to really start worrying about your IT debt if you accept the inevitability of the transformation driven by the confluence of Cloud, Mobile and Social (see Consumerization of Enterprise Software). No matter what industry you are in, the versatility, modularity, flexibility and mobility of the forthcoming consumerized enterprise software apply to every aspect of your business design. The IT debt you did not ‘pay back’ stands in the way of modernizing your business design.

Footnotes:

[1] Kyte/Gartner define IT Debt as “the costs for bringing all the elements [i.e. business applications] in the [IT] portfolio up to a reasonable standard of engineering integrity, or replace them.” In essence, IT Debt differs from the definition of Technical Debt used in The Agile Executive in that it accounts for the possible costs associated with replacing an application. For example, the technical debt calculated through doing code analysis on a certain application might amount to $500K. In contrast, the cost of replacement might be $250K, $1M or some other figure that is not necessarily related to intrinsic quality defects in the current code base.

Coming on the heels of Gartner’s research note projecting $1 trillion in IT Debt by 2015, CAST’s study provided a more granular view of the debt, estimating an average of over $1 million in technical debt per application in a sample of 288 applications. Between these two studies, the situation examined at the micro-level seems to be quite consistent with the state of affairs estimated and projected at the macro-level.

My hunch is that the gravity of the situation from a software quality and maintenance perspective is actually masked by efforts of IT staffs to compensate for programming problems through operational excellence. For example, carefully staged deployment and quick rollback often enable coping with defects that could/should have been handled through higher test coverage, lesser complexity or a more acceptable level of code duplication.

Part of the reason that the masking effects of IT staffs are not always fully appreciated is that they are embedded in the business design of IT Outsourcing companies. The company to which you outsourced your IT is ‘making a bet’ it can run your IT better than you can. It often succeeds in so doing. The unresolved defects in your old code plus those that evolved over time through software decay have not necessarily been fixed. Rather, the manifestations of these defects are handled operationally in a more efficient manner.

Think again if your visceral reaction to the technical debt situation described in the Gartner research note and the CAST study is of the “This can’t possibly be true” variety. It is what it is – just take a quick look at Nemo to see representative technical debt data with your own eyes. And, as indicated in this post, it might even be worse than what it looks. As Gartner puts it:

The results of such [IT Debt] an assessment will be, at best, unsettling and, at worst, truly shocking.