]]>Decisions to retire the legacy technical debt carried by irreplaceable assets are not to be taken lightly. As decision makers gather information and recommendations from all around the organization, most will discover that information and recommendations aren’t sufficient for making sound decisions about technical debt retirement. The issues are complex. Education is also needed. It’s entirely possible that in some organizations, confronted with a set of decisions regarding legacy technical debt retirement in irreplaceable assets, the existing executive team might be out of its depth. To understand how this situation can arise, let’s explore the nature of legacy technical debt retirement decisions.

A common technical debt retirement scenario

What compels the leaders of a large enterprise to consider retiring the technical debt encumbering one of its irreplaceable assets is fairly simple: cost. Decision makers usually begin by investigating the cost of replacing the asset—the option I’ve oh-so-cleverly called “Replace the Asset.” They then typically conclude that replacement isn’t affordable. At this point, many decision makers choose the option I’ve called “Do nothing.” Time passes. A succession of incidents occurs, in which repairs to the asset or enhancements of the asset are required. And I use the term required here to mean “essential to the viability of the business.”

Two alternatives to retiring legacy technical debt in irreplaceable assets. Neither one works very well.

Engineers then do their best to meet the need, but the cost is high, and the work takes too long. The engineers explain that the problems are due, in part, to the heavy burden technical debt in this particular asset. Eventually the engineers are asked to estimate the cost of “cleaning things up.” Decision makers receive the estimates and conclude that it’s “unaffordable right now.” They ask the engineers to “make do.” In other words, they stick with the Do Nothing option.

After a number of cycles repeating this pattern, decision makers finally agree to provide time and resources for technical debt retirement, but only because it’s the least bad alternative. The other alternatives—Replace the Asset, and Do Nothing—clearly won’t work and haven’t worked, respectively.

So there we are. The organization has been forced by events to address the technical debt problems in this irreplaceable asset. And that’s where the trouble begins.

Decisions about retiring legacy technical debt

In scenarios like the one above, the fundamental decision has already been made: the enterprise will be retiring legacy technical debt from an irreplaceable asset. But that’s just the first ripple of waves of decisions to come, made by many people in a variety of roles throughout the enterprise. Let’s now have a look at a short catalog of what’s in store for such an enterprise.

Recall that most large technical debt retirement projects probably exhibit a high degree of wickedness in the sense of Rittel and Webber [Rittel 1973]. One consequence of this property is the need to avoid do-overs. That is, once we make a decision about how to proceed to the next bit of the work, we want that decision to be correct, or at least, good enough. It should not leave the enterprise in a state that’s more difficult to resolve than the state in which we found it. Since another property of wicked problems is the prevalence of surprises, most decisions must be made in a collaborative context, which affords the greatest possibility of opening the decision process to diverse perspectives. We must therefore regard collaborative decision-making at every level as a highly valued competency.

Strategic decisions

This decision category leads the list because it provides the highest leverage potential for changing enterprise behavior vis-à-vis technical debt. Organizations that are confronting the problem of technical debt retirement from irreplaceable assets would do well to begin by acknowledging that although they might be able to devise tactics for dealing with the debt burdening these assets right now, they must make a strategic change if they want to avoid a recurrence. Accumulating debt to a level sufficient to compel chartering a major debt retirement project took time. It took years of deferring the inevitable. A significant change of enterprise strategy is necessary.

When changing complex social systems, applying the concept of leverage provides a critical advantage. In this instance, following the work of Meadows [Meadows 1997] [Meadows 1999] [Meadows 2008], we can devise interventions at several points that can have great impact on both the level of technical debt and its rate of accumulation. The leverage points of greatest interest are Feedback Loops, Information Flows, Rules, and Goals. For example, the enterprise can set a strategic goal of a specific volume of incremental technical debt incurred per project, normalized by project budget, as I discussed in the post, “Leverage points for technical debt management.”

One might reasonably ask why enterprise strategy must change; wouldn’t a change in technology strategy suffice? Changing how engineers go about their work would help—indeed in most cases it’s necessary. But because the conditions and processes that lead to technical debt formation and persistence transcend engineering activities, additional changes are required to achieve the objective of controlling technical debt.

Organizational decisions

Before chartering a technical debt retirement project (DRP) for an irreplaceable asset, or a group of irreplaceable assets, it’s wise to consider how to embed that project in the enterprise.

The default organizational form for debt retirement projects concerned with an asset A is usually the same form that would be used for major projects focused on asset A. If the Information Technology (IT) unit would normally address issues in A, the debt retirement effort usually would be organized under IT. If A is a software product normally attended to in a product group, that same group would likely have responsibility for the DRP for asset A.

Although these default organizational structures are somewhat sensible, both technically and politically, there’s an alternative approach worth investigating. It entails establishing a technical debt retirement function that becomes a center of excellence for executing technical debt retirement projects, and for developing and injecting sound technical debt management practice into the enterprise. Such an approach is especially useful if multiple debt retirement projects are needed.

The fundamental concept that makes the center-of-excellence approach necessary is the wickedness of the technical debt retirement problem. To address the problem at scale requires capabilities beyond what IT can provide; beyond what product units can provide; indeed, beyond what any of the conventional organizational elements can provide. The reason for this is that the explosion of technical debt in most organizations is an emergent phenomenon. Every organizational unit contributed to the formation of the problem. And every organizational unit must contribute to its resolution.

A technical debt center of excellence is an approach that might be capable not only of synthesizing the expertise of all elements of the enterprise, but also might be capable of bringing new approaches into the enterprise from external sources.

Engineering decisions

Engineers have a tendency to identify and classify technical debt items on technical grounds. Further, they tend to set technical debt retirement priorities on a similar basis. That is, they’re inclined to set priorities highest for those debt items that they (a) recognize as debt items and (b) see as imposing high levels of MICs charged to engineering accounts. Engineers are less likely to assign high priorities to technical debt that generates MICs that are charged to revenue, or to other accounts, because those MICs are less evident—and in many cases invisible—to engineers.

Decisions regarding recognition of technical debt items and setting priorities for retiring them must take technological imperatives into account, but they must also account for MICs of all forms. Priorities must be consistent with enterprise imperatives.

Decisions about pace

Paraphrasing Albert Einstein, technical debt retirement projects should be executed as rapidly as possible, and no faster. The tendency among non-engineers and non-technical decision-makers is to push for rapid completion of debt retirement projects, for three reasons. First, everyone, like the engineers, wants the results that debt retirement will bring. Second, everyone, like the engineers, wants an end to the inevitable disruptions debt retirement projects cause. And finally, the longer the project is underway, the more it might cost.

For these reasons, once the decision to retire the debt is firmly in hand, the enterprise might have a tendency to apply financial resources at a rate that exceeds the ability of the project team to execute the project responsibly. When that happens, rework results. And for wicked problems like debt retirement, rework is the path to catastrophe.

Decisions about pace and team scale need to be regarded as tentative. Regular reviews can ensure that the resource level is neither too low nor too high. Even when the engineers are given control over these decisions, they must be reviewed, because pressures for rapid completion can be so severe that they can compromise the judgment of engineers about how well they can manage the resources applied to the project.

Resource decisions

Debt retirement projects concerned with legacy irreplaceable assets are different from most other projects the enterprise undertakes. Estimates of the labor hours required are more likely to be incorrect on the low side than are analogous estimates for other projects, because so much of the work involves pieces of assets with which few engineering staff have any experience. But with respect to resources, underestimating labor requirements isn’t the real problem. Non-labor resources are the real problem.

Because the assets are irreplaceable, it’s likely that they’re needed for ongoing operations. In some cases, the assets are needed continuously. Many organizations have kept such assets operational by exploiting hours of downtime during periods of low demand, usually scheduled and announced in advance. While these practices are likely sufficient for the relatively minor and infrequent changes usually associated with routine maintenance and enhancement, debt retirement imposes much more severe burdens on the organization than these short access windows can support. Effective debt retirement projects need far more access to the asset—a level of access that continuous delivery practices can provide [Humble 2010].

However, assets whose designs predate the widespread use of modern practices such as continuous delivery might not be compatible with the infrastructure that these practices require. And in organizations that haven’t yet adopted such practices, staff familiar with them might be in short supply. For these reasons, we must regard as developmental any early projects whose objectives are retiring technical debt from irreplaceable assets. They’re retiring the technical debt, of course, but they’re also developing the practices and infrastructure needed to support technical debt retirement projects. This dual purpose is what drives the surprisingly high non-labor costs and investments associated with early technical debt retirement projects.

The investments required might include such “items” as a staging environment, which “is a testing environment identical to the production environment” [Humble 2010]; extensive test automation, including results analysis; blue-green deployment infrastructure; automation-assisted rollback; and zero-downtime release infrastructure. Decisions to make investments require an appreciation of their value to the enterprise. They enable the enterprise to deal effectively with the wicked problem of technical debt retirement.

Last words

Because every situation and every organization is unique, few general guidelines are available for making these decisions. The criteria most organizations have been using for dealing with (or avoiding) the issue of technical debt have produced the problems they now face. So, to succeed from this point, whatever criteria they use in the future must be different. My own view is that short-term thinking is at the heart of the problem, but it’s a wicked problem. The long-term solution will not be simple.

]]>https://techdebtpolicy.com/legacy-technical-debt-retirement-decisions/feed/02407Rules of engagement for auxiliary technical debthttps://techdebtpolicy.com/rules-of-engagement-for-auxiliary-technical-debt/
https://techdebtpolicy.com/rules-of-engagement-for-auxiliary-technical-debt/#respondMon, 03 Dec 2018 21:03:54 +0000https://techdebtpolicy.com/?p=2373When retiring one kind of technical debt from an asset, auxiliary technical debt is any other kind of technical debt in the asset. It can be tempting to try to retire auxiliary technical debt too. Sometimes that’s wise, but it can lead to scope creep. Rules of engagement can control this temptation.

]]>As noted in an earlier post, a technical debt retirement project (DRP) is a project whose primary objective is retirement of a particular kind of technical debt—or particular kinds of technical debt—from a specified set of assets. But those assets might also carry other kinds of technical debt. With respect to a given DRP, we can call these other kinds of technical debt Auxiliary Technical Debt(ATD). Because the presence of ATD can defocus debt retirement projects, it presents a risk that must be anticipated and well understood, if it is to be mitigated.

This post explores concepts and approaches for mitigating the risks associated with the auxiliary technical debt (ATD) of a given technical debt retirement project (DRP). As might already be evident, these initialisms (ATD, DRP, and one more to come) can be difficult to keep straight. Here’s a quick guide: T always means Technical, D always means Debt, R always means Retirement, and P always means Project. Also, if you have a pointing device, and you hover the cursor over the first mention of each initialism in each paragraph, your browser displays the expansion of the term. Touch screen users and keyboarders: sorry, I haven’t yet figured out how to help you in an analogous way, so let me know if you have an idea.

The temptation to retire auxiliary technical debt

Guardrails in a track bed as a rail line crosses a bridge. The guardrails are the inner pair of rails. The rails outside the inner pair are the running rails. Guardrails (also known as check rails) function to keep the wheels of derailed cars from straying too far from their proper locations. This is a useful risk mitigation function in high-risk geometries such as curves. It’s also advantageous even if the probability of risk events is low, as in this straight section of track. It’s a worthwhile measure when the consequences of risk events are extremely costly, as in this case. A derailment on a railway bridge or in steep terrain can result in rail vehicles falling to the earth below, which can cause them to pull other vehicles with them. Derailments under highway overpasses can also be problematic. Such derailments can result in damage to rail or highway bridge structures, resulting in loss of service for periods extending far beyond the time needed to clear the derailment. For this same reason, guardrails are also used in tunnels and tunnel approaches. Because uncontrolled scope expansion can have such devastating effects, we need policy guardrails to control scope expansion when retiring technical debt from assets that contain auxiliary technical debt.

I’ve been using the term TDIQ—Technical Debt In Question—to denote the kinds of technical debt whose retirement is the objective of a given DRP. The ATD of that DRP, then, is the collection of instances of any other kinds of technical debt, of types differing from the TDIQ of the DRP, and which are present in the assets being modified by the DRP. Notice that the property of being auxiliary technical debt is relative. It’s relative to the objectives of a given DRP. A particular instance of technical debt might be ATD for one DRP, and TDIQ for another DRP, depending on the respective objectives of each DRP. Notice also that the ATD of a given DRP can include several different kinds of technical debt.

Let’s now examine a scenario in which ATD can generate risk for a DRP. In this scenario, we’ll consider only one kind of ATD; call it ATD0.

Suppose that several members of the DRP team undertake work to retire the DRP’s TDIQ in a portion of one of the debt-bearing assets. In performing this work, they encounter some instances of ATD0. Studying these instances of ATD0 carefully, they conclude that “fixing” the ATD0 along with the TDIQ in that portion of the asset would be easier and less risky than leaving the ATD0 in place and attending only to the TDIQ. Let’s call their approach the ATD approach. And let’s say that the TDIQ approach is one in which the team addresses only the TDIQ, and leaves in place the ATD0 and all other ATD it finds.

Compared to the TDIQ approach, the advantages of the ATD approach are fairly clear. After the work is complete, in either approach, the asset must be tested and re-certified. In the TDIQ approach, when a subsequent DRP is chartered to retire ATD0, that second DRP team will need to test and re-certify the asset again when it completes its work. In the ATD approach, we can avoid modifying, re-testing, and re-certifying the asset a second time, if we’ve already retired all instances of ATD0 from the asset. Thus, in the ATD approach we can avoid a second round of modification, testing, and re-certification.

Risks associated with retiring auxiliary technical debt

But the ATD approach also has some serious disadvantages.

Enterprise assets might be left in a mixed state

Unless the team plans to retire all instances of ATD0, then upon completion of the DRP, enterprise assets will be in a mixed state. Some will be free of both the TDIQ and ATD0; some will be free of the TDIQ but continue to harbor ATD0. This non-uniformity can create complications for subsequent maintenance, documentation, testing, training, enhancement, automation assist development, and so on.

Complications in testing and re-certification

If test results for the modified assets indicate the possibility of new defects, the cause might be associated with the TDIQ work, or the ATD work, or both. Resolving any issues in the test results is thus more complicated under the ATD approach than it is under the TDIQ approach. Similar considerations affect re-certification. Thus, there is a risk that the ATD approach will complicate interpretation of test and re-certification results.

Questions about the reliability of technical debt inventory data

As noted in an earlier post, for any given DRP, the DRP team needs to know which assets bear that project’s TDIQ. In the TDIQ approach, any data previously or concurrently gathered about the location of instances of ATD0 remains valid, because the TDIQ approach doesn’t retire any instances of ATD0. However, in the ATD approach, such inventory data must be corrected to account for the retirement of whatever instances of ATD0 are retired in the ATD approach. Thus, if ATD0 inventory data has already been collected, or if it’s being collected in parallel with the DRP, the DRP team must take steps to adjust the inventory data regarding locations of ATD0 as it retires instances thereof. There is of course a risk that this will not occur as needed, which can create problems for any subsequent DRP for which the ATD0 is contained in its TDIQ. This can be especially challenging if there are multiple DRPs in process simultaneously, each working on different TDIQs, potentially in different debt-bearing assets, but all encountering and retiring instances of ATD0.

Unconstrained scope creep

Suppose there is a DRP whose objective is retiring its TDIQ, and that it has decided to also retire some (or all) instances of a particular kind of ATD, say ATD0. Although that activity would represent an expansion of scope beyond retiring the TDIQ, it might be acceptable and it might even be prudent. But as the team undertakes to retire ATD0, it might confront a similar quandary relative to the relationship between the ATD0 and yet another kind of ATD, which we might call ATD1. The DRP team might then decide to expand scope again. And so on. In general, there is no self-evident stopping point for such a chain of scope expansion. In these circumstances, scope creep can become an unmitigated risk, threatening the coherence and focus of the DRP, with consequences for its budget and schedule.

Last words

In some cases, some of the ATD might be so intertwined with the TDIQ that retiring some instances of the TDIQ necessarily retires some of the ATD. And in other cases, leaving the ATD in place severely complicates retiring the TDIQ. In still other cases, leaving the ATD in place leaves the assets in a complex state that makes ongoing maintenance or enhancement work more difficult. In these cases, what I called the ATD approach above is plainly the wiser course, compared to the TDIQ approach.

Policymakers have a role to play here. They can develop guidance for DRP teams to apply as they come upon these difficult situations to help them decide whether to take the ATD approach or the TDIQ approach. The military calls this guidance “rules of engagement,” while politicians call it “guardrails.”

Deciding between the ATD and TDIQ approaches on a whim, or on what feels right at the moment, inevitably leads to a chaos of inconsistency and scope creep. The safest course is to adopt wise policy—rules of engagement—and to adjust them as the organization learns more and more about retiring technical debt from its assets.

]]>https://techdebtpolicy.com/rules-of-engagement-for-auxiliary-technical-debt/feed/02373Technical debt retirement: where is the technical debt?https://techdebtpolicy.com/where-is-the-debt/
https://techdebtpolicy.com/where-is-the-debt/#respondSat, 01 Dec 2018 13:08:48 +0000https://techdebtpolicy.com/?p=2346To retire technical debt, we need to know where it is. And if service disruptions are necessary, we need to know who will be affected. Here’s a survey of some of the issues, and suggestions for resolving them.

Determining which of the enterprise’s many technological assets might be carrying the Technical Debt In Question (TDIQ) can be a complex exercise in itself, because inspecting the asset might be necessary. Inspection might require temporarily suspending operations, or determining windows of time during which inspection can be performed safely and without interfering with operations. Further, inspection might require knowledge of the asset that the DRP team doesn’t possess. Moreover, access to the asset might be restricted in some way. In these cases, staff from the unit responsible for the asset must be available to assist with the inspection.

Although asset inspection might be necessary or preferable, it might not be sufficient for determining which assets are carrying the TDIQ. This is easy to understand for physical assets, like, say, determining the release version of the firmware of the hydraulic controller electronics of a tunnel boring machine. But asset inspection might also be insufficient for purely software assets. For determining the presence of the TDIQ in software assets, reading source code might not be sufficient or efficient. It might be easier, faster, and more accurate to operate the asset under special conditions. For example, an inspector might want to provide specific inputs to the asset and then examine its responses. As a second example, we might use automation assistance to examine the internal structure of the asset, searching for instances of the TDIQ. And as with other assets, the assistance of the staff of the unit responsible for the asset might be necessary for the inspection.

Which enterprise operations depend on debt-bearing assets?

Knowing which assets bear the TDIQ is useful to the DRP team as it plans the work to retire the TDIQ. But part of that plan could include service disruptions. If so, it’s also necessary to determine how those disruptions might affect operations, to enable the team to control the effects of the disruptions, and negotiate with affected parties. Thus for each asset that bears the TDIQ, we need to determine what operations would be affected if the asset is removed from service temporarily.

Observing actual operations in conditions in which the asset is out of service in whole or in part might be the only economical way to discover which enterprise functions depend on the assets that carry the TDIQ. Other techniques include examining historical data such as trouble reports and outstanding defect lists, and correlating them across multiple asset histories and operations histories.

In some cases, these investigations produce results that have a limited validity lifetime, owing to ongoing evolution of the debt-bearing assets and the assets that interact with them. For that reason, the actual work of retiring the TDIQ must begin as soon as possible after the inventory is complete, and possibly even before that. This suggests that the size of the DRP team is a critical success factor, because size enables the team to complete the inventory inspections rapidly, before the end of the validity lifetime of the team’s research results.

Managing teams of great size is a notoriously difficult problem. For this reason, delegating some of the DRP research effort directly to the business units that own the assets in question can provide the labor hours and expertise needed for the research. In this way, the DRP can deploy a team-of-teams structure, known as a Multi-Team System (MTS) [Mathieu 2001] [Marks 2005]. The DRP team can then bring to bear a large force in a way that renders the overall MTS manageable.

Before designing a project to retire some portion of the technical debt borne by a critical, irreplaceable asset, it’s best to acknowledge that the project design problem is very likely a wicked problem in the sense of Rittel and Webber [Rittel 1973]. (See my post “Retiring technical debt can be a wicked problem”) In the series of posts of which this is the first, I suggest some basic preparations that form a necessary foundation for success in approaching the problem of designing projects to retire technical debt in irreplaceable assets.

A map of the U.S. Interstate Highway System. The map shows primary roadways, omitting most of the urban loop and spur roads that are actually part of the system. In 2016, the total length of highways in the system was about 50,000 miles (about 80,000 km). About 25% of all vehicle miles driven in the U.S. are driven on this system. The cost to build it was about USD 500 billion in 2016 currency. Given the advances since the 1950s in technologies such as rail, electronics, data management, and artificial intelligence, and given the effects of petroleum combustion on global climate, one wonders whether such a system would be the right choice if construction were to begin today. If alternatives would be better, then this system might be regarded as technical debt. But replacing it might not be practical. Finding a way to retire the technical debt without replacing the entire asset might be the most viable solution. Image by SPUI courtesy Wikipedia.

As I’ve noted in previous posts, the problems associated with retiring technical debt can be wicked problems. And if some of these problems aren’t strictly wicked problems, they can possess many of the attributes of wicked problems in degrees sufficient to challenge the best of us. That’s why approaching a technical debt retirement project as you would any other project is a high-risk way to proceed.

For convenience and to avoid confusion, in my last post I adopted the following terminology:

DRP is the Debt Retirement Project

DDRP is the effort to design the DRP

DBA be the set of Debt Bearing Assets undergoing modification in the context of the DRP

IA is the set of assets, excluding the DBA assets, that interact directly or indirectly with assets in the DBA

In the posts in this thread, convenience demands that we add at least one more shorthand term:

TDIQ is the Technical Debt In Question. That is, it’s the kind of technical debt we’re trying to retire from the assets among the DBA. Other instances of the TDIQ might also be found elsewhere, in other assets, but retiring those instances of the TDIQ is beyond the scope of the DRP.

Know when and why we need to retire technical debt

For those technical debt retirement projects that exhibit a high degree of wickedness, clearly communicating the mission of the DRP is essential to success. The DRP team will be dealing with many stakeholders who are in the early stages of familiarity with the term technical debt. Some of them might be cooperating reluctantly. Expressing the objectives and benefits of the DRP in a clear and inspiring way will be very helpful. With that in mind, I offer the following reminder of the reasons for tackling such a large and risky project that produces so few results immediately visible to customers.

Examining alternatives to retiring the TDIQ is a good place to begin. One alternative is simply letting the TDIQ remain in place. Call this alternative “Do Nothing.” A second alternative to retiring the TDIQ is replacing the debt-bearing asset with something fresh and clean and debt-free. Call this alternative “Replace the Asset.” The problem many organizations face is that they cannot always rely on these alternatives. In some circumstances, the only viable option is debt retirement. And because these two alternatives to debt retirement aren’t always practical, some organizations must develop the expertise and assets necessary to retire widespread technical debt in large, critical, irreplaceable systems. Below is a high-level discussion of these alternatives to debt retirement.

Do Nothing

The first alternative is to find ways to accept that the DBA will continue to operate in their current condition, bearing the technical debt that they now bear. This alternative might be acceptable for some assets, including those that are relatively static and which need no further enhancement or extension. This category also includes those assets the organization can afford to live without.

One disadvantage of the “Do Nothing” approach is that technology moves rapidly. What seems acceptable today might be, in the very near future, old-fashioned, behind the times, or non-compliant with future laws or regulations. Styles, fashions, technologies, laws, regulations, markets, and customer expectations all change rapidly. And even if the asset doesn’t change what it does, the organization might need to enhance it in ways that become very expensive to accomplish due to the technical debt the asset carries.

For these reasons, Do Nothing can be a high-risk strategy.

Replace the Asset

The second alternative to retiring the TDIQ is to replace the entire asset. For this option, the question of affordability arises. In some instances this alternative is practical, but for many assets, the organization simply cannot afford to purchase or design and construct replacements. And for those assets that “learn”, and which contain data gathered from experience over a long period of time, retiring the asset can require developing some means of recovering the experience data and migrating it to the replacement asset—a potentially daunting effort in itself.

Replacement is especially problematic when the asset is proprietary. If the organization created the asset itself, they might have constructed it over an extended period of time. Replacement with commercial products will require extensive adaptation of those products, or adaptation of organizational processes. Replacement with assets of its own making will likely be costly.

Thus, when organizations depend on assets that they must enhance or extend, and which they cannot afford to replace in their entirety, they must develop the expertise and resources needed to address the technical debt that such assets inevitably accumulate.

This series of posts explores the issues that arise when an organization undertakes to retire the technical debt that its irreplaceable assets are carrying. Below, I’ll be inserting links to the subsequent posts in this series.

]]>https://techdebtpolicy.com/technical-debt-in-irreplaceable-assets/feed/02326Nine indicators of wickednesshttps://techdebtpolicy.com/nine-indicators-of-wickedness/
https://techdebtpolicy.com/nine-indicators-of-wickedness/#respondThu, 01 Nov 2018 21:02:23 +0000https://techdebtpolicy.com/?p=2274By carefully observing what happens when we actually try to retire some kinds of technical debt, we can better understand the degree of the degree of wickedness of the effort. That understanding helps manage risk in technical debt retirement projects, reducing costs and speeding execution.

Several properties of the problem of designing technical debt retirement projects tend to make those design problems more likely to be wicked problems—that is, more likely to satisfy all ten of the criteria of Rittel and Webber [Rittel 1973]. I call these properties indicators of wickedness.

The interchange between Interstate 35 and U.S. Route 30, just outside Ames, Iowa. The new flyover ramp is indicated in red. It replaces the cloverleaf ramp in the upper right quadrant of the cloverleaf. A construction error has forced a delay, while the piers of the flyover ramp bridge are corrected. The cloverleaf, which was designed with curves a bit too tight, was a high-accident area, and thus constituted a technical debt, with the ongoing vehicle accidents comprising the metaphorical interest charges. The construction error in the flyover ramp piers necessitated a rollback. Rollbacks often indicate wickedness in the projects to design technical debt retirement projects, but in this case, the indicated wickedness is not much greater than was anticipated by the project designers. Construction drawing by Iowa Department of Transportation [Iowa DOT 2016].Although we usually have some notion of the degree of wickedness of a given design effort for a technical debt retirement project, actually executing the debt retirement project can reveal unanticipated issues and complexity. Some of what’s revealed can cause us to adjust our estimate of the degree of wickedness of the design effort. If we know in advance what kinds of revelations are most likely to cause such adjustments, we can reduce the incidence of unanticipated revelations.

As I noted in my post, “Degrees of wickedness,” we can regard all problems as lying on a Tame/Wicked spectrum, with wicked problems lying at the extreme Wicked end of the spectrum, and the tamest of the tame lying at the opposite end. As for the ten criteria of wickedness developed by Rittel and Webber, I proposed that they could be satisfied in degrees, with the most wicked problems satisfying all ten criteria absolutely.

As a quick review, here are the attributes of wicked problems as Rittel and Webber see them [Rittel 1973], rephrased for brevity:

There is no clear problem statement

There’s no way to tell when you’ve “solved” it

Solutions aren’t right/wrong, but good/bad

There’s no ultimate test of a solution

You can’t learn by trial-and-error

There’s no way to describe the set of possible solutions

Every problem is unique

Every problem can be seen as a symptom of another problem

How you explain the problem determines what solutions you investigate

The planner (or designer) is accountable for the consequences of trying a solution

Below is a sample of conditions or situations that tend to increase the wickedness of the problem of designing a technical debt retirement project. I have no data to support these conjectured effects. But the principles I used to generate them are three:

If a phenomenon expands the set of stakeholders in a debt retirement project, it tends to enhance the wickedness of the design problem.

If a phenomenon increases the number or heterogeneity of the assets or processes that must be considered, it tends to enhance the wickedness of the design problem.

If a phenomenon creates a need for a rollback of work performed as part of the debt retirement project, and that rollback creates a need to re-design the debt retirement project, it tends thereby to enhance the wickedness of the design problem.

In what follows, I use the term “DRP” to indicate the Debt Retirement Project itself, as distinguished from the effort to design the DRP, which I refer to as “DDRP.” The problem whose wickedness we’re considering is not the DRP itself, but the DDRP. Also, let DBA (for debt-bearing assets) be the set of assets undergoing modification in the context of the DRP, and let IA (for interacting assets) be the set of assets, excluding the DBA assets, that interact directly or indirectly with assets in the DBA.

With all this in mind, I offer the following nine examples of indicators of wickedness of the DDRP.

1. A previous attempt to retire this debt was abandoned

Perhaps the most significant indicator of the wickedness of the DDRP is either the failure of a previous attempt to execute a DRP with similar objectives, or the failure of a previous attempt to execute a DDRP for a DRP with those objectives. There are two reasons why such failures are significant indicators of wickedness.

First, it’s reasonable to assume that these previous attempts weren’t founded on any recognition of the wickedness of the DRP or the DDRP. Few such efforts are. (A Google search for the two phrases “technical debt” and “wicked problem” yields less than 1000 results) (update 12 Nov 2018: 1160 results) Consider first the DDRP. If it is a wicked problem, proceeding as if it were not would very likely fail. If the designers of the previous DDRP did assume that it was a wicked problem, investigating their approach could prove invaluable, and save much time and effort. An analogous argument applies for the DRP itself.

Second, if the previous attempt to execute a DRP with similar objectives has left traces of itself in the DBA, and if those traces must be taken into account while executing the DDRP, they might complicate the DRP, and they might be incompletely addressed in the DDRP. To the extent that these conditions prevail, Criterion 5 is satisfied, and the DDRP exhibits wickedness.

2. Some revenue streams need to be interrupted

If the work of the DRP entails temporary interruption of revenue streams, executing the DRP can have significant and long lasting effects on the organization. In estimating the cost of the DRP, it’s clearly necessary to account for the financial impact of any revenue shifted into the future, and any revenue irretrievably lost as well. And in some cases, market share might also suffer. All of these factors tend to increase the wickedness of the DDRP.

When these effects are expected, political opposition to the DRP can develop. Senior management can prevent this opposition from halting the DDRP inappropriately by requiring that the business case for the DRP include these financial factors and demonstrate clearly the need to proceed despite them. Involving potential political opponents of the DRP in business case development can be an effective means of ensuring the strength of the business case.

The ability to model all these financial effects is an important organizational asset that can be developed and maintained, for deployment across multiple DDRPs. The organization can monitor DRPs, gathering actual experience data for comparison to the effects projected in the respective business cases of the DRPs. Those comparisons are useful for enhancing the modeling capability.

3. Some assets not directly touched need to be re-tested

The DDRP is more likely to be a wicked problem if, as a result of the changes executed in the DRP, any of the assets in IA need to be re-tested after or during DRP execution. The need to re-test any assets in IA typically arises when there’s some risk that the DRP’s changes in the DBA could somehow affect the performance of the assets in IA, and when the consequences of such a risk event are severe.

This scenario enhances of the wickedness of the DDRP for at least five possible reasons.

Baseline testing of IA is necessary to enable the DRP team to recognize the effects of the DRP on IA behavior. But this baseline testing can reveal pre-existing and unaddressed faults. Because leaving those faults in place can seriously complicate interpretation of anomalies that appear in IA assets after DRP work has begun, the DDRP team might insist that the owners of the IA assets in question address some of these faults. With regard to these issues, political differences between the DDRP team and the owners of IA assets are possible.

The additional testing of IA assets tends to expand dramatically the set of stakeholders affected by the DRP, to include the owners, users, and maintainers of the IA assets.

The additional testing of IA assets can increase the need to interrupt revenue streams temporarily, and increase the number, duration, and frequency of such interruptions.

The additional testing of IA assets can require expertise and staffing beyond the DRP project team, which can disrupt other elements of the organization as the people needed are temporarily assigned to IA testing.

The additional testing of IA assets can reveal unanticipated consequences of the DRP alterations, which can trigger re-planning or re-design of the DRP during its execution. That re-planning or re-design, in turn, can trigger alterations in the DDRP.

The need to re-test assets not directly touched in the DRP is more likely when the DRP alters the external behavior of any of the DBA assets. The goal of many DRPs is improvement of the internals of assets without altering their external behavior, except possibly for performance improvements. This goal is desirable because it limits the need for re-testing and re-certification of IA assets. However, some kinds of technical debt appear in the externals of the DBA assets, including their architecture, behavior, appearance, or interfaces. Compared to DRPs that do not alter the externals of DBA assets, retiring the kinds of technical debt that alter the externals of DBA assets is inherently more difficult and more risky because it requires more extensive re-testing and re-certification of both DBA and IA assets.

4. Multiple sites are directly touched

When the DRP entails modification of technological assets of geographically dispersed organizations, one consequence is a tendency to increase the wickedness of the DDRP. This comes about because of factors including the following:

Sites might be dispersed not only geographically, but they might also be separated by language boundaries, legal jurisdictions, cultural divides, time zones, and much more. The required work can vary from site to site for technical reasons and because of variations arising from these non-technical factors.

The multiple sites might have different landlords, with different lease agreements governing the organization’s occupancy of the property. This is just one of many factors that increase the numbers of stakeholders and exacerbate their heterogeneity. And the leases might constrain the kind of work that can be performed according to the day of the week or time of day.

If local vendors provide services such as communications or Internet connections to some of the sites, and if the work of the DRP involves these technologies and the local vendors, the task of coordinating all the different players can be complex and riddled with unanticipated obstacles.

For example, if the work involves networking hardware and software, work that we might prefer to perform at night or on a weekend might need to be carried out during business hours for some of the sites. For a global enterprise, there might not be a time of day when no sites are conducting regular business.

As a second example, consider a network upgrade for the retail branch offices of a global bank. If that upgrade requires trenching for new cable connections, the project design must take into account local regulations governing the trenches, including how they must be permitted, dug, re-filled, covered, and marked while still open. These regulations vary with national and sometimes local jurisdiction. The complexity causes most organizations to rely on local vendors, but even then, the vendor selection process must include reliable vendor assessment and evaluation. Scheduling becomes a complex and risky endeavor.

For these reasons, a DDRP that involves technological assets housed at multiple sites geographically dispersed has an elevated probability of exhibiting the properties of a wicked problem.

Another driver of stakeholder expansion is the need for re-certification of assets after they’ve been modified in the course of executing the DRP. The certification agencies can range from local and municipal regulators to national regulators and pan-industrial standards organizations. The sheer number of possibilities is itself a contributor to increased wickedness, but the nature of the operating style of these organizations merits special notice.

For the most part, these agencies operate without competitors, whether they are government elements or private organizations. Perhaps for this reason, “customer service” might not be their strength, and gaining timely cooperation from them might be a challenging undertaking. Even though re-certification might be a small part of the DRP, it can easily become a blocking obstacle. Researching these requirements and their associated lead times, and maintaining a current knowledge base about them, can be a non-trivial task of the DDRP.

6. Non-technical stakeholders must change their behavior

Generally, people don’t like to change how they do their jobs. There are exceptions, of course, if they recognize a benefit that arrives in some immediate and visible way. But unless there is a recognizable benefit, requiring people to change their work patterns as part of a DRP is likely to increase the wickedness of the DDRP. And the difficulty is more problematic if the people affected are technically unsophisticated, because they’re less likely to appreciate the value of managing technical debt, and less likely to accept explanations of that value when those explanations are offered.

DDRP wickedness increases in this case because, in addition to retiring the technical debt, it must address the tasks of motivating and training the affected population. That requires preparing materials, scheduling and accounting for the time spent in training, and monitoring training effectiveness. The business case must also address these issues, but in addition, it must provide evidence required to defuse any political opposition that might otherwise develop.

7. Discovery of major unanticipated complexity triggers re-design

Unanticipated complexity happens in almost every project of almost any kind. But for DDRPs, unanticipated complexity significant enough to trigger significant adjustment or re-design of the DRP during execution of the DRP is another matter. Such a discovery can mean that the DBA assets or their connections to the IA assets have changed since the plan was developed. Or it can mean that the design team had an incomplete or incorrect understanding of the problem they were trying to solve. These events can occur for a number of reasons.

Technical causes are perhaps more easily imagined, so I’ll focus on non-technical causes, which can actually be more serious. For example, suppose that a political alliance enabled the VP of Sales and the VP of Engineering to reach a deal that enabled the DRP team to work on some DBA assets critical to the Sales function, by taking them off line for defined periods. If that political alliance weakens, or if the deal between the two VPs collapses for some other reason, the scheduled downtime of those assets might vanish or be shortened dramatically. This pattern is more likely to arise in situations in which the DDRP team is not a party to such agreements. The DDRP team must be a party to any agreements regarding access to assets by the DRP team.

As a second example, consider what happens when the enterprise undertakes an acquisition that wasn’t revealed to the DDRP team during their design effort. Because chances are good that the DDRP would have a significant amount of re-work to do in such cases, the DDRP team must be kept informed of any organizational changes that could affect the DRP, for the active life of the DRP.

Re-designing the DDRP can take time. Elements of the DDRP that have short shelf lives must be revisited during re-design. And the need to re-design can also indicate gaps in the DDRP team’s understanding of the problem.

All of these conditions tend to move the DDRP in the direction of increased wickedness.

8. Weekend or middle-of-the-night work periods are required

The need to perform critical operations on weekends or in nighttime hours suggests three things. First, the work is risky in the sense that undetected faults that go into production can lead to costly operational errors. Second, the organization lacks a simulated operating environment that emulates the actual operating environment faithfully enough to enable detection of errors before deployment. Third, and finally, the organization lacks a rapid rollback mechanism that can restore the original state of an asset if the new modified state proves problematic when deployed.

These last two factors—the lack of a simulated operating environment and the lack of a rapid rollback mechanism—should be corrected if multiple DRPs are anticipated. Cost is usually the blocking issue. However, that cost must be compared against the cost of retarding all future DRPs, and the cost of any operational failures arising from deploying faulty systems.

Continued refusal to provide a simulated operating environment with rapid rollback increases the wickedness of this and any future DDRPs.

9. Rollback(s) of attempted changes triggers re-design

A rollback is an incident in which, in the course of executing the DRP, it has become necessary to revert some (or all) of the work that has already been performed. Minor rollbacks do happen. But a major rollback, or a rollback the necessity of which is discovered long after completion of the work in question, could be an indication of a deep misunderstanding of the consequences of the work involved. Because that misunderstanding could have consequences not yet recognized, such a rollback could suggest that the wickedness if the DDRP had been underestimated.

Let DBAf (faulty DBA) be the set of assets in the DBA that formerly contained some of the debt being retired, and which had been altered as part of the DRP, and whose alterations contained or led to exposure of some kind of fault(s) that forced a rollback after they were deployed. Let DBAfw (wicked-faulty DBA) represent the subset of DBAf for which that rollback did trigger a re-design of the DDRP. Then wickedness of the DDRP is correlated with the size of DBAfw and the extent of the DDRP re-design that the rollback triggered.

For example, let Efw be a member of DBAfw. And suppose that Efw is a modular element of a system that monitors the click-through behavior of users of a Web site. It records data for later analysis, and because of the fault it does so incorrectly. When the errors are discovered, the module is withdrawn and replaced by the original, unaltered, debt-bearing form. Because Efw contaminated the original database, data rollback is impossible. The error was discovered, but there is no way to re-capture the data that has been lost. That’s why the DDRP (and after that, possibly the DRP) must be re-designed. This scenario is an example of Criterion 5. If there are political consequences for the loss of data, it could be an example of Criterion 10.

This example suggests how the frequency of incidents that trigger re-design of the DDRP can be an indicator of the wickedness of the DDRP.

Example of a non-indicator: the I-35 SR-30 interchange near Ames, Iowa

Just outside Ames, Iowa, is an interchange between Interstate 35 (a four-lane, divided, limited-access roadway) and U.S. Route 30 (also four-lane, divided, but not limited-access). The interchange is a conventional cloverleaf design. The “leaves” are rather tight, though, and consequently, there have been numerous rollovers and crashes at this interchange. We can regard these tight cloverleaf ramps as technical debt in the highway system, and the rollovers and crashes as metaphorical interest charges on that debt.

In 2016, construction began on a new “flyover” exit ramp from northbound Interstate 35 onto westbound U.S. Route 30. The objective was to reduce the number of accidents at the interchange by replacing the current tight-curvature cloverleaf ramp with a flyover exit ramp with a longer radius of curvature. We can regard this project as a Debt Retirement Project (DRP). The project that planned that DRP was an effort to Design a Debt Retirement Project (DDRP).

Completion of the DRP was scheduled for November 2018. When completed, the new ramp will replace the northeast leaf of the cloverleaf. Like most civil engineering projects, this project does have some elements of wickedness, but they were dealt with effectively. Nevertheless, a construction error is delaying completion [Magel 2018] [Iowa DOT 2018]. The error involves the height and position of the bolt anchors where steel bridge beams will connect to the concrete piers of the new flyover ramp. Six piers have been constructed to support the flyover. Those piers are being corrected by jackhammering the concrete tops, leaving the steel reinforcement in place. Then the beam anchors are positioned correctly, and concrete re-poured. The length of the delay in completion hasn’t yet been announced.

This effort, which constitutes a rollback and re-deployment, is a significant project in itself. It requires scheduling the work to be performed, but it also requires scheduling highway lane closures and lane shifts, working around high-volume traffic periods, and possibly pouring concrete in winter conditions. And after the piers are corrected, the bridge beam placement and bridge roadbed work must proceed on a new schedule.

Consequently, the construction error triggered a redesign of the flyover project’s DRP. But it probably did not trigger a significant re-design of the DDRP. The construction error is therefore unlikely to be an indicator of significant additional wickedness for the DDRP.

Last words

You can become better managers of the risk of unanticipated wickedness. If your organization is embarking upon a long-term program of technical debt retirement, you’ll be executing many DDRPs and DRPs. Gathering data about incidents of unanticipated wickedness in DDRPs can be a useful practice, if you use that data when you design new technical debt retirement projects.

]]>https://techdebtpolicy.com/nine-indicators-of-wickedness/feed/02274Degrees of wickednesshttps://techdebtpolicy.com/degrees-of-wickedness/
https://techdebtpolicy.com/degrees-of-wickedness/#respondSun, 28 Oct 2018 23:22:41 +0000https://techdebtpolicy.com/?p=2243Designing technical debt retirement projects can be a wicked problem. But there can be degrees of wickedness. And wickedness can have dimensions, too.

In a recent post I explored conditions that tend to make designing a project to retire technical debt a wicked problem. And in another post I noted some conditions that tend to make designing a project to retire technical debt a super wicked problem. But not all technical debt retirement project design efforts are wicked problems, and “wickedness” can occur in degrees. Designing these projects can be a tame problem, especially if we target the technical debt for retirement early in its existence. In this post I’ll explore degrees of wickedness in retiring technical debt, and propose a framework for dealing with technical debt retirement project design problems that are less-than-totally wicked.

The degree of wickedness of a problem

As a quick review, here are the attributes of a wicked problem as Rittel and Webber see them [Rittel 1973], rephrased for brevity:

There is no clear problem statement

There’s no way to tell when you’ve “solved” it

Solutions aren’t right/wrong, but good/bad

There’s no ultimate test of a solution

You can’t learn by trial-and-error

There’s no way to describe the set of possible solutions

Every problem is unique

Every problem can be seen as a symptom of another problem

How you explain the problem determines what solutions you investigate

The planner (or designer) is accountable for the consequences of trying a solution

Window blinds with some slats open and some closed. We can think of the slats as the wicked problem criteria of Rittel and Weber. A closed slat is a criterion satisfied by the problem; a partially open slat is a criterion that the problem satisfies to some extent. A wicked problem has all slats closed; a tame problem has at least one slat open or partially open. When many slats are mostly closed, the problem isn’t wicked, but it can be very difficult to resolve. That’s the way it is with most technical debt retirement project design problems. When even one slat is partially open, we can get a peek at the other side of the blinds, and use that information to pry open some of the other slats.

Rittel and Webber held that wicked problems possessed all of these characteristics, but Kreuter, et al., take a different view, which I find compelling [Kreuter 2004]. Their view is that wicked problems and tame problems lie at opposite ends of a spectrum. A problem that satisfies all ten of the criteria would lie at the wicked end of the spectrum; one that satisfies none would lie at the tame end.

A close examination of Rittel’s and Webber’s ten criteria reveals that they aren’t black-and-white; that we can regard each one as occurring in various degrees. For example, consider Criterion 1: “There is no clear problem statement,” which Rittel and Webber express as, “There is no definitive formulation of a wicked problem.” Burge and co-author McCall, who was a student of Rittel, offer this interpretation [Burge 2015]:

Here by the term formulation Rittel means the set of all the information need [sic] to understand and to solve the problem. By definitive he means exhaustive.

The original language of Rittel and Webber, with the interpretation of Burge and McCall, is indeed black-and-white. But one can easily imagine problems that satisfy this criterion to varying degrees. That is, one problem formulation might have almost everything one might need to understand and solve the problem, while another might have almost none of what one might need. In some of these cases, the problem solver might be able to make reasonable assumptions to fill in any gaps and then make some progress towards a solution. Or the formulation as given might be incomplete, but by working on a solution despite these lacunae, the missing information might reveal itself, or might arrive as a result of other research. For these reasons, I find it possible to regard the degree to which a problem satisfies Criterion 1 as residing on a continuum. And I expect one can make analogous arguments for all ten criteria.

This “continuum hypothesis” doesn’t conflict with the definition of a wicked problem. Wickedness still requires that all ten criteria be satisfied absolutely. But the position where a problem resides on the Tame/Wicked spectrum can be determined, conceptually, by the degree to which the problem fits the ten criteria of Rittel and Webber. In other words, as we address the problem of designing a technical debt retirement project, we can contemplate and consider the degree of wickedness of the problem; not merely that a problem is wicked or it isn’t.

The degree of a problem’s wickedness provides useful guidance. Specifically, if a problem clearly satisfies nine of the ten criteria, but not the tenth, according to Rittel and Webber, it would not be categorized as a wicked problem. Because it might be extraordinarily difficult to resolve, we would do well to treat it as wicked with respect to the nine criteria it satisfies. We would use that information to guide our decisions about where we apply resources, and what kind of resources we apply. The model of wicked problems provided by Rittel and Webber would be useful, even though the problem itself might not meet their definition in the strictest sense.

And so we’re led to the concept of the dimensionality of wickedness.

The dimensionality of wickedness

If we regard the ten criteria of Rittel and Webber as dimensions in a ten-dimensional space, then our “wickedness spectrum” becomes much richer. Maybe too rich, in the sense that its complexity presents difficulty when we try to think about it. But the concept of the dimensionality of wickedness can be useful, if we consider each dimension as having a degree of wickedness. This enables us to choose problem-solving techniques that work well for wicked problems that owe their wickedness to specific dimensions. That is the approach taken by Kreuter, et al. [Kreuter 2004]

Use that information to determine which of the ten criteria of Rittel and Webber are most relevant to this particular technical debt retirement project design problem

Apply established approaches that account for the relevant criteria to formulate a project design

This program is too much for a single post. But I can make a start in my next post with descriptions of the indicators of wickedness, including an examination of the implications of each of these indicators relative to the presence of each of the ten criteria of Rittel and Webber. The next step will be to suggest techniques for technical debt retirement project design problems that meet, to some degree, the criteria of Rittel and Webber.

]]>https://techdebtpolicy.com/degrees-of-wickedness/feed/02243Retiring technical debt can be a super wicked problemhttps://techdebtpolicy.com/retiring-technical-debt-can-be-a-super-wicked-problem/
https://techdebtpolicy.com/retiring-technical-debt-can-be-a-super-wicked-problem/#respondTue, 23 Oct 2018 14:28:06 +0000https://techdebtpolicy.com/?p=2213A subset of wicked problems can be viewed as super wicked, if time is running out, if the central authority fails to act, and if the people who cause them influence the solution. Some technical debt retirement situations qualify as super wicked problems.

In my last post I provided a list of attributes of wicked problems [Rittel 1973], and listed the reasons why I feel that technical debt retirement projects can qualify as wicked problems. As a quick review, here are the attributes of a wicked problem as Rittel and Webber see them, rephrased for brevity:

There is no clear problem statement

There’s no way to tell when you’ve “solved” it

Solutions aren’t right/wrong, but good/bad

There’s no ultimate test of a solution

You can’t learn by trial-and-error

There’s no way to describe the set of possible solutions

Every problem is unique

Every problem can be seen as a symptom of another problem

How you explain the problem determines what solutions you investigate

The planner (or designer) is accountable for the consequences of trying a solution

A subset of wicked problems can be viewed as super wicked [Levin 2012]. Levin, et al. list the following four properties of super wicked problems. With each one, I’ve added my thoughts about how retiring technical debt can qualify as a super wicked problem.

Time is running out

Two pilots line up their F/A-22 Raptor aircraft behind a KC-10 Extender to refuel while en route to Hill Air Force Base, Utah. When Hurricane Michael made landfall just after noon local time on October 10, 2018, it swept almost directly over Tyndall Air Force Base. Tyndall is the base for the U.S. Air Force 325th Fighter Wing, which has primary responsibility for air dominance training for pilots of F-22s, and maintenance personnel and air battle managers. 55 F-22s are based at Tyndall. When Hurricane Michael was approaching, 33 of the F-22s were repositioned to Wright-Patterson Air Force Base near Dayton, Ohio [Phillips 2018a]. According to U.S. Senator Bill Nelson (D-Florida), the aircraft that were left behind were undergoing maintenance and were not in condition to be flown [Gabriel 2018]. Some of these aircraft are known to have been damaged in the storm. At this writing, the extent of any damage hasn’t been disclosed, but it’s believed that the aircraft are repairable.Given the reality of anthropogenic climate change [Cook 2016], further increases in storm intensity and frequency are likely, and air bases located in vulnerable coastal regions are at risk [Phillips 2018b]. Because these bases were constructed during the period before the general recognition of anthropogenic climate change, they now constitute a technical debt. Relocating them would undoubtedly be a wicked problem, and possibly a super wicked problem. However, if federal policies continue to fail to account for anthropogenic climate change, those policies could prevent relocating these bases. If that happens, the policies themselves represent a technical debt. Changing federal policy to account for anthropogenic climate change is certainly a super wicked problem. U.S. Air Force photo by TSgt Ben Bloker, courtesy Wikipedia

The problem presents difficulty that has an inherent timescale. For example, many believe that climate change will become irreversible if it isn’t dealt with successfully within 30 years.

Technical debt retirement can have an inherent time scale, though it varies with the debt in question. For example, Microsoft ended mainstream support for Windows 7 in January 2015. At that point, by any definition, every computer running Windows 7 incurred a technical debt. Yet by September 2018, almost four years later, 46.7% of all computers running Windows were running Windows 7, and that number represented an increase from the previous month by 0.6% [Keizer 2018]. The safe-operability window for Windows 7 is closing fast. Well, to be honest, no faster than the calendar. But the end of standard support is January 2020, which represents the inherent timescale for this kind of technical debt. The consequences are much more serious than just converting to Windows 10. All applications running on those machines are also potentially reaching end of life, unless they operate correctly on Windows 10. And all users of converted machines will need to learn how to use the new Windows 10 OS and any applications that must be updated or replaced. So there’s also a training issue, a learning curve, and period of elevated user error rates to deal with, too.

Other technical debts might also have explicitly inherent timescales, of course. But for enterprises whose securities are publicly traded, all technical debt has a shared inherent timescale associated with the cost of retiring it. For any technical debt that must eventually be retired, if the cost of retirement is high enough to be noticed as a decline in net income in enterprise financial reports, the impact of debt retirement can be severe and negative. Letting debt remain in place, unaddressed and growing, can be a financially dangerous strategy. A safer strategy is to spread the cost of retirement across as many fiscal quarters as possible. For Windows 10 conversions, there are only about five fiscal quarters remaining for that purpose. For other technical debts, the number of fiscal quarters available for diluting the costs of retirement might be more—or less.

Those who cause the problem also seek to provide a solution

The phrase “seek to provide a solution” might be somewhat tactful. I expect that some super wicked problems have the property that those who cause the problem exert some degree of control over what kinds of solutions are acceptable, or even discussible. In many cases, this represents a conflict of interest that can prevent the organization from deploying the more effective options.

That conflict of interest is certainly present in the context of technical debt retirement projects. Technical debt formation and persistence are due, in part, to a failure to commit resources to retiring it, or, at least, to inhibiting its formation. That failure is the responsibility of those in leadership roles in the enterprise. Typically, these are the same people who must decide to commit resources to retire technical debt in the future.

The central authority needed to address it is weak or non-existent

Again, I find this description unnecessarily limiting. I would prefer a phrasing such as, “The central authority, for whatever reason, chooses to exert, or is unable to exert, its authority in furtherance of solution, or even investigation.” In other words, the central authority need not be weak for it to be a source of difficulty in addressing the super wicked problem. It need only choose not to act. This can happen when those who cause the problem are the people who constitute the central authority, or they capture the central authority, or they capture the function to which the central authority has delegated responsibility for solution.

With respect to technical debt retirement, consider this scenario. At AMUFC, A Made-Up Fictitious Corporation, the sales and marketing functions have repeatedly struggled with the engineering function for shares of budget resources. Engineering has argued repeatedly, and unsuccessfully, that it needs additional resources to address the technical debt that has accumulated in several products. But the CEO is a former VP Sales, and a close friend of the CFO. Together, they have always decided to defer technical debt retirement in favor of new products and enhancements favored by the VP Marketing, by customers, and by investors.

Scenarios like this are common. Enterprise leadership is strong, but not inclined to address the technical debt retirement issue.

Partly as a result, policy responses discount the future irrationally

Irrational discounting of future costs and benefits occurs when policies are deployed that give too much emphasis to producing short-term benefits and/or to avoiding short-term costs or inconveniences. Benefits are pulled in from the future towards the present; costs and inconveniences are pushed out toward the future and deferred. One form of this discounting scheme—one of many—is hyperbolic discounting.

This tendency is one way of distracting attention from the actual problem. It is the principal tactic that enables the persistence of technical debt, and the means by which enterprises repeatedly defer attention to the problem of retiring technical debt.

Both the problem of managing technical debt and the problem of designing technical debt retirement projects, exhibit all of these properties to some degree. It’s likely, in my view, that these problems are super wicked problems.

Intervention strategies for super wicked problems

Levin, et al., recommend four distinct strategies for resolving super wicked problems. They are all approaches to devising policies that are difficult to alter, thus committing the organization to a particular path forward.

Exploit lock-in

Lock-in is usually regarded as dysfunctional adherence to a strategy or course of action despite the existence of superior alternatives [Brenner 2011]. It occurs when a policy confers some kind of immediate benefit on a subset of the population. If that benefit is significant, if the population subset would be harmed by alterations of the policy that remove the benefit, and if the subset has enough political power to defend the benefit, the policy will be “locked in” and thus difficult to change. Levin, et al., suggest that this phenomenon can serve a beneficial purpose by protecting a constructive policy, thus preventing its abandonment.

Most technical debt retirement efforts are explicitly designed to focus solely on retiring the debt, with all (or most) of the benefit appearing in the form of increased engineering productivity, decreased sources of frustration for engineers, or increased engineering agility. Benefits for non-engineering stakeholders tend to be indirect. To establish policies that exploit lock-in we must craft them so that they provide ongoing, direct benefit to the most politically powerful stakeholders. For example, addressing first the forms of technical debt that are most likely to lead to product innovations that non-engineering stakeholders would value highly could cause those stakeholders to favor further technical debt retirement efforts.

Positive feedback

This strategy makes policies durable when people or organizations already supporting the policy derive some kind of increased benefit, leading to others not yet supporting or covered by the policy to decide to support it. This mechanism is sometimes known as a “network effect.” When network effects are present, the value of a product or service increases as the size of the population using it increases [Shapiro 1998].

To exploit network effects when devising technical debt retirement efforts, focus on retiring the kinds of technical debts that confer benefits on stakeholders of platform assets. A platform asset is an asset that supports multiple other assets. Examples: an application development tool suite, a product line architecture, or an enterprise data network. Platform assets that support collaboration communities are more likely to generate network effects.

Increasing Returns

Policies and interventions that enable increasing returns to the population are more likely to be durable than those that offer steady returns. Because people adapt to steady levels of stimuli, policies that produce a change in the context only during the period immediately following initial adoption of those policies are less likely to maintain popular support than are policies that continue to provide increasing returns as long as they’re in place.

But among policies that provide increasing returns, Levin, et al., identify two types. Type I policies, which are less durable, confer their benefits on an existing population of supporters. Type I policies don’t cause others to become supporters. Type II policies also confer benefits on their supporters, but they do cause others to become supporters. Type II policies are thus far more durable than are Type I, because they foster growth in the supporting population.

Framing technical debt retirement projects as individual projects with the objective of retiring a specified kind of technical debt is likely to lead to the enterprise population viewing the effort as a Type I policy at best. But framing each project as a phase of a longer-term effort could position the larger effort as a Type II policy, if the larger effort is designed to affect increasing portions of the enterprise population.

Self-reinforcing

A self-reinforcing policy is a policy that creates a dynamic that causes that policy to be maintained. Reinforcement can come about either as a result of increases in the benefits the policy generates, or by causing increases in the cost of rescinding the policy, or a combination of both effects. As with the strategy of Increasing Returns, there are two types of self-reinforcing policies. Type I self-reinforcing policies focus on maintaining support for the policy within the subset of the population consisting of its original supporters. In analogy with Type II Increasing Returns policies, Type II Self-reinforcing policies affect both the original supporting population and portions of the population not yet affected directly by the policy.

To exploit self-reinforcement, technical debt retirement programs must emphasize retiring debts that have curtailed organizational agility in recognizable ways, or which have prevented introduction of capabilities that the population values. Communicating these objectives is an important part of the program, because self-reinforcing popular support is possible only if the population understands the strategy and how it benefits the enterprise.

Because of the essential uniqueness of any wicked problem (Proposition 7 of Rittel and Webber), it is futile to attempt to apply as a template any retirement program that worked for some other organization, or for some other portion of a given organization at an earlier time with a different form of technical debt. But these four strategies, implemented carefully and communicated widely and effectively within the organization, can build organizational commitment to a long-term technical debt retirement program, even though retiring technical debt may be a super wicked problem.

]]>https://techdebtpolicy.com/retiring-technical-debt-can-be-a-super-wicked-problem/feed/02213Retiring technical debt can be a wicked problemhttps://techdebtpolicy.com/retiring-technical-debt-can-be-a-wicked-problem/
https://techdebtpolicy.com/retiring-technical-debt-can-be-a-wicked-problem/#respondSat, 20 Oct 2018 09:57:01 +0000https://techdebtpolicy.com/?p=2189Retiring technical debt isn’t just a problem, it’s a wicked problem. And that explains many of the difficulties we associate with retiring technical debts that involve many stakeholders, assets, revenue streams, infrastructure, architectures, policies, or strategies.

The theory of wicked problems originated with Horst Rittel in the mid-1960s. He was trying to address “that class of problems which are ill-formulated, where the information is confusing, where there are many decision makers and clients with conflicting values, and where the ramifications in the whole system are confusing.” [Churchman 1967] The term wicked isn’t intended as a moral judgment. It suggests the mischievous streak in these problems, many of which have the property that proposed solutions often lead to conditions even more problematic than the situation they were intended to resolve. Is it just me, or are you also thinking, “Ah, technical debt”? In this post, I suggest that retiring technical debt can be a wicked problem. I’ll show how wickedness explains many of the difficulties we associate with retiring forms of technical debt that involve many stakeholders, assets, revenue streams, infrastructure, architectures, policies, or strategies.

Introduction

Prototypes of President Trump’s “border wall.” Building such a wall is undoubtedly an example of a wicked problem. Building prototypes in short segments of the wall to see what they would look like is a tame problem. But these are just prototypes of short segments of the wall. They aren’t prototypes of the project. They don’t demonstrate the process by which private property will be taken for the wall, or how construction access roads will be built, or how wildlife will be affected (and respond), or how the government of the adjacent nation (in this case, Mexico) will respond, or how the wall will be repaired when drug gangs destroy sections of it in isolated regions, or even if the wall would actually work as promised. Prototyping works well for tame problems. It helps us project how the finished project will perform, and how difficult it will be to complete the project. But for wicked problems, prototyping is of limited value and as Rittel observes, prototyping can make the problem worse.Photo by Mani Albrecht, U.S. Customs and Border Protection Office of Public Affairs, Visual Communications Division.

Horst Rittel was a design theorist at the University of California at Berkeley. His interest in wicked problems came about because of the need for designers to deal with the interactions between architecture and politics. In today’s technology-dependent enterprises, analogous problems arise when we try to retire technical debt, because multiple sets of quasi-independent stakeholders are affected by major technical debt retirement projects.

In the years since he first originated the wicked problem concept, people have extended it in ways that have led some to regard the concept as inflated and less than useful. But since extension and imitation rarely occur unless the extended concept has value, I take this phenomenon to mean that Rittel’s version of the concept is worthwhile. The focus of this post, then, is Rittel’s version of wicked problems, as applied to the problem of designing a complex technical debt retirement project.

The wicked problem concept has propagated mostly in the realm of public policy and social planning. Certainly wicked problems abound there: poverty, crime control, and climate change are examples. I know of no attempt to explore the wickedness of retiring technical debt in large enterprises, but have a look below and see what you think.

Rittel defines a problem as the discrepancy between the current state of affairs and the “state as it ought to be.” For the purposes of technical debt retirement planning, the state as it ought to be might at times be a bit ambitious. So I take the objective of a technical debt retirement project to be an attempt to resolve the discrepancy between the current state of affairs and some other state that’s more desirable. For the present purpose, then, the problem is designing a technical debt retirement project that converts the current state of affairs to a more desirable state that might still contain technical debt in some form, but it leaves us in a better position.

For now, though, let’s examine the properties of wicked problems, and see how well they match up with the problem of designing technical debt retirement projects.

Attributes of wicked problems

Rittel’s summary of the attributes of wicked problems [Rittel 1973] convinced me that major technical debt retirement projects present wicked problems. Here are those attributes. In what follows, I use Rittel’s term tame problem to refer to a problem that isn’t wicked.

For any given tame problem, it’s possible to state it in such a way that it provides the problem-solver all information necessary to solve it. That’s what is meant by definitive formulation. For wicked problems, on the other hand, one’s understanding of the problem depends on the solution one is considering. Each candidate solution might—potentially—require its own understanding of the problem.

When designing a technical debt retirement project, we must fully grasp the impact of the effort on all activities in the enterprise. Each proposed project plan has its own schedule and risk profile, affecting enterprise activities in its own way. In principle, each candidate approach to the effort affects a different portfolio of enterprise assets in its own unique order. Because it isn’t practical to examine all possible candidate project plans, choosing a project plan by seeking an optimal set of effects isn’t practical. By the time you’re ready to execute the project plan you like best, the elapsed time required for such an analysis could render obsolete the data you collected first.

2. [No Stopping Rule] Wicked problems have no stopping rule

For any given tame problem, solutions have “stopping rules”—signatures that indicate clearly that they are indeed solutions. For example, in a chess problem to be solved in N moves, we know how to count to N and the position of checkmate is well defined.

Wicked problems have no stopping rule.

When planning a major technical debt retirement project, we must determine a task breakdown, a sequence for performing the tasks (which might be only partially ordered), a resource array including both human and non-human resources, a risk plan including risk mitigations and risk responses, a revenue stream interruption schedule, and so on. For each such plan, we can estimate the direct and indirect costs to the enterprise, and project the effects of the plan on market share for every affected product or service. Every plan has these attributes; when we compute them for a given candidate plan, the result doesn’t reveal that we’ve found “the solution.” We will have found only an estimate for that given solution. What we learn by doing this doesn’t reveal whether or not a “better” solution exists. There is no indicator contained in any given candidate solution that tells us we can “stop” solving the problem. Most often, we just stop when we run out of time for finding solutions. In some cases, we stop when we find just one solution.

3. [Solutions Are Good/Bad] Solutions to wicked problems are not true-or-false, but good-or-bad

The criteria for finding solutions to tame problems are unambiguous. For example, if a candidate function satisfies a differential equation, it’s a solution to the equation. The volume of concrete required to pave a section of roadway is a single number, determined by computing the area of roadway and multiplying by the thickness of the roadbed, and subtracting the volume of any reinforcing steel.

The solutions to wicked problems have no such clarity. When evaluating a candidate project plan for retiring a technical debt, we can estimate its cost, the time required, interruptions in revenue streams, and the timing of resource requirements. But determining how “good” that is might be difficult. Much depends on what other demands there might be for those resources or funds, and the political power of the people making those demands. No single number measures that.

4. [No Test of Solutions] There is no immediate and no ultimate test of a solution to a wicked problem

To test a candidate solution to a tame problem, the problem solving team determines whether the solution meets the requirements set in the tame problem statement. The consequences of implementing the solution are all evident to the problem solving team, and they’re well equipped to judge the success of the solution.

Not so with wicked problems. Any candidate solution to a wicked problem will generate waves of consequences. As these waves propagate, some of the problem’s stakeholders might find the solution unsatisfactory, and they’ll report this, possibly through politically powerful people or organizations, to the problem solving team. Because the consequences can be so diverse, the team can’t anticipate all of them, and in some cases, the team might have difficulty understanding how the troubles that plague some stakeholders were actually related to the implemented solution. Some undesirable consequences can be far more severe than any intended benefits. In other cases, the undesirable consequences might not be discovered until long after the solution has been deployed and accepted as operational.

When designing a technical debt retirement project, it’s necessary to determine everything that must be changed, what resources must be assembled to do the work, and what processes might be interrupted, when and for how long. Only rarely, if ever, can all of that be determined with certainty in advance. For that reason, determining that the design of the project is “correct” isn’t possible, except perhaps in the probabilistic sense. We never really know in advance that we’ve found a solution. Most of the time, after execution begins, we must make adjustments along the way, in real time.

5. [No Trial-and-error] Every solution to a wicked problem is a “one-shot operation”; because there is no opportunity to learn by trial-and-error, every attempt counts significantly

When solving tame problems, we can try candidate solutions without incurring significant penalties for the solution-finding effort. That is, trying a solution might require some effort (and therefore incur a cost), but it doesn’t otherwise affect the ability to find other solutions. Wicked problems are different. Every attempt to “try” a solution leaves traces that can potentially make further solution attempts more difficult, costly, or risky than they would have been if we hadn’t tried that solution. These traces of past solution attempts might also impose constraints on future solutions such that the wicked problem is effectively transformed into a new wicked problem. This property makes trial-and-error approaches undesirable and possibly infeasible. Indices of such undesirability are the half-lives of the traces of attempts to address the problem. A long half-life may mean that the problem solver has only one shot at addressing the problem.

When designing a technical debt retirement project, we sometimes try to “pilot” a potential approach to determine difficulty, costs, feasibility, political issues, or risk profiles. Even when we can revert the asset to its former state after a pilot is completed or suspended, the consequences for stakeholders and for stakeholder operations might not be reversible. When we next try another “pilot,” or perhaps a fully committed retirement project, these stakeholders might be significantly less willing to cooperate. Every attempted solution can leave political or financial traces like these, making future attempts riskier and more challenging.

6. [Solutions Are Not Describable] Wicked problems do not have an enumerable (or an exhaustively describable) set of potential solutions, nor is there a well-described set of permissible operations that may be incorporated into the plan

In devising solutions to tame problems, one common approach entails gathering the full set of possibilities, and screening them according to a set of favorability criteria. Reducing the field of possibilities is a useful strategy for finding optimal solutions—or even acceptable solutions—to tame problems.

Wicked problems defy such strategies. Gathering the full set of possible solutions to a given wicked problem can in itself be a wicked problem. The set of possible solutions cannot be parameterized; no finite set of attributes fully covers the solution space. We can never be certain that the set of candidates we have is a complete set.

Candidate designs for technical debt retirement projects present this same quality. We’re confronted with a dizzying array of choices, including the order in which we retire different kinds of technical debts, the order in which we address the debts borne by different assets, the possibilities of “refinancing” portions of the debt to intermediate forms of technical debt, what kinds of refactoring to perform and when, and much more. Because options like these are neither denumerable nor parameterizable, it’s difficult to know whether a given set of candidate project designs includes all possible project designs.

7. [Essential Uniqueness] Every wicked problem is essentially unique

Among tame problems, we can define classes or categories that have the property that all problems in a given class can be solved using the same method. For example, all the elements of the class of second order linear differential equations can all be solved using similar methods.

There are no classes of wicked problems that have such a property. There are no classes of wicked problems for which we can define a solution strategy that is effective for every member of the class. Even though we can define classes of wicked problems whose members are in some sense similar, that similarity doesn’t enable us to find a unified solution strategy that works for every member of the class.

So it is with designing technical debt retirement projects. Certainly, the collection of all technical debt retirement projects is a class; but the problem of designing a given retirement project is essentially unique. What “works” for one project in one enterprise in one fiscal year probably won’t work for another project in another enterprise in another fiscal year, or even another project in that same enterprise in that same fiscal year. Elements of the solution for one project might be useful for another project, but even then, we might need to adapt them to the conditions of that next project.

This essential uniqueness property of technical debt retirement projects collides with a common pattern decision-makers use when chartering major efforts. That pattern is reliance on consultants, employees, or contractors who “have demonstrated success and experience with this kind of work.” Because each technical debt retirement project is essentially unique, relying on a history of demonstrated success is a much less viable strategy than it would be with tame problems. Decision-makers would do well to keep this in mind when they seek approaches, leaders, and staff for major technical debt retirement efforts: no major technical debt retirement project is like any other.

8. [Problems as Symptoms] Every wicked problem can be considered to be a symptom of another problem

With tame problems or wicked, we typically begin the search for solutions by inquiring as to the cause of the current condition. When we find the cause or causes, and remove them, we usually find a new problem underlying them. Thus, for wicked problems, what we regarded initially as the problem is thereby converted into a symptom of newly recognized underlying problem. By repeating this process, we escalate the “level” of the problem we’re addressing. Higher-level problems do tend to be more difficult to resolve, but addressing symptoms, though easier, isn’t a path to ultimate resolution.

Rittel also observes that incremental approaches to resolving wicked problems can be self-defeating. The difficulty arises from the traces left behind by incrementalism, as described in the discussion of the unworkability of trial-and-error strategies. Rittel provides the example of the increase in difficulty of changing processes after we automate them.

To regard the wicked problem of designing a technical debt retirement project as a symptom of a higher-level wicked problem, we must be willing to regard as problems the very things that make the technical debt retirement project design effort a wicked problem. That is, the processes that lead to formation of technical debt, or that enhance its persistence, are themselves wicked problems. For example, one might inquire about how to change the enterprise culture so as to reduce the incidence of technical debt contagion. To undertake major technical debt retirement efforts without first determining what can be done to limit technical debt formation or persistence, due to contagion or due to other processes, might be unwise.

9. [No Controlled Experiments] The existence of a discrepancy representing a wicked problem can be explained in numerous ways. The choice of explanation determines the nature of the problem’s resolution

When addressing tame problems, problem-solving teams can often perform controlled experiments. The general framework of these experiments is as follows. They form a hypothesis H as to the cause of the problem, conjecturing a solution. Then assuming that H is correct, and given a set of conditions C, they deduce a set of consequences E that must follow if H is correct. If any elements of E don’t occur, then H is falsified. The process repeats until an H’ is found that cannot be falsified. H’ is then used to develop a solution. This is, essentially, the scientific method.

With wicked problems, the method fails in numerous ways. Foremost among these failure modes is the absence of the ability to control C. That is, interventions that might be required to set C to be a desired C0 tend to be impossible. Moreover, even if one can establish C0, the experiments one performs to determine whether E is observed tend to leave the kinds of traces discussed in Proposition 5 [No Trial-and-error]. Finally, the nature of the elements of E is usually such that determining their presence or absence is largely subjective.

When planning enterprise-scale technical debt retirement projects, as with many projects of similar scale, we believe that we can benefit from running a pilot of our proposed plan, to determine its fitness. However, because we cannot control the conditions in which we execute the pilot, we cannot be confident that our interpretation of the results of the pilot will apply to the actual project. Moreover, a small-scale pilot cannot generate some of the effects we most want to observe because they occur only at full scale. These effects include staff shortages, resource contention, and revenue interruption incidents.

10. [100% Accountability] The planner (or designer) has no right to be wrong

In solving tame problems, solvers are free to experiment with proposed solutions. They make conjectures about what might work, and gather the results of trials to determine how to improve their conjectured solutions. There is no social or legal penalty for failed conjectures.

In solving wicked problems, experiments don’t exist. Any trial solution is a real solution, with real effects on stakeholders and later, on the problem solvers themselves. Problem solvers are held accountable for the undesirable consequences of each solution, whether it’s a trial or not.

In planning a technical debt retirement project, any attempt to gather data about how the approach would affect the enterprise could potentially have real, lasting, deleterious effects. The costs associated with these consequences are charged to the project, if not officially and financially, then politically. The politics of failure can lead to serious consequences for the problem solvers. Any approach that the team deploys, on any scale no matter how small, can potentially create financial problems for the enterprise, and political problems for anyone associated with the technical debt retirement project.

Last words

The fit between wicked problems and technical debt retirement project design looks pretty good to me. But the research on a subset of wicked problems—super wicked problems—is also intriguing. I’ll look at that in my next post. After that, we’ll be ready to examine which approaches to retiring technical debt take these matters into account.

]]>https://techdebtpolicy.com/retiring-technical-debt-can-be-a-wicked-problem/feed/02189Synergy between the reification error and confirmation biashttps://techdebtpolicy.com/synergy-between-reification-error-and-confirmation-bias/
https://techdebtpolicy.com/synergy-between-reification-error-and-confirmation-bias/#respondTue, 09 Oct 2018 20:31:38 +0000https://techdebtpolicy.com/?p=2120The tendency to underestimate technical debt retirement costs is due, in part to the interaction between the consequences of the reification error and the consequences of confirmation bias. Here are some recommendations for mitigating these effects.

In deciding whether or not to undertake technical debt retirement projects, organizations are at risk of making inappropriate decisions because of a synergy between the reification error and confirmation bias. Together, these two errors of thought create conditions that make committing appropriate levels of resources difficult. And in those cases in which resources are committed, there is a tendency to underestimate costs, which can lead to an elevated incidence of failures of technical debt retirement projects.

How the reification error affects management

The reification error might be responsible, in part, for a widely used management practice that often appears in the exploratory stages of undertaking projects. Let’s start with an illustration from the physical world.

A feedback loop that now provides budgetary control in most organizations.

In the physical world, when we want cherries, we go to the produce section of the market and check the price per pound or kilo. Then we decide how many pounds or kilos we want. If the price is high, we might decide that fewer cherries will suffice. If the price is low, we might purchase more cherries. We have in mind a total cost target, and we adjust the weight of the cherries to meet the target, given the price. In the physical world, we can often adjust what we purchase to match our ability to pay for it.

Retiring technical debt doesn’t work like that, in part, because technical debt is an abstraction. But we try anyway; here’s how it goes. Management decides to retire a particular class of technical debt, and asks an engineer to work up an estimate of the cost. Sometimes Management reveals the target they have in mind if they have one; sometimes not. The estimate comes back as Total ± Uncertainty. Management decides that’s too high—or the Uncertainty is too great—and asks the engineer to find a way to do it for less, with less Uncertainty, maybe by being clever or doing less.

Management—the “customer” in this scenario—makes this request, in part, based on the belief that it’s possible to adjust the work to meet a (possibly unstated) target, in analogy to buying cherries in the produce department. That thinking is an example of the reification error. In this dynamic, we rarely take into account the fact that retiring technical debt isn’t exactly like buying cherries.

How confirmation bias affects engineering estimates

Now back to the interaction between Management and estimator. The engineer now suspects that Management does have a target in mind. Some engineers might ask what the target is. Some don’t. In any case, the engineer comes back with a lower estimate, which might still be too high. This process repeats until either Management decides against retiring the debt, or accepts the lowest Total ± Uncertainty, hoping for a final cost that isn’t too high.

In adjusting their estimates, engineers have a conflict of interest that can compromise their objectivity through the action of confirmation bias. In the case of technical debt retirement efforts, engineers are usually highly motivated to gain Management approval of the project, because the technical debt in question depresses engineering productivity and induces frustration. And since engineers typically sense that Management approval of the project is contingent on finding an estimate that’s low enough, the engineers have a preconception. That is, they have an incentive to convince themselves that the budget and schedule adjustments they make are reasonable. Because of the confirmation bias, they tend to seek justifications for the belief that lowering costs and tightening schedules is reasonable, while they tend to avoid seeking justifications for believing that their adjustments might not be feasible. That’s the confirmation bias in action.

How synergy between the reification error and confirmation bias comes about

Steve McConnell provides a clear framework for understanding the differences between estimates, targets, and commitments. An estimate “is a prediction of how long a project will take or how much it will cost.” It’s a tentative evaluation, rough calculation, or a prediction. As such it has a nominal value and an uncertainty band. A target is “a statement of a desirable business objective.” And a commitment is a promise. [McConnell 2006]So, because of the reification error, Management tends to believe that the work needed to retire technical debt of a particular kind is more adjustable than it actually is. And because of confirmation bias, engineers tend to believe that they can do the work for a cost and within a schedule that Management is willing to permit. Too often, the synergy between the two errors of thinking provides a foundation for disaster.

Management usually interprets estimates as commitments. Engineers do not equate estimates with commitments. Management usually forgets or ignores the upside Uncertainty. So typically, when Management finally accepts an estimate, the engineering team finds that it has made a commitment to deliver the work for the cost Total, with zero upside Uncertainty. Few engineering teams are actually asked to make an explicit commitment to perform the work for a cost Total with zero upside Uncertainty. An analogous problem occurs with schedule.

By ignoring the Uncertainty, Management (the buyer) transfers the uncertainty risk to the project team. That strategy might work to some extent with conventional development or maintenance projects, where we can adjust scope and risk before the work begins. But for technical debt retirement projects, this practice creates problems for two reasons.

Adjusting the scope of debt retirement projects is difficult

First, with technical debt retirement we’re less able to adjust scope. To retire a class of technical debt, we must retire it in toto. If we retire only some portion of a class of technical debt, we would leave the asset in a mixed state that can actually increase MICs. So it’s usually best to retire the entirety of any class of technical debt, so as to leave the asset in a uniform state.

Debt retirement efforts are notoriously unpredictable

Second, the work involved in retiring a particular class of technical debt is more difficult to predict than is the work involved in more conventional projects. (See “Useful projections of MPrin might not be attainable”) Often, we must work with older assets, or older portions of younger assets. The people who built them aren’t always available, and documentation can be sparse or unreliable. Moreover, it’s notoriously difficult to predict with accuracy when affected assets must be temporarily withdrawn from production—and for how long—to support the technical debt retirement effort. Revenue stream interruptions, which can be significant portions of total costs, can be difficult to schedule or predict. Thus, technical debt retirement projects tend to be riskier than other kinds of projects. They have wider uncertainty bands. Ignoring the Uncertainty, or trying to transfer responsibility for it to the project team, is foolhardy.

A strategy for reducing the effects of this synergy

To intervene in the dynamic between the consequences of the reification error and the consequences of confirmation bias, we must find a way to limit how their consequences can interact. That will curtail the ability of one phenomenon to reinforce the other. This task is well suited for application of Donella Meadows’ concept of leverage points [Meadows 1999], which I discussed in an earlier post, “Leverage points for technical debt management.”

In that post, I summarized Meadows’ idea that to alter the behavior of a complex system, one can intervene at one or more of 12 categories of leverage points. These are elements in the system that govern the behavior of the people and institutions that comprise the system. In that post, I sketched the use of Leverage Point #9, Delays, to alter the levels of technical debt in an enterprise.

In this post, I’ll sketch the use of interventions at Leverage Point #8, which Meadows calls, “The strength of negative feedback loops, relative to the impacts they are trying to correct against.”

A feedback loop that now provides budgetary control in most organizations

One feedback loop at issue in this case, illustrated above, provides budgetary control. It influences managers who might otherwise overrun their budgets by triggering some sort of organizational intervention when they do overrun their budgets. And it leads to increases in the portfolios of managers who handle their budgets responsibly. Presumably, that’s why managers compel estimators to find approaches that cost less. The feedback loop to which managers are exposed causes them to establish another feedback loop involving the engineer/estimator, and later the engineering team, to hold down their estimates, and later their actual expenditures.

We can use a diagram of effects [Weinberg 1992] to illustrate the feedback mechanism commonly used to control the performance of managers who are responsible for portfolios of project budgets. In the diagram, the oval blobs represent quantities indicated by their respective captions. Each of these quantities is assumed to be measurable, though their precise values and the way we measure them are unimportant for our rather qualitative argument.

Notice that arrows connect the blobs. The arrows represent the effect of changes in the value represented by one blob on the value represented by another. The blob at the base of the arrow is the effector quantity. The blob at the point of the arrow is the affected quantity. Thus, the arrow running from the blob labeled “Actual Spend” to the blob labeled “Overspend” expresses the idea that a positive (or negative) change in the amount of actual spending on projects causes a positive (or negative) change in Overspend. When a change in the effector quantity causes a like-signed change in the affected quantity, we say that their relationship is covariant.

Because increases in Budget Authority tend to decrease Overspend, all other things being equal, the relationship between Budget Authority and Overspend is contravariant. We represent a contravariant relationship between the effector quantity and the affected quantity as an arrow with a filled circle on it.

Finally, notice that the arrow from Overspend (effector) to Promotion Probability (affected) has a filled Delta on it. This represents the idea that as Overspend increases, it negatively affects the probability that the manager will be promoted at some point in the future. The Delta indicates a delayed effect; that the Delta is filled indicates a contravariant relationship. (An unfilled Delta would indicate a delayed covariant effect.)

This diagram, which contains a loop connecting Budget Authority, Overspend, and Promotion Probability, has the potential to “run away.” That is, as we go around the loop, we find self-re-enforcement, because the loop has an even number of contravariant relationships. It works as follows:

As Overspend increases, after a delay, the Probability of Promotion decreases. This causes reductions in Budget Authority because, presumably, the organization has reduced faith in the manager’s performance. Reductions in Budget Authority make Overspend more likely, and round and round we go.

Similarly:

As Overspend decreases, after a delay, the Probability of Promotion increases. This causes increases in Budget Authority because, presumably, the organization has increased faith in the manager’s performance. Increases in Budget Authority make Overspend less likely, and round and round we go.

Fortunately, other effects usually intervene when these self-re-enforcing phenomena get too large, but that’s beyond the scope of this argument. For now, all we need observe is that managers who manage their budgets effectively tend to rise in the organization; those who don’t, don’t.

The result is that managers seek to limit spending so as to avoid overspending their budget authority. And that’s one reason why they push engineers to produce lower estimates for technical debt retirement projects.

How this feedback loop overlooks important drivers of technical debt formation

To break the connection between the managers’ reification error and the engineers’ confirmation bias, our intervention must cause the managers and the engineers to make calculations differently. We can accomplish this by requiring that they consider more than the mere cost of retiring the class of technical debt under consideration. They must estimate the consequences of not retiring that technical debt, and they must also estimate costs beyond the cost of retiring the debt. In what follows, I’ll use the shorthand TDBCR to mean the class of Technical Debt Being Considered for Retirement.

Specifically, the estimates that are now typically generated for such projects cover only the cost of performing the work required to retire the TDBCR. It’s then left to Management to decide whether, when, and to what extent to commit resources to execute the project. The primary consideration is the effect on the decision-maker’s budget, and the consequences for achieving the goals for which the decision-maker is responsible.

Since the retirement project can potentially provide benefits beyond the manager’s own portfolio, failing to undertake the project can have negative consequences for which the manager ought to be held accountable. That’s the heart of the problem. So let’s look at some examples of considerations that must be taken into account.

Adjustments that would be needed in these feedback loops to gain control of technical debt

In making a resource allocation decision for a technical debt retirement project, there are considerations beyond the cost of retiring the debt. A responsible decision regarding undertaking technical debt retirement projects is possible only if other kinds of estimates are also generated and available. Here are some examples:

The effects of retiring TDBCR on the cost of executing any other development or maintenance efforts contemplated or already underway

The effects of retiring TDBCR on revenue and market share for all existing assets that directly produce revenue and which could be affected by retiring TDBCR

The revenue that would be generated (and timing thereof) by any new products or services that would be enabled by retiring TDBCR

The effects of retiring TDBCR on the cost of executing other technical debt retirement efforts

And these items might not be related to anything for which the decision-maker is responsible. That’s the core of the problem we now face: the feedback loop we now use to influence the decision-maker excludes considerations that are affected by the decision-maker’s decisions. Until we install feedback loops that cause the decision-maker to consider these consequences, or until we make decisions at levels that include these other consequences, the effects of the decision-maker’s decisions are uncontrolled, and might not lead to decisions optimal for the enterprise.

]]>https://techdebtpolicy.com/synergy-between-reification-error-and-confirmation-bias/feed/02120The resilience error and technical debthttps://techdebtpolicy.com/resilience-error-and-technical-debt/
https://techdebtpolicy.com/resilience-error-and-technical-debt/#respondSun, 23 Sep 2018 13:50:30 +0000https://techdebtpolicy.com/?p=2101The resilience error is an error of thinking in which we regard an abstraction as more resilient, pliable, adaptable, or extensible than it actually is.

The future USS Zumwalt (DDG 1000) is underway for the first time conducting at-sea tests and trials in the Atlantic Ocean Dec. 7, 2015. The first of the Zumwalt class of US Navy guided missile destroyers, it is designed to be stealthy, and to be supported by a minimal crew. After the program experienced explosive cost growth, the class has been downsized from 32 ships to three, and complement increased from 95 to over 140 to reduce capital costs. The three vessels on order now have significantly reduced missions. As one might expect, the causes of these troubles are much debated. But it’s possible that the resilience error plays a role. Before the first of a new class of ships goes to sea, it exists as an abstraction—a collection of concepts, plans, promises, and technologies, tried and untried. Many elements of this collection have never inter-operated with other elements. The first ship represents the first opportunity to see how all the elements work together. Although troubles often appear even before the ship is fully assembled, anticipating all troubles is extraordinarily difficult.

Reification risk is the risk that an error of reasoning known as the reification error might affect decisions—in this case, decisions regarding technical debt. The reification error [Levy 2009] [Gould 1996] (also called the reification fallacy, concretism, or the fallacy of misplaced concreteness [Whitehead 1948]) is an error of reasoning in which we treat an abstraction as if it were a real, concrete, physical thing. Reification is useful in some applications, such as object-oriented programming and design.

But when we reify in the domain of logical reasoning, troubles can arise. For example, we can encounter trouble when we think of “measuring” technical debt. Strictly speaking, we cannot measure technical debt. It isn’t a real, physical thing that can be measured. What we can do is estimate the cost of retiring technical debt, but estimates are only approximations. And in the case of technical debt, the approximations are usually fairly rough—they have wide uncertainty bands. That’s one way for trouble to enter the scene. When we regard the estimate as if it were a measurement, we tend to think of it as more certain than it actually is. Technical debt retirement projects then overrun their budgets and schedules, and chaos reigns.

For example, if we think we’ve measured the MPrin of a class of technical debt, rather than that we’ve estimated it, we’re more likely to believe that one measurement will suffice, and that it will be valid for a long time (or indefinitely). On the other hand, if we think we’ve estimated the MPrin of a class of technical debt, we’re more likely to believe that obtaining a second independent estimate would be wise, and that the estimate we do have might not be valid for long. These are just some of the consequences of the reification error.

The resilience error

If the reification error is risky because it entails regarding an abstraction as a real, physical thing, we might postulate the existence of a resilience error that’s risky because it entails regarding an abstraction as more resilient, pliable, adaptable, or extensible than it actually is.

When we commit the resilience error with respect to an abstraction, we adopt the belief—usually without justification, and possibly outside our awareness—that if we make changes in the abstraction without fully investigating the consequences of those changes, we can be certain that the familiar properties of the abstraction we modified will apply, suitably modified, to the new form of the abstraction. Or we assume incorrectly that the abstraction will accommodate any changes we make to its environment.

Sometimes we benefit when we modify abstractions; usually we encounter unintended and unpleasant consequences. For example, unless we examine our modifications carefully, it’s possible that the implications of a modification might conflict with one or more of the fundamental assumptions of the abstraction.

Examples of the resilience error

Perhaps a (ahem) concrete example will illustrate. Consider the steel hull of an ocean liner. We can manufacture it more cheaply if we can devise a way to use less steel. So one approach to that goal is to remove a small portion of the bottom of the hull, say, a circular hole one meter in diameter. We send some people into the ship to do the work, and they return with panicky reports of water coming in. But the ship seems fine, so we reject the reports. Even a day later, all seems well. But by the end of the second day, the trouble is obvious. The ship is sinking.

The problem in our example is that the circular hole in the hull violated a fundamental assumption about how ship hulls work: they work by keeping all water out of the ship. We had extended the idea of hull to make it lighter, but in doing so, we encountered some unintended consequences because our extension violated a fundamental property of hulls.

Now for a less fanciful example.

Consider the fictitious company Alpha Properties LLC, which manages small condominium associations (from 25 to 100 units). Things have been going swimmingly at Alpha Properties, and they’ve decided to expand to handle large condominium associations. Their financial accounting software has worked well, and their employees have become quite expert in its use. Alpha management has heard good reports from other management companies that deal with large client associations. So Alpha decides to use the same software for its larger accounts too. But things don’t work out so well.

The software is fine, but the processes used by the staff are cumbersome and slow. For example, setting up a new association requires too much manual data entry. For a 100-unit association, client setup wasn’t a burden, but for a 900-unit association the problem is just unmanageable.

This is a fine example of the resilience error. When we make this error, we fail to appreciate how an abstraction can encapsulate assumptions that make for difficulties when we try to extend it or apply it in a new or altered context. In this example, Alpha’s data flow processes are the abstraction. The context is signing up a new client association. When the context (signing up a large new client) is different, it violates an internal assumption of the abstraction (the data flow process for signing up a new client).

How the resilience error leads to technical debt

In many cases, the resilience error is at the heart of the causes of technical debt. It works like this. We have an asset that works perfectly well for one set of applications or in one set of contexts. We want to apply that asset in a new way, which might (or might not) require some minor extensions. When we try it, we find that the asset incorporates some assumptions about the application or the context, and one or more of those assumptions are violated by the new application or the new context. Scrambling, we find some quick fixes that can get things working again, but those fixes usually aren’t well designed or easily maintained. The result is a trail of technical debt.

Acquiring companies is like that. Before the acquisition, we think we’ll be able to merge the IT operations to save some expenses in operations. When we actually try it, though, merging them proves to be far more expensive than we imagined. Ah, the resilience error.

What makes this situation so difficult is that often we’re unable to anticipate what assumptions we might be about to violate. That’s why we make the resilience error.

Spotting difficulties with adapting to new applications and new contexts isn’t so difficult with physical entities. For example, we can see in advance that a square peg won’t fit into a round hole. But with abstractions, we can’t always see the problems in advance. Piloting, prototypes, games, and simulations can help us avoid some trouble, but not all.