Demodularization can help control technical debt

Last updated on July 24th, 2018 at 07:26 pm

Modularity is a widely accepted design approach for complex systems. But because modularity can be implicated in the accumulation and persistence of technical debt, temporary demodularization can help control technical debt, and longer-term demodularization can reduce the rate of accumulation of technical debt.

Modularization is a widely used approach to complex system design

Two shipping containers resting on a “spine car,” a kind of rail car used for shipping containers. The container on the left is a so-called tank container, used for bulk cargo. Various types of tank containers are available for transporting different types of cargo, such as wine, oils, ammonia, and even cryogenic liquids. The steel frame around the tank provides compatibility with the standard container profile, which makes the tank compatible with equipment built to handle standard shipping containers. This functions as an interface between the tank and the container-handling equipment. Photo (cc) Mr Snrub at the English language Wikipedia

Since the 1970s, modular design of systems has been de rigueur in both software and hardware. More than that, modularization has been demonstrated to be an essential feature of maintainable, adaptable, and extensible systems [Parnas 1979] [Sullivan 2001]. And we now understand that modularization is a foundational attribute of loose coupling in systems, which enables system designers and maintainers to work in parallel, with independence, on system elements, rendering systems economical at levels of complexity beyond what is achievable with tighter coupling [Orton 1990].

Eliminating duplication is one reason why modularization reduces maintenance and enhancement costs. Modularization enables system designers to create a single system element that provides needed functionality to other parts of the system. Because there is then only one system element that provides that capability, adapting it in response to a new need, or to correct a defect, need be done only once.

That’s a big deal. If that capability were provided in multiple system elements, adaptation would be necessary for each of those elements. Moreover, the multiplicity of elements opens the possibility that adaptation might not be performed consistently, which could create further problems. Eliminating duplication is a most useful property of modularization.

Modularization provides many other advantages. For example, modularization shortens time-to-market for new capabilities. When extending the system by adding new capability, we sometimes need access to capabilities present in existing modules. In modularized systems, since those modules are already in a form that permits invocation by system components, we can access them easily. We have no need to recreate them for the new capability we’re implementing. They exist, and they’re already tested and ready to go. In this way, modularization shortens time-to-market for new capability.

Modularization has a dark side

It’s a little bit of a story to show how the dark side of modularization works, so let’s give a name to the modular system element whose duplicates have been excised. Since it’s now unique, I’ll call that modular system element “U.” Any system element that interacts with U is now indirectly coupled to every other system element that interacts with U. And that’s where the trouble comes in.

When we’re implementing a new capability N, and N needs access to U, we gain the advantages described above. But suppose N needs U to do something just a little bit unusual — a little bit differently from what U now does. Sometimes we can extend U in ways that accommodate N without disturbing U’s existing “client base” — the system elements that are already interacting with U. There’s no problem then. But let’s suppose that what N now needs U to do would disturb U’s client base if we implement the changes in the “correct,” most elegant way — the way we would do it if we were starting fresh. Sadly, in that case, all of U’s existing clients would have to be modified, and then re-tested. So let’s suppose that we don’t have time or resources to do all that work. We requested them, but we were denied.

So instead, we found a way to extend U in a less elegant, less maintainable, but still reliable way that doesn’t disturb U’s existing clients, and does meet N’s needs. We do that instead, promising ourselves that we’ll go back someday, when we’re granted the time and resources, and “fix” U so that it serves both its existing clients and N in the “correct” way.

That’s one form — exactly — of what we call technical debt. In this scenario we’ve illustrated one way in which modularization leads to technical debt formation.

So a natural question arises: would it make sense instead to create a new system element — call it U2 — that meets N’s needs, and also meets the needs of U’s existing client base, if only they knew about U2 and could be altered to use U2? My proposed answer to that question is: “Yes it would, in many cases.” To create such a U2 would be demodularization — that is, a violation of modularity — and that is indeed heresy. It also creates a different technical debt: the obligation to convert U’s clients to become U2 clients someday, and then to delete U. But it might be the right approach.

When would demodularization help?

Under what conditions would demodularization be sensible? Here are three possibilities.

When a new and necessary adaptation is incompatible with existing forms

The scenario above is one situation in which demodularization can help. Demodularization helps when adding new capability, or adapting to a new need, requires a change to a shared module, and that change is incompatible with the existing uses of that module. Demodularization is than a useful technique, provided that the technical debt that results is retired with due dispatch.

When retiring technical debt requires an incompatible adaptation

A second situation arises during technical debt retirement operations. During technical debt retirement, it might be necessary to alter a shared module in a way that would be incompatible with the needs of its existing client base. In that case, the approach used above can be useful. First, create a successor (“U2”) to the original shared module (“U”) in a form that isn’t burdened with the technical debt that’s being retired. Then, at the same time, or over an extended period, convert all the clients of U to use the successor U2. In the meantime, the demodularization comprises a technical debt. When the conversion is complete, the original technical debt will have been retired. Finally delete the original shared module U, thereby retiring the technical debt that consisted of the demodularization.

This approach entails some risk. In the interim period before U is retired, when demodularization is still in place, changes to both U and U2 might be required. When that happens, duplication of effort can occur. This approach is useful, though, provided the interim period of demodularization is short compared to the anticipated intervals between incidents that require alterations to U and U2. There is risk, of course, that the resources committed to finally retiring U might become unavailable after U2 is in place. In that case, the technical debt portfolio will have been expanded to no good end. To manage this risk, the artifice of secured technical debt can prove useful.

Partial demodularization helps when adaptations are focused

In some instances, portions of a shared system element — call it “U” — evolve very rapidly, while most of the rest of U remains stable. Technical debt can accumulate rapidly if the element remains unitary — that is, in one piece. However, in some cases we can segregate the rapidly evolving portion of U into a smaller unit — call it “S.” If we provide S as a separate shared system element, those portions of the system that are experiencing rapid evolution can access S separately, without disturbing the system elements that require access only to the stable portions of U.

Such segregation might require a bit of duplication, because there might be pieces of S that are needed by U, and which must therefore be duplicated in U. Likewise, there might be pieces of U that are needed in S, and which must therefore be duplicated in S.

But the segregation might be worthwhile, because changes in S usually require testing S and S’s clients. Testing can be expensive in time and resources, and because test coverage isn’t always 100% (read: test coverage is rarely 100%), changes in S entail some operational risk. Segregating S reduces that risk by protecting U’s clients from changes in S.

Later, when the rapidly evolving S stabilizes, it can be re-integrated into its former residence in U. Until that point, its segregation — and the attendant duplications — might constitute a technical debt.

Conclusion

Accepting modularization as an inviolable design principle is one cause of unnecessary accumulation of technical debt. It makes retiring legacy technical debt more difficult. Be prepared to violate modularity, but do so judiciously.

A subset of wicked problems can be viewed as super wicked, if time is running out, if the central authority fails to act, and if the people who cause them influence the solution. Some technical debt retirement situations qualify as super wicked problems.

Retiring technical debt isn’t just a problem, it’s a wicked problem. And that explains many of the difficulties we associate with retiring technical debts that involve many stakeholders, assets, revenue streams, infrastructure, architectures, policies, or strategies.

The tendency to underestimate technical debt retirement costs is due, in part to the interaction between the consequences of the reification error and the consequences of confirmation bias. Here are some recommendations for mitigating these effects.

Contact me on social media:

Reference posts

A collection of definitions of terms as we use them in this blog, with links to longer discussions of each term. Along with each definition is a link to a post that discusses that term in more detail. Read more…

Welcome to Technical Debt for Policymakers. What you’ll find here are resources, insights, and conversations of interest to policymakers who are concerned with managing technical debt within their organizations. Read more…

When retiring one kind of technical debt from an asset, auxiliary technical debt is any other kind of technical debt in the asset. It can be tempting to try to retire auxiliary technical debt too. Sometimes that’s wise, but it can lead to scope creep. Rules of engagement can control this temptation. [More]

To retire technical debt, we need to know where it is. And if service disruptions are necessary, we need to know who will be affected. Here’s a survey of some of the issues, and suggestions for resolving them. [More]

For some assets, we can’t allow debt to persist, and we can’t afford replacements. We must retire the debt. This post begins exploring what it takes to design projects to retire technical debt in irreplaceable assets. [More]

By carefully observing what happens when we actually try to retire some kinds of technical debt, we can better understand the degree of the degree of wickedness of the effort. That understanding helps manage risk in technical debt retirement projects, reducing costs and speeding execution. [More]