High Complexity Architecture

High complexity architecture involves applying architecture practices and principles to business and technical problems involving many interdependent integrating parts that interact in a number of ways resulting in unpredictable, emergent outcomes.

We’ve been lucky enough to work on some of the most complex architectures in the world. Most organizations work with complicated problems, not high complexity issues. However, to innovate or invent ahead of competitors or adversaries, organizations often have to work in higher complexity areas. By working on very new ideas, there is a lack of knowledge and tried and tested techniques; this widens the cone of uncertainty significantly. Where there are a large number of interacting parts and a large number of interactions complexity can be very high. For more on complexity in general, see Simple, Complicated and Complex.

Part of the job of architecture is to reduce complexity. That’s ok for simple and complicated systems using the techniques mentioned above but it’s a little trickier for very high complexity systems.

Very high complexity architectures are those in which a number, or all, of the following conditions are true:

There is extensive integration between independent components, technologies and/or products

The work is highly speculative and unpredictable, we expect to fail fast and often

There isn’t a large amount of domain knowledge or experience in the field

One or more of the dimensions of an architectural profile are “Very high”

Estimates are extremely uncertain

Risks are very high

Extremely large scale (of data, change and distribution)

There is a complex logical relationship between inputs and emergent properties or behaviors

When working in high complexity systems, the intent is often more important than the detail. Work is often highly speculative as people are inventing new techniques that may or may not work. Empirical feedback becomes more useful than specification, and due to the experimental “sensing” nature of high complexity work details will change significantly.

We have seen many pieces of work attacking high complexity problems that have failed, not due to the inherent complexity and risk, but because they used practices and ways of working suited to complicated problems to try and attack them.

Failure is always likely in high complexity work but it’s made even more likely by trying to “control” complexity. Practices that have worked for previous work or seem sensible become highly dysfunctional. Examples include:

forward planning - We don’t yet know how we’re going to solve the problem if we even can. Plans will change quicker than they can be written down. “Probing and sensing” is more appropriate.

detailing requirements - The intent is important, not the detail of how. Our technical solutions are likely to change significantly meaning that details will change. A sub-optimal solution might be the only cost-effective option significantly affecting scope.

analysis and design - Decomposing problems into manageable chunks isn’t the right answer when the complexity is in the number of “chunks” and their interactions. Instead we need to manage emergent properties and create architectural experiments (spikes) to prove or disprove our ideas.

user-centric design - Users are unlikely to have resolved the high complexity issues and may not even understand the problem fully. Giving the users what they want, often a good idea, is often the opposite of a high complexity solution. Users seek simplicity, and although this is a good idea in terms of interaction with high complexity architecture, designing interaction doesn’t help solve the problem. Since scope and technology are likely to change, UX detail is best left until the solution is more stable.

estimation - By the nature of the complexity estimations will all be extremely uncertain. Numbers may be significant orders of magnitude out. Instead organizations are better off funding experimentation in a number of timeboxes to see how much complexity can be produced in those timeboxes through spiking and experiments.

When working with high complexity architecture, standard Enterprise Architecture may need to be abandoned to deal with emergent behavior. However, the cost of not using Enterprise Architecture should not be considered as an inhibitor to high complexity work if the Business Value is important enough or if the work is related to a strategic direction to beat competitors.

In Solution Architecture for high complexity architectures, a common dysfunction is to try and connect all of the architectural information and system architectures. We’ve seen organizations create massive over-complicated models that try and resolve integration complexity by documenting it all. Well-meaning, but ineffective and wasteful. This approach is counter-productive because the complexity comes not from the number of items, but the dynamism of their interactions and unpredictable aggregate behavior. These large models are, at best, a snapshot view of complexity at a single point in time but they don’t help anyone solve the problems.

Instead, when working with high complexity Solution and System Architectures we recommend architecting for change, not solution requirements. By that we mean that since requirements, scope and emergent behavior are so likely to change the only thing we really know the architecture needs to support is change. Therefore, focus on the principle, technology and mechanisms that enable change such as:

Since the behavior and properties of high complexity systems are emergent we recommend creating a “pull” towards Business Value through rapid, preferably continuous, integration and deployment. Creating measures that reflect desired outcomes (not intermediary stages or logical decomposition) and test every change in terms of moving towards or away from “good” emergent behavior.

At the extreme end of this high complexity scale, using evolutionary techniques to generate possible solutions and test for emergent properties can be useful to speed up iteration cycles or artificial intelligence networks can be used evolve network weighting to recognize successful progress.