Continuous Testing and DevOps Transformation

Why Continuous Testing should be the focus of your DevOps transformation

Many organisations are currently undergoing DevOps transformations, but not many have testing, or continuous testing as the focus area for their transformation.

Should you focus on continuous testing as part of your DevOps transformation?

First of all, let’s start with a recap of what DevOps is.

DevOps is the combination of culture, tools, and practices that enables organisations to deliver software and services at increased speed and with reduced friction.

Organisations often, mistakenly, view that DevOps is just about bridging the silos between Dev and Ops. And while it is true that DevOps was initially a desire to close that gap, the movement is now far more than that. In order to underline that other groups have a key role, we are now seeing new terms such as DevTestOps or DevSecOps popping up to highlight the importance that other groups play in this transformation. My view is that “true” DevOps incorporates not just Dev and Ops, but also all other groups that are part of delivering business value via the software delivery pipeline.

To underscore this, let’s start with a key principle adopted by the DevOps and Lean movements called “The Theory of Constraints” from Elijah Goldratts novel “The Goal”.

e.g. “every process has a single constraint and that total process throughout can only be improved when the constraint is improved”

This theory tells us that we should look at the software delivery pipeline as a process and that at any time there is only one key constraint in the process. We should focus on improving that one area that is the key constraint or bottleneck rather than trying to do everything, and therefore doing nothing well.

Think of a factory line A > B > C

A and C can both process 100 units per hour, but B can only process 50 units per hour.

A very clever engineer has come up with a way to improve the throughput of A by 50%. We now can get 150 units per hour through A. Should we do this?

No! Of course not. We should focus on what we can do with B to raise the throughput until it reaches, and exceeds, the throughput of A and C. Then we identify the next constraint and improve that.

So often with software development we forget to apply these simple concepts because the systems we are dealing with are more complex and they involve people. People who are vying for our time and budget as managers. People who have their own area of expertise that they want to showcase. People who don’t always have a holistic view of the overall system we are trying to optimize.

To counter this, we need to use techniques like value stream mapping to map out the entire software delivery pipeline and find where the bottlenecks are. Each organisation's pipeline will be different and one size does not fit all, so it's important to go through this process to identify the constraints in your organisation.

In my view, this is one of the most difficult changes that organisations need to make. Most organisations tend to try to fix everything at once and usually end up doing nothing well. This needs to shift to focusing on improving one key constraint so that it is no longer a bottleneck.

Now, while recognising that all organisations are different, there are some common challenges that organisations face.

Computing Research conducted a survey as part of the DevOps Review 2017 where they asked “What are the main hold ups in the software production process?” In other words; In your software delivery pipeline what is the key constraint?

The majority of respondents (63%) indicated that Testing/QA was the bottleneck.

Anecdotally, we also see this to be true. For many organisations, it is relatively easy to deliver a new build in days, hours or minutes, yet testing still takes weeks or months to be completed. As the overall cycle time can only be as fast as the slowest component of the process we are limiting our ability to deliver truly complete software (software that has been developed, tested and deployed) to the speed of testing at best.

So, let’s return to the original question; Should you focus on continuous testing as part of your DevOps transformation?. A key principle to understand when shifting to DevOps is “The Theory of Constraints” and from this theory we know that we should focus on optimizing the key constraint. If you are part of the majority of companies where testing is the bottleneck, you must focus on continuous testing to truly gain value from your shift to DevOps.