Take any part of the development process – say testing, my bread and butter. Your team has it all figured out, with a magic automation tool. By magic, I mean exactly that. Push a button and you will get instant test results, clearly articulated.Not fast — instant. I’m talking about magic.

The automation tool never makes a mistake. It magically knows what a change will be and how it should impact the system, so it never reports a false error when the system behavior changes. Because of this, it needs no maintenance. Just change the code, click the button, see all the problems.

Now, will having that accelerate your software delivery at all?

In many cases, the answer is “no.” Even with the organizations that benefit, the benefit might only be a two to four percent increase in throughput.

Here’s why, and how to do better.

Test Execution != All Of Testing

That first thing to come back from that test automation tool is a set of results. Humans will need to examine those results, to understand what went wrong, and to explore to find if the problem is a core bug or the symptom of something larger. Then the bugs need to be defined in a way that can be fixed.

Next we argue about which of the bugs needs fixing.

Then we wait. And wait. And wait some more.

We wait for fixes to be made, unit tested, for a build, and a push to the test server.

Then we can press the button again.

When Tom Wissink was director of Test at Lockheed Martin, he used the phrase “Find->Fix->Retest loop” to the whole process. Fifteen years ago the famed software Michael Bolton, made a similar observation, pointing out that once the first blocking bug is found, it is no longer a “testing” phase, but instead a “fixing” phase.

Bolton’s observation was before Agile Development was popular, but it bears notice. If the primary delay in testing is not testing but instead “fixing, re-testing and re-fixing”, then automating test execution won’t save much time.

Oh, it might save a little. Assuming the tooling was good enough to find most of the important bugs enough of the time, test automation tools could enable release of features to production as soon as they are created, instead of “batching them up” to be deployed every two weeks, two months, or twice a year.

When automation tools won’t help

If the bottleneck on the system is not the task, then automating that task won’t help. The fix->build loop is too slow, and the number of defects is too large, meaning you’d have to press the button more times.

The solution to that often is not faster testing. Instead it is higher first-time quality (less test cycles), faster fixes, better builds, quicker test environments with better test data. Once that’s done, testing might be the bottleneck, and then there will be some work to do.

But there’s another problem.

What Happens When You Assume

Earlier I wrote “Assuming the tooling was good enough to find most of the important bugs enough of the time” and sidestepped if that were really possible. Tom, Michael and I have done plenty of writing on that; you can google our names to find out about test automation tools. Today’s post isn’t about that.

Today’s post is just about finding the bottleneck in your system. Find the thing that if you had a magic button, you could go faster. You don’t need to build a button for that — just find a way to help that activity happen faster. In programming, that could be more powerful programming languages, frameworks like rails, or just bringing in lunch and getting programmers out of useless meetings so they can focus on, well, programming.

Instead of a single silver bullet, look for what Fred Brooks called a collection of bronze bullets, centered around the bottleneck. Once that ceases to be the bottleneck – and yes, some of our customers have done that in testing – figure out where the new bottleneck is.

I agree to TechTarget's Terms of Use, Privacy Policy, and the transfer of my information to the United States for processing to provide me with relevant information as described in our Privacy Policy.

I agree to my information being processed by TechTarget and its Partners to contact me via phone, email, or other means regarding information relevant to my professional interests. I may unsubscribe at any time.

Processing your reply...

There was an error processing your information. Please try again later.

I agree to TechTarget's Terms of Use, Privacy Policy, and the transfer of my information to the United States for processing to provide me with relevant information as described in our Privacy Policy.

I agree to my information being processed by TechTarget and its Partners to contact me via phone, email, or other means regarding information relevant to my professional interests. I may unsubscribe at any time.

Processing your reply...

About This Blog

From the Cloud to Virtualization, Software As a Service to Web Services, ideas seem to be everywhere all the time. Somewhere, somehow, someone needs to separate the wheat from the chaff. We’ve asked Matt Heusser to provide his insight and commentary on the prevailing issues for IT staff today.