1 May 2017

Powerful automated software testing metrics & why you need them

Automated software testing metrics by themselves are not powerful. It is the smart use of these metrics that makes them powerful and gives you the ability to transform your software quality program. Unlike Qsome's test automation tools, most testing tools do not offer test-related analytics. The analytics that are built into our automated testing tools are one of the differentiators that sets us apart from our competitors.

Elfriede Dustin's book, Implementing Automated Software Testing, is considered gospel for those wishing to put in place a set of metrics to assess the effectiveness of their automated testing program. However, the pace of change in software quality requirements and new development methodologies have necessitated a need to prioritise some automated software testing metrics over others.

At Qsome, there are two types of metrics that help convey very different messages:

Metrics to use to assess automated test preparation; and

Metrics to use during test execution to help with delivery-related decision making.

It's important to be SMART

Whatever metrics you choose to follow as part of your automated testing program, apply the SMART principle to make sure that you don't waste your time. Applying this elegant principle helps to reduce errors and forces your team to focus on only those metrics that add real value to the analysis of your testing program.

I find that the part of this acronym that people most struggle with is "time-bound". In the metrics that we track with our testing tools, time-bound takes two forms:

The time period over which that activity is being tracked; and (more importantly)

A comparison of the result over different time periods.

Applying point 2 to all analyses of your metrics will help you quickly understand where your automated testing program is ripe for optimisation. This is where automated software testing statistics go from mere metrics to those of the powerful variety.

Metrics to assess test preparation & test worthiness

As the guide to 10x testing effectiveness emphasises, it is very ineffective if manual test cases are simply converted to automated test cases. To work out Percent Automatable make sure you convert your test cases into user journeys before calculating how many can be automated.

While it may be possible to automate every journey, it may not be the most effective or efficient use of resources to do this. My teams aim for a percent automatable metric of over 90%, especially for web-based and mobile app testing. Depending on your application architecture, it may be excessively cumbersome and unnecessary to automate the testing of setup or initialisation journeys.

Automation Progress refers to the number of user journeys that have been automated, from those that were deemed automatable at the outset. This is an important number for customers who want to understand how their automated testing service provider is progressing with building the automated regression testing suite.

Great decisions are made with good data. Track your own software testing metrics with our dashboard template.

Metrics to assess test progress and success

The 10x testing effectiveness guide focuses a lot on how to make your software testing program faster and more effective at catching bugs before they get to production. Increasing the effectiveness of anything, particularly software testing programs, requires participants to conduct a series of small experiments, track key metrics and iterate after changing one or two elements to assess the impact of the said changes.

Now that you know what to do with these test automation metrics, the first one to track is very simple, but very few teams actually track it: number of defects in production. This overarching statistic should be tracked for every version that is released to production. Plot it on a graph and if you see a downward trend, then you know that your software testing program is actually working. If your graph resembles more of the latter parts of a J-curve or has flatlined, then you need help.

The percentage of defects resolved is a very simple statistic that tells you the workload remaining in a sprint before the application needs to be tested again prior to release. Ideally, that number should be zero before a version is released to production. It's not rocket science!

An advanced version of these two metrics is defect density. This is the percentage of all defects contributed by each module that is being tested. Because we create automated tests based on user stories, this can be a little more difficult to measure, but it will provide great insights into which parts of your development process may require greater emphasis on quality.

Two metrics that we ask our customers to focus on prior to any release are percentage of tests passed and test coverage. These two are quite self-explanatory but must form the basis on which all application delivery decisions are made. If both are not above a pre-agreed threshold, that release does not make it into production. Quite simple really!

For large applications like ERP and CRM systems, test coverage can be split into the coverage of high and medium-risk user journeys or coverage of only those user-journeys that that are affected by any code changes made in the current sprint. To accomplish the latter you require your automated software testing tool to be able to automate test scope selection.

The list of metrics that can be tracked is literally endless. The question you have to ask yourself is whether it is necessary for other metrics to be tracked within your software testing program.

How will you track these automated software testing metrics?

Knowing what to track is one thing, but executing the measurement process without burdening your software testing program with loads of extra resources, inefficiencies and costs are also important. Ask your testing tool vendor what metric tracking capability is already built into the tool.

You will likely find that very few automated software testing tools have metric tracking included out-of-the-box. Qsome's testing tools do include critical test analytics to drive the software delivery process. This is one of the reasons that our customers have switched to us.