Practical Test Reporting

Developing a usable and consumable test-metric-reporting system is a challenge for all testing organizations. This article describes a system employable by small and large organizations and all test efforts. By using existing tools, test teams can show current progress and predict future test efforts.

Getting Started Test professionals know that metrics are an important project deliverable. Developing a system to create, track, and report test results is a different story. The "whys" of test metrics have been covered before, so I'll instead show you how to produce the data to address those "whys." The data includes:

Quantifying the test team's contribution to the project

Showing a project's testing progress (Pass %)

Developing product quality measures (Executed/Pass or Fail)

Gathering historical data to predict future progress

There are, of course, limitations to any measurement system:

For large projects, it is difficult to execute every test for every new iteration.

Test quality will affect measurement quality.

Varying levels of specificity may create false positives or negatives.

Limitations of the test tools (measurement devices) can affect the accuracy of metrics.

You don't have to spend a lot of money to start a test metrics program; a spreadsheet-based system will do. There are benefits to commercial test management tools that should drive your decision. Even so, many of these tools use industry-standard databases that you can read from directly to create reports that meet your organization's needs.

What Data Do I Start With? It's easy to get carried away with tracking, but there are only a few measurements needed to start a rich test metrics program. Start with the basics and determine which are most valuable and can be used to derive other metrics.

Make sure everyone in your organization uses the same terminology. Terms such as "% Complete" to describe the "Pass %" are inaccurate unless the definitions are clear. For this article, let's use these definitions :

Term

Definition

Test

An instruction to perform a particular action with a clear purpose, assumed setup parameters, and expected results

Test Execution

Performing a test or test group

Test Status

The results of a test when executed

Let's keep to a few simple and well-understood measurements:

Term

Definition

Pass

This test passed during this test execution.

Fail

This test failed during this test execution.

Not Executed

This test was not executed during this test execution.

These measurements are the most basic we can track when executing a test. They can be tallied along with our total number of tests to form percentages.

Tracking Progress

Figure 1: Set of tests and its execution status

We have determined the total number of tests, their results, and the Pass % , Fail % , and Not Executed % using these simple formulas:

Label

Calculation

Total Tests

A count of all tests to be measured

Total by Test Status

The count of tests by their status

Pass %

Percentage of tests in Pass status

Fail %

Percentage of tests in Fail status

Not Executed %

Percentage of tests not executed

The Daily News--Summarizing What Testing Accomplished Today We've uncovered some useful information. At the time that this report was generated, the Pass % was 55% and 27% of the tests hadn't been executed. There should be several issues in the defect management system to record the failures.

Anyone involved can get a quick idea of the current testing status. Combining this with previous project reports, the project team can see what happened yesterday and forecast what may happen tomorrow.

By managing the reporting and raw data separately, we can add more tests and still get the data we need. Let's use an example where "Setup" is the functional area being tested (see Figure 2).

Figure 2

This allows the report to show information for specific areas of the project and provide more granular information. Now let's manipulate the data a bit to glean