Analytical risk based testing offers a number of benefits to test teams and
organizations that use this strategy.

One of those benefits is the opportunity to make risk-aware release decisions.
However, this benefit requires risk based test results reporting, which many
organizations have found particularly challenging. This article describes the
basics of risk based testing results reporting, then shows how Rex Black
(of RBCS ) and Nagata Atsushi (of Sony) developed and implemented new and
ground-breaking ways to report test results based on risk.

Analytical Risk Based Testing Testing can be thought of as (one) way
to reduce the risks to system quality prior to release. Quality risks typically
include possible situations like slow system response to use input, incorrect
calculations, corruption of customer data, and difficulty in understanding
system interfaces. All testing strategies,competently executed, will reduce
quality risks. However, analytical risk based testing, a strategy that allocates
testing effort and sequences test execution based on risk, minimizes
the level of residual quality risk for any given amount of testing effort.
There are various techniques for risk based testing, including highly
formal techniques like Failure Mode and Effect Analysis (FMEA). Most
organizations find this technique too difficult to implement, so RBCS
typically recommends—and helps clients to implement—a technique
called Pragmatic Risk Analysis and Management (PRAM).

In analytical risk based testing,stakeholder interviews, requirements
specifications, past defect history, and other sources of information are used
to develop a categorized list of quality risk items, which are then assessed
to determine, for each risk item, the likelihood of bugs related to the risk
item and the impact of such bugs should they exist in the system after release.
Impact is typically determined from a business perspective, while likelihood
involves consideration of technical issues. Likelihood and impact are then
combined to determine an overall rating of risk for each risk item (see Rex Black’s
book Managing the Testing Process, 3e, for a detailed description of the process
of quality risk analysis).

This catalog of prioritized risk items is then used to develop and execute
tests. Tests are developed for each risk item, with the precise number of tests
covering each risk item based on the level of risk. As tests are developed,
they are given a priority based on the priority of the risk item they cover.

Then, during test execution, tests are run in risk priority order. This provides
three immediate benefits:

n Tests effort is tightly calibrated to the level of risk reduction that
any given functional or non-functional attribute requires.
n Tests are run in an order that maximizes the chances of discovering
the most important bugs early in test execution.
n If necessary due to schedule pressures, less important tests (which
are sequenced towards the end of test execution) can be eliminated.
However, if, during test development and execution, the test team uses a
test management tool, a database, or even a simple spreadsheet to maintain
traceability between risk items, the tests that cover them, the results of those
tests, and bugs found during testing, the organization gains a fourth benefit:
the ability to make fully-informed, risk-aware decisions about whether to
release software based on the residual level of quality risk.
This benefit is of great importance, because a large number of project
teams make release decisions without a full understanding of the current
quality status of the system under test. This is because the standard test
management reporting dashboards, based on charts and tables showing bug
status and test pass/fail status, are at best indirect and imperfect measures
of quality and quality risk. Like a shadow thrown by a candle in a dark
room, the picture given by such charts and graphs is flickering, unclear in
the details, and distorted. Risk based results reporting, when done properly,
allows everyone, testers and non-testers alike, to see a clear, direct, steady
picture of quality risk.

Basic Risk Based Test Results Reporting
How does risk based test results reporting work? There are three
basic approaches.

The first approach, and simplest, approach relies on reorganizing
the traditional metrics of testing, bug status and test pass/fail status,
based on their relationship to risks. An example of this kind of reporting
is shown in Table 1, using a hypothetical e-commerce application.