9 Code Coverage Analysis

Although Common Test was created primarly for the purpose of
black box testing, nothing prevents it from working perfectly as
a white box testing tool as well. This is especially true when
the application to test is written in Erlang. Then the test
ports are easily realized by means of Erlang function calls.

When white box testing an Erlang application, it is useful to
be able to measure the code coverage of the test. Common Test
provides simple access to the OTP Cover tool for this
purpose. Common Test handles all necessary communication with
the Cover tool (starting, compiling, analysing, etc). All the
Common Test user needs to do is to specify the extent of the
code coverage analysis.

To specify what modules should be included
in the code coverage test, you provide a cover specification
file. Using this file you can point out specific modules or
specify directories that contain modules which should all be
included in the analysis. You can also, in the same fashion,
specify modules that should be excluded from the analysis.

If you are testing a distributed Erlang application, it is
likely that code you want included in the code coverage analysis
gets executed on an Erlang node other than the one Common Test
is running on. If this is the case you need to specify these
other nodes in the cover specification file or add them
dynamically to the code coverage set of nodes. See the
ct_cover page in the reference manual for details on the
latter.

In the cover specification file you can also specify your
required level of the code coverage analysis; details or
overview. In detailed mode, you get a coverage overview
page, showing you per module and total coverage percentages, as
well as one HTML file printed for each module included in the
analysis that shows exactly what parts of the code have been
executed during the test. In overview mode, only the code
coverage overview page gets printed.

You can choose to export and import code coverage data between
tests. If you specify the name of an export file in the cover
specification file, Common Test will export collected coverage
data to this file at the end of the test. You may similarly
specify that previously exported data should be imported and
included in the analysis for a test (you can specify multiple
import files). This way it is possible to analyse total code coverage
without necessarily running all tests at once.

To activate the code coverage support, you simply specify the
name of the cover specification file as you start Common Test.
This you do either by using the -cover flag with ct_run.
Example:

$ ct_run -dir $TESTOBJS/db -cover $TESTOBJS/db/config/db.coverspec

You may also pass the cover specification file name in a
call to ct:run_test/1, by adding a {cover,CoverSpec}
tuple to the Opts argument. Also, you can of course
enable code coverage in your test specifications (read
more in the chapter about
using test
specifications).

By default the Cover tool is automatically stopped when the
tests are completed. This causes the original (non cover
compiled) modules to be loaded back in to the test node. If a
process at this point is still running old code of any of the
modules that are cover compiled, meaning that it has not done
any fully qualified function call after the cover compilation,
the process will now be killed. To avoid this it is possible to
set the value of the cover_stop option to
false. This means that the modules will stay cover
compiled, and it is therefore only recommended if the erlang
node(s) under test is terminated after the test is completed
or if cover can be manually stopped.

The option can be set by using the -cover_stop flag with
ct_run, by adding {cover_stop,true|false} to the
Opts argument to ct:run_test/1, or by adding
a cover_stop term in your test specification (see chapter
about test
specifications).

The incl_dirs_r and excl_dirs_r terms tell Common
Test to search the given directories recursively and include
or exclude any module found during the search. The
incl_dirs and excl_dirs terms result in a
non-recursive search for modules (i.e. only modules found in
the given directories are included or excluded).

Note: Directories containing Erlang modules that are
to be included in a code coverage test must exist in the code
server path, or the cover tool will fail to recompile the modules.
(It is not sufficient to specify these directories in the cover
specification file for Common Test).

The cross cover mechanism allows cover analysis of modules
across multiple tests. It is useful if some code, e.g. a library
module, is used by many different tests and the accumulated cover
result is desirable.

This can of course also be achieved in a more customized way by
using the export parameter in the cover specification and
analysing the result off line, but the cross cover mechanism is a
build in solution which also provides the logging.

The mechanism is easiest explained via an example:

Let's say that there are two systems, s1 and s2,
which are tested in separate test runs. System s1 contains
a library module m1 which is tested by the s1 test
run and is included in s1's cover specification:

s1.cover:
{incl_mods,[m1]}.

When analysing code coverage, the result for m1 can be
seen in the cover log in the s1 test result.

Now, let's imagine that since m1 is a library module, it
is also used quite a bit by system s2. The s2 test
run does not specifically test m1, but it might still be
interesting to see which parts of m1 is actually covered by
the s2 tests. To do this, m1 could be included also
in s2's cover specification:

s2.cover:
{incl_mods,[m1]}.

This would give an entry for m1 also in the cover log
for the s2 test run. The problem is that this would only
reflect the coverage by s2 tests, not the accumulated
result over s1 and s2. And this is where the cross
cover mechanism comes in handy.

If instead the cover specification for s2 was like
this:

s2.cover:
{cross,[{s1,[m1]}]}.

then m1 would be cover compiled in the s2 test
run, but not shown in the coverage log. Instead, if
ct_cover:cross_cover_analyse/2 is called after both
s1 and s2 test runs are completed, the accumulated
result for m1 would be available in the cross cover log for
the s1 test run.

The call to the analyse function must be like this:

ct_cover:cross_cover_analyse(Level, [{s1,S1LogDir},{s2,S2LogDir}]).

where S1LogDir and S2LogDir are the directories
named <TestName>.logs for each test respectively.

Note the tags s1 and s2 which are used in the
cover specification file and in the call to
ct_cover:cross_cover_analyse/2. The point of these are only
to map the modules specified in the cover specification to the log
directory specified in the call to the analyse function. The name
of the tag has no meaning beyond this.

To view the result of a code coverage test, click the button
labled "COVER LOG" in the top level index page for the test run.

Prior to Erlang/OTP 17.1, if your test run consisted of
multiple tests, cover would be started and stopped for each test
within the test run. Separate logs would be available via the
"Coverage log" link on the test suite result pages. These links
are still available, but now they all point to the same page as
the button on the top level index page. The log contains the
accumulated results for the complete test run. See the release
notes for more information about this change.

The buttonc takes you to the code coverage overview page. If you
have successfully performed a detailed coverage analysis, you
find links to each individual module coverage page here.

If cross cover analysis has been performed, and there are
accumulated coverage results for the current test, then the -
"Coverdata collected over all tests" link will take you to these
results.