Inkscape Testing Framework

In order to detect and identify bugs quickly and reliably, we should set up a testing framework for Inkscape.

There are several different ways to test a desktop Linux application. Each method provides different kinds of
information about the codebase and are each valuable in different ways. A multi-pronged approach that uses
more than one kind of testing is expected to gain the best results.

Compile Testing

In compile testing, the codebase is repetitively checked out and compiled in one or more different ways.
For example, with different compilers or for different platforms (Linux, Windows, Mac OSX, ...) Errors in
configuration, compilation, linking, or basic executation are detected, collected, and reported on a periodic
basis (such as daily compile reports).

It may be possible to set up compile testing using SourceForge's compile farm. Another option to
investigate is the Mozilla project's "Tinderbox".

Unit Testing

Unit testing is implemented at the file level. Each *.c or *.cpp file is provided with a test file containing
calls for each of the functions, and that exercises the API, including a wide variety of error situations such
as NULL pointers, buffer overflows, very large or very small floating point numbers, zero or negative int's,
uninitialized objects, etc.

The intention of unit tests is to provide sanity checks that the developer can use as they implement the
code. Ideally, when implementing a new code file, the programmer could first implement the unit test
from a spec, and then work on implementing the code to pass the unit test; this is the development
style advocated in the Extreme Programming philosophy.

Regression Testing

In regression testing, the philosophy is to implement test cases for each bug found, and to run these
test cases against the codebase to determine that bug fixes actually fix the bug, and to ensure in the
future that the bugs do not re-occur.

This can be implemented by assembling a repository of "problem" SVG files that have caused Inkscape
to error in programmatically detectable manners (such as crashes or emission of invalid SVG or PNG's).

GUI Testing

To eneable a11y in Inkscape for testing with dogtail, start Inkscape with the following command.

GTK_MODULES=gail:atk-bridge inkscape

Fault Response Testing

In fault response testing, the application is put into various critical situations and the behavior
detected and compared against expectations. Examples include out-of-disk-space, out-of-memory,
response to different signals, loading and saving very large files, starting up with invalid or unreadible
config files, and so forth.

Specification Compliance Testing

In spec testing, the application is run against various input cases and caused to generate output that
can be tested against "known" correct data. For instance, the W3C produces an SVG compliance test
suite with SVG files and the expected rendered output as PNG's. Inkscape can be run on each of the
SVG's from the commandline and caused to save the rendering as a PNG. An image-difference tool
can then be used to calculate a diff-score between the emitted PNG and the W3C image.

Similarly, this process can be used against other applications such as Batik or librsvg, to check
inter-application conformance on the same SVG files. Ideally, all SVG applications should render
a given SVG file identically.

Performance Testing

In performance testing the application is measured as to how well it works against various criteria such
as start-up time, output quality, processing speed, etc.

Valgrind Memory Leak Testing

What I did here is simply ran inkscape through valgrind with the following command: