The Goal

Improve stability of the Common Lisp ecosystem
by performing automated tests on as wide set of
environments as possible. Environments vary
mainly in 3 dimensions:

Common Lisp implementations. They have incompatibilities,
sometimes allowed by the standard (features specified
as implementation-dependent), sometimes due to bugs,
and sometimes in non-standardized features (threading,
sockets, FFI, etc.). Moreover, implementations are
evolving over the time, we should care about different
versions of the same implementation.

Growing number of CL libraries (which are also evolving
over the time).

Compatibility between library versions - libraries
depend on other libraries and work correctly
only with particular versions of the dependencies.

Therefore, when we run test suite of some library, we can speak
about success or failure only in context of given Common Lisp
implementation, and versions of all the dependency libraries.

Lets call the set of libraries with specified versions a “lib-world”.
Important example of lib-world are Quicklisp distros.

It is hoped that constantly running tests on wide variety
of environments will help the CL community by:

Fast response to the library authors in case new
changes cause bugs on implementations not available
to the author. Otherwise author may receive notification
about the bug years after he made the change, and
thus the cost of fixing the bug may be much higher than
fixing it week or two after the change.

The same benefit for CL implementors - when they
release new version, run test suites of large
number of libraries and quickly detect
possible regressions in the new release.

Help to discover and maintain compatible
set of library versions (e.g. Quicklisp distros).

Limitations.

Test suite success does not always mean the library is workable -
there might be bugs which are not covered by the tests.

And the other way around - failed tests not always
means the library is broken - it may be just
a bug in the tests themselves.

Reducing this gap increases the utility of automated testing.

The Implementation Idea

Everyone can run a simple command which will run tests
of Common Lisp libraries and publishes results
online.

That way, instead of setting up a central build farm with
all the possible hardware/OS/Lisp implementation combinations,
we provide a way for Common Lisp users to contribute
test results from their systems, and collectively
monitor the CL world. (Of course, if anyone would want
to setup a test farm, cl-test-grid simplifies this task too,
by providing required building blocks).

Status

We provide tools to:

run tests

publish results online and access them programmatically

perform massive comparison of results and print HTML
reports. For example, compare results of two Quicklisp
versions, or compare two versions of a lisp implementation
on the same Quicklisp.

The tests are run by a lisp program called test-grid-agent.
User configures it with a list of CL implementations installed
on his machine, and test-grid-agent runs tests of common lisp
libraries on these implementations.

The tests performed by agent include:

fresh recompilation and loading of every ASDF system found in Quicklisp;

testsuites of some of the libraries.

For testsuites we have so far considered 129 libraries,
starting from more often downloaded ones.
74 of these libraries have appropriate test suites
(fully automated, no user interaction is needed) which
are added to the cl-test-grid.

The test results represented as lisp data are stored online
in so called test-grid-storage. By default test-grid-storage
named “main” is where the test results are published.

You may also choose other storage name if you want to do some private
testing and don’t want to mix your results with “main” results.
The use cases for this are pre-release testing of CL implementations,
ASDF, or patches to particular libraries.

Besides this, we store online the output produced by
each lisp process running test suites or compiling ASDF systems.
The logs are referenced from the test results lisp data.
This allows interested parties to navigate to the corresponding
log to study the failure details.

Some HTML reports are demonstrated above, in the Test Results
section. More information about reporting may be found
in the test-gird-reporting module documentation.

Participation

Running tests

Adding testsuite of your library

It’s necessary to contact the cl-test-grid mailing list
and add a method for the libtest generic function to this file:
testsuites/testsuites.lisp. The libtest function converts
results of library test suite to a unified format
understood by test-grid.

If you use some of the popular CL test frameworks,
the implementation is usually simple, like this: