quickcheck-state-machine: Test monadic programs using state machine based models

This is a package candidate release! Here you can preview how this package release will appear once published to the main package index (which can be accomplished via the 'maintain' link below). Please note that once a package has been published to the main package index it cannot be undone! Please consult the package uploading documentation for more information.

Maintainers' corner

Readme for quickcheck-state-machine-0.2.0

quickcheck-state-machine

quickcheck-state-machine is a Haskell library, based
on QuickCheck, for testing
stateful programs. The library is different from
the
Test.QuickCheck.Monadic approach
in that it lets the user specify the correctness by means of a state machine
based model using pre- and post-conditions. The advantage of the state machine
approach is twofold: 1) specifying the correctness of your programs becomes less
adhoc, and 2) you get testing for race conditions for free.

The combination of state machine based model specification and property based
testing first appeard in Erlang's proprietary QuickCheck. The
quickcheck-state-machine library can be seen as an attempt to provide similar
functionality to Haskell's QuickCheck library.

Example

As a first example, let's implement and test programs using mutable
references. Our implementation will be using IORefs, but let's start with a
representation of what actions are possible with programs using mutable
references. Our mutable references can be created, read from, written to and
incremented:

When we generate actions we won't be able to create arbitrary IORefs, that's
why all uses of IORefs are wrapped in Reference v, where the parameter v
will let us use symbolic references while generating (and concrete ones when
executing).

In order to be able to show counterexamples, we need a show instance for our
actions. IORefs don't have a show instance, thats why we wrap them in
Opaque; which gives a show instance to a type that doesn't have one.

Next, we give the actual implementation of our mutable references. To make
things more interesting, we parametrise the semantics by a possible problem.

The pre-condition of an action specifies in what context the action is
well-defined. For example, we can always create a new mutable reference, but
we can only read from references that already have been created. The
pre-conditions are used while generating programs (lists of actions).

The transition function explains how actions change the model. Note that the
transition function is polymorphic in v. The reason for this is that we use
the transition function both while generating and executing.

If we run the sequential property without introducing any problems to the
semantics function, i.e. quickCheck (prop_references None), then the property
passes. If we however introduce the bug problem, then it will fail with the
minimal counterexample:

As we can see above, a mutable reference is first created, and then in
parallel (concurrently) we do two increments of said reference, and finally we
read the value 1 while the model expects 2.

Recall that incrementing is implemented by first reading the reference and
then writing it, if two such actions are interleaved then one of the writes
might end up overwriting the other one -- creating the race condition.

We shall come back to this example below, but if your are impatient you can
find the full source
code
here.

How it works

The rough idea is that the user of the library is asked to provide:

a datatype of actions;

a datatype model;

pre- and post-conditions of the actions on the model;

a state transition function that given a model and an action advances the
model to its next state;

a way to generate and shrink actions;

semantics for executing the actions.

The library then gives back a bunch of combinators that let you define a
sequential and a parallel property.

Sequential property

The sequential property checks if the model is consistent with respect to the
semantics. The way this is done is:

generate a list of actions;

starting from the initial model, for each action do the the following:

check that the pre-condition holds;

if so, execute the action using the semantics;

check if the the post-condition holds;

advance the model using the transition function.

If something goes wrong, shrink the initial list of actions and present a
minimal counterexample.

Parallel property

The parallel property checks if parallel execution of the semantics can be
explained in terms of the sequential model. This is useful for trying to find
race conditions -- which normally can be tricky to test for. It works as
follows:

generate a list of actions that will act as a sequential prefix for the
parallel program (think of this as an initialisation bit that setups up
some state);

generate two lists of actions that will act as parallel suffixes;

execute the prefix sequentially;

execute the suffixes in parallel and gather the a trace (or history) of
invocations and responses of each action;

try to find a possible sequential interleaving of action invocations and
responses that respects the post-conditions.

The last step basically tries to find
a linearisation of calls that
could have happend on a single thread.

More examples

Here are some more examples to get you started:

The water jug problem from Die Hard 2 -- this is a
simple
example of
a specification where we use the sequential property to find a solution
(counterexample) to a puzzle from an action movie. Note that this example
has no meaningful semantics, we merely model-check. It might be helpful to
compare the solution to the
Hedgehog
solution and
the
TLA+
solution;

The
union-find
example --
another use of the sequential property, this time with a useful semantics
(imperative implementation of the union-find algorithm). It could be useful
to compare the solution to the one that appears in the paper Testing
Monadic Code with
QuickCheck [PS],
which the
Test.QuickCheck.Monadic module
is based on;

Mutable
reference
example --
this is a bigger example that shows both how the sequential property can
find normal bugs, and how the parallel property can find race conditions.
Several metaproperties, that for example check if the counterexamples are
minimal, are specified in a
separate
module;

Circular buffer
example
-- another example that shows how the sequential property can find help find
different kind of bugs. This example is borrowed from the paper Testing the
Hard Stuff and Staying Sane
[PDF,
video];

Ticket
dispenser
example --
a simple example where the parallel property is used once again to find a
race condition. The semantics in this example uses a simple database file
that needs to be setup and cleaned up. This example also appears in the
Testing a Database for Race Conditions with QuickCheck and Testing the
Hard Stuff and Staying
Sane
[PDF,
video] papers;

CRUD
webserver
example --
create, read, update and delete files on a webserver using an API written
using Servant. The
specification uses two fixed file names for the tests, and the webserver is
setup and torn down for every generated program;

CRUD webserver where create returns unique
ids
example --
create, read, update and delete users in a sqlite database on a webserver
using an API written
using Servant. Creating a user
will return a unique id, which subsequent reads, updates, and deletes need
to use. In this example, unlike in the last one, the server is setup and
torn down once per property rather than generate program.

All examples have an associated Spec module located in
the
example/test directory.
These make use of the properties in the examples, and get tested as part
of
Travis CI.

To get a better feel for the examples it might be helpful to git clone this
repo, cd into the example/ directory and fire up stack ghci and run the
different properties interactively.

How to contribute

The quickcheck-state-machine library is still very experimental.

We would like to encourage users to try it out, and join the discussion of how
we can improve it on the issue tracker!

See also

The QuickCheck
bugtrack issue -- where
the initial discussion about how how to add state machine based testing to
QuickCheck started;

Finding Race Conditions in Erlang with QuickCheck and
PULSE
[PDF,
video] -- this is the first paper to describe
how Erlang's QuickCheck works (including the parallel testing);

Linearizability: a correctness condition for concurrent
objects [PDF], this
is a classic paper that describes the main technique of the parallel
property;

Aphyr's blogposts about Jepsen, which
also uses the linearisability technique, and has found bugs in many
distributed systems: