Does this function work as expected? I could run a few tries in GHCi, but that’s a little unsatisfactory: I have to do the work to think up a test, but I get to use it only once. Instead, it’s much better to write the test in my file, and that way I can re-run it every time I update my merge function.

The technique described above is often referred to as unit testing and is used exensively in the real world. But is unit testing even that great? Sure you can re-run all your tests whenever you want, but that doesn’t get around the issue that you actually have to write all of the tests in the first place. Coming up with specific test cases is often tedious, repetitive, and arbitrary.

Can we do better?

Property-based testing

Writing test cases is boring. And, it’s easy to miss out on unexpected behavior. Much better (and, more along the lines of wholemeal programming) is to define properties we wish our function to have. Then, we can get the computer to generate the test cases for us.

QuickCheck is the standard Haskell library for property-based testing. The idea is that you define a so-called property, which is then tested using pseudo-random data.

The first thing we notice is that our function is clearly wrong, with lots of stars and even an exclamation point! We then see that QuickCheck got through 5 tests before discovering the failing test case, so our function isn’t terrible. QuickCheck tells us what the failing arguments are: [] and [0]. Indeed GHCi tells us that merge [] [0] is [], which is wrong.

What’s so nice here is that QuickCheck found us such a nice, small test case to show that our function is wrong. The way it can do so is that it uses a technique called shrinking. After QuickCheck finds a test case that causes failure, it tries successively smaller and smaller arguments (according to a customizable definition of “smaller”) that keep failing. QuickCheck then reports only the smallest failing test case. This is wonderful, because otherwise the test cases QuickCheck produces would be unwieldy and hard to reason about.

(If you want to observe QuickCheck doing that, use verboseCheck instead of quickCheck.)

A final note about this property is that the type signature tells us that the property takes lists of integers, not any type a. This is so that GHC doesn’t choose a silly type to test on, such as (). We must always be careful about this when writing properties of polymorphic functions. Numbers are almost always a good choice.

In prop_sorted, we see the use of the operator (==>). Its type is Testable prop => Bool -> prop -> Property. It takes a Bool and a Testable thing and produces a Property. Note how prop_sorted returns a Property, not a Bool. We’ll sort these types out fully later, but I wanted to draw your attention to the appearance of Property there.

Let’s see how this works:

*Main> quickCheck prop_sorted
*** Gave up! Passed only 21 tests.

There aren’t any failures, but there aren’t a lot of successes either. We can see the problem if we use verboseCheck instead of quickCheck: QuickCheck will run the test only when both randomly-generated lists are in sorted order. The odds that a randomly-generated list of length n is sorted is 1/n!, which is generally quite small. And we need two sorted lists. This isn’t going to work out well.

QuickCheck’s types

How does QuickCheck generate the arbitrary test cases, anyway? It uses the Arbitrary class:

classArbitrary a where arbitrary ::Gen a
shrink :: a -> [a]

We’ll leave shrink to the online documentation and focus on arbitrary. The arbitrary method gives us a Gen a – a generator for the type a. Of course, the arbitrary method for lists doesn’t care about ordering (indeed, it can’t, due to parametricity), but we do. Luckily, this is a common problem, and QuickCheck offers a solution in the form of OrderedList, a wrapper around lists that have the right Arbitrary instance for our needs:

We can quickCheck anything that’s Testable. Boolean values are Testable, as are the somewhat mysterious Propertys. But it’s the last instance listed here of Testable that piques our curiosity. It says that a function is Testable as long as its argument has an arbitrary method, the argument can be printed (in case of failure), and the result is Testable.

Is [Integer] -> [Integer] -> BoolTestable? Sure it is. Recall that [Integer] -> [Integer] -> Bool is equivalent to [Integer] -> ([Integer] -> Bool). Because [Integer] has both an Arbitrary instance and a Show instance, we can use the last instance above as long as [Integer] -> Bool is Testable. And that’s Testable because we (still) have an Arbitrary and a Show instance for [Integer], and Bool is Testable. So, that’s how quickCheck works – it uses the Arbitrary instances for the argument types. And, that’s how changing the argument types to OrderedList Integer got us the result we wanted.

Generating arbitrary data

When you want to use QuickCheck over your own datatypes, it is necessary to write an Arbitrary instance for them. Here, we’ll learn how to do so.

If we want an Arbitrary instance, we must define the arbitrary method, of type Gen (List a). Luckily for us, Gen is a monad (did you see that coming?), so some of its details are already familiar. We also realize that if we want arbitrary lists of a, we’ll need to make arbitrary as. So, our instance looks like

instanceArbitrary a =>Arbitrary (List a) where
arbitrary = genList

At this point, check out the combinators available in the “Generator combinators” section of the QuickCheck documentation.

It’s helpful to think about how you, as a human, would generate an arbitrary list. One way to do it is to choose an arbitrary length (say, between 0 and 10), and then choose each element arbitrarily. Here is an implementation:

The lengths are unbounded, but we’re getting a lot of empty lists. This is because at every entry in the list, there’s a 50% chance of producing Empty. That means that a list of length \(n\) will appear only one out of \(2^n\) times. So, lengths are unbounded, but very unlikely.

The way to make progress here is to use the sized combinator. QuickCheck is set up to try “simple” arbitrary things before “complex” arbitrary things. The way it does this is using a size parameter, internal to the Gen monad. The more generating QuickCheck does, the higher this parameter gets. We want to use the size parameter to do our generation.

That worked nicely – the lists tend to get longer the later they appear. The idea is that sized takes a continuation: the thing to do with the size parameter. We just use a lambda function as the one argument to sized, where the lambda binds the size parameter, and then we can use it internally. If that’s too painful (say we just want to produce the size parameter, without using a continuation), you could always do something like this:

getSize ::GenInt
getSize = sized return

I’ll leave it to you to figure out how that works. Follow the types!

As one last example, we can also choose arbitrary generators from a list based on frequency. Although the length method of genList works well for lists, the following technique is much better for trees:

It takes a list of (Int, Gen a) pairs and produces a Gen a. The numbers in the list give the likelihood of choosing that element. Above, we fixed the frequency of Empty at 1, but let the likelihood of Entry vary according to the desired size. Then, in the recursive call to genList, we used resize to lower the size parameter. Otherwise, it’s too likely to get a runaway list that goes on toward infinity.

Conclusion

Property based testing is an amazingly effective and satisfying approach to testing. It works especially well in Haskell because of the combination of purity, which means that one only needs to vary the arguments to a function, and the type system, which allows us to use type classes to very generically and composable create code that comes up with random input. QuickCheck came out in 1999 has since then inspired re-implementations in other languages as well.

Practical hints

Very likely, your project has some code that can be tested this way, and you should do that. Here are some pointers to look at.

QuickCheck is random, which is nice. An alternative approach is to test all input, up to a given size. This way your test suite is deterministic, and at least no small corner cases remain. A library implementing that approach is SmallCheck.

You do not want to run your tests in GHCi only. There are libraries that allow you to name and group your tests and create an executable that runs your tests and has nice shiny colored output. Check out tasty for one of those. It has support for QuickCheck, SmallCheck, HUnit and more.