2009-07-28

I think I've found some answers to my practical QuickCheck questions. This post may be fairly long as I'm trying to make it concrete and explicit enough to overcome the kind of inertia I had when I was still resisting testing.

The code snippet for your Setup.hs file comes from Greg Bacon's Setting up a Simple Test with Cabal (I tacked on an import). As you can see, the recipe assumes you're building an executable called "test" (see Greg's post on how to do this)

3. Bake your unit tests in

This may go down as the kind of bad advice that "seemed like a good idea at the the time". For now, I can justify this by saying that it may be reassuring to users to be able to just run the same tests that I'm running and see for themselves that their program thinks it's working.

I've been working on a program called GenI. To help people test this program, I've added a simple "--tests" switch. Now people can run geni --tests for a self check. If they want, they can also "cabal test", using this slight modification to Greg's setup file (to call geni itself and to pass the --tests flag in).

As for GenI, whenever I see --tests in my arguments (for example "--tests" `elem` args), I just pass control to another module, which in turn strips the switch out and passes the rest of the arguments to test-framework.

There's some other things going on in this file, notably the organisation of test suites. More on that later.

Where should I put my properties?

4. Put tests in the same module (where relevant)

If a test is specific to one module, I tend to put them in that same source file. I do this because

It lets me test functions that I don't want to export

The tests serve as documentation

It forces me to update my tests along with my code

This approach is in contrast to (a) having one big tests module and (b) having a separate test hierarchy. It may turn out to be useful to have a single big tests module as well, for example, for tests that cross the boundary from one module to the next. That need has not arisen for me yet. Likewise, I don't particularly believe in a separation between tests and code, although on the other hand some very experienced hackers seem to do so, so I'll just have to let experience teach me why.

How do I avoid repeating myself?

5. Provide a testSuite function for each module

Commenting on my last post, Josef kindly pointed out that the book-keeping I feared isn't so bad in practice. He's right. Nevertheless, I want to avoid it. To do this, I make each of my modules export a testSuite function. Here is what one of my modules looks like, just focusing on the test suite

If you'll scroll up to the example that's marked TEST-FRAMEWORK WRAPPER, you'll see how these test suites are used in practice. Note the small trick of using the qualified module name to identify the test suite.

Anyway, the general principle of having a per-module test suite comes from Aidan Delaney's Organising Unit Tests in Haskell. The main difference between his approach and my approach are that I mix tests and code rather liberally.

Conclusion

I hope that some of these hints will make testing easier for you, or perhaps even get you started. If you still find yourself putting testing off, let me know. I'll be curious to see what else makes us resist. One thing that would probably be helpful is an extra guide to writing Arbitrary instances for QuickCheck, and also writing good properties that control the space well. Maybe even getting started with SmallCheck.

Note that I am still somewhat new to testing and have only recently started these practices. So take these ideas with the usual salt. Thanks to Greg, Reinier, Aidan, and also folks who commented on my previous posts.

I think I've found some answers to my practical QuickCheck questions. This post may be fairly long as I'm trying to make it concrete and explicit enough to overcome the kind of inertia I had when I was still resisting testing.

The code snippet for your Setup.hs file comes from Greg Bacon's Setting up a Simple Test with Cabal (I tacked on an import). As you can see, the recipe assumes you're building an executable called "test" (see Greg's post on how to do this)

3. Bake your unit tests in

This may go down as the kind of bad advice that "seemed like a good idea at the the time". For now, I can justify this by saying that it may be reassuring to users to be able to just run the same tests that I'm running and see for themselves that their program thinks it's working.

I've been working on a program called GenI. To help people test this program, I've added a simple "--tests" switch. Now people can run geni --tests for a self check. If they want, they can also "cabal test", using this slight modification to Greg's setup file (to call geni itself and to pass the --tests flag in).

As for GenI, whenever I see --tests in my arguments (for example "--tests" `elem` args), I just pass control to another module, which in turn strips the switch out and passes the rest of the arguments to test-framework.

There's some other things going on in this file, notably the organisation of test suites. More on that later.

Where should I put my properties?

4. Put tests in the same module (where relevant)

If a test is specific to one module, I tend to put them in that same source file. I do this because

It lets me test functions that I don't want to export

The tests serve as documentation

It forces me to update my tests along with my code

This approach is in contrast to (a) having one big tests module and (b) having a separate test hierarchy. It may turn out to be useful to have a single big tests module as well, for example, for tests that cross the boundary from one module to the next. That need has not arisen for me yet. Likewise, I don't particularly believe in a separation between tests and code, although on the other hand some very experienced hackers seem to do so, so I'll just have to let experience teach me why.

How do I avoid repeating myself?

5. Provide a testSuite function for each module

Commenting on my last post, Josef kindly pointed out that the book-keeping I feared isn't so bad in practice. He's right. Nevertheless, I want to avoid it. To do this, I make each of my modules export a testSuite function. Here is what one of my modules looks like, just focusing on the test suite

If you'll scroll up to the example that's marked TEST-FRAMEWORK WRAPPER, you'll see how these test suites are used in practice. Note the small trick of using the qualified module name to identify the test suite.

Anyway, the general principle of having a per-module test suite comes from Aidan Delaney's Organising Unit Tests in Haskell. The main difference between his approach and my approach are that I mix tests and code rather liberally.

Conclusion

I hope that some of these hints will make testing easier for you, or perhaps even get you started. If you still find yourself putting testing off, let me know. I'll be curious to see what else makes us resist. One thing that would probably be helpful is an extra guide to writing Arbitrary instances for QuickCheck, and also writing good properties that control the space well. Maybe even getting started with SmallCheck.

Note that I am still somewhat new to testing and have only recently started these practices. So take these ideas with the usual salt. Thanks to Greg, Reinier, Aidan, and also folks who commented on my previous posts.

Thanks for the link! For the moment, my test logic looks like it's cleanly separated from the rest of my code, no crazy interleaving (thanks to the functional style, perhaps?). But it's interesting to consider.

I know some great programmers who are pretty clear that this is a bad idea, so maybe I will burn my fingers on this :-)

I've another reason to separate tests from functionality: simplifying dependencies. Since my tests are in their own files and have their own executable section in my .cabal, the main of the program doesn't need to depend on QuickCheck/HUnit/etc. This is a win if someone ever wants to link my library and use, e.g. QuickCheck-1 rather than QuickCheck-2.

Mark Wotton and I have recently tried to address this problem. TBC has Cabal integration, a fact I should have pushed more in my haskell-cafe announcement.

Hackage: http://hackage.haskell.org/package/TBC

It makes tests easy to run by adopting conventions for identifiers, e.g. "prop_blah" is a QuickCheck property, etc. No more forgetting to add tests to lists!

The other key feature is that it will run all runnable tests, so you can actually do test-driven development.

"cabal test" is broken, forget about it for now. TBC provides a hook but it is a lot more primitive than the 'tbc' binary. A quick look at Cabal's trac will show that the 'test' hook has been in need of love for years.

TBC is agnostic about where the tests go - conventionally, separate tests go in $PROJECT/Tests, but you could bake them into your source code if you like.

Peter: anything that will let me write less code is welcome! I'll add it to my list of things to try.

Echo and Ivan: points taken. Ivan, I don't really mind making folks install QC because it just automagically happens with cabal anyway, although I do sympathasise more with Echo's point that people might like more control, for example over QC version numbers.

I definitely seem to be in the minority here about mixing tests with code, and also about baking them in and probably for good reason. Hopefully I will figure it out before I have to learn the hard way!