Automated Testing – Bringing out the big guns – Part 1

Since the last gen_fsm test post already tip-toed on the subject of Automated Testing, I thought it would be natural to go into QuickCheck and “bring out the big” guns.

Enter QuickCheck.

What is quickcheck?

QuickCheck for Erlang is a commercial tool from QuviQ for Property Based Testing by automatic generation of testdata that is fed into property specifications. Behind this we find John Hughes and Thomas Arts. Written in Erlang.

For the Haskell fans out there, I hope you recognize QuickCheck from the Haskell history, through this incredibly well-designed page (irony). As a side-note, I did firewall testing with the Haskell QuickCheck some time ago for a company called Witsbits.

How do I install it?

This is seriously easy, and takes under 10 seconds (no honestly). Unzip the eqcmini.zip archive and you have your “installation”. As the product is shipped with the beamfiles, you don’t need to (you actually can’t) compile anything.

How do I use it?

You need three things in order to get QuickChecking (qc’ing : queue-see-ing)

A module including the quickcheck application header file,

-include_lib(“eqc/include/eqc.hrl”).

Inside the same module, properties you will run your test on, each property must be exported.

A module with the actual implementation of what we wish to test.

This is very similar as to EUnit testing. We have a test-module consisting of tests and the EUnit header file, and an implementation file with the actual code.

The idea is that the test module exports the properties that will be tested by quickcheck.

How can I do TDD with QuickCheck?

For the purpose of this example, we shall develop parts of a small clock library that makes it possible to add / subtract time with its basis in the time() format.

As usual, we will probably a standard layout with the minimals, ebin/ src/ test/ and a makefile.

First we see that the the property has arity 0 (zero). The name also begins with prop_. The ?FORALL is a quickcheck macro

?FORALL(X,Gen,Prop)

with the first element being the pattern; in this case the 3-tuple {H, M, S} which will be pattern matched for value binding to generated values from the Generator. The second element is the Generator from where values (3-tuple of natural numbers) are generated and bound to the Pattern and the third being the Property. For this example, the property is everything from the next line (?IMPLIES…) to the end.

The ?IMPLIES macro has two elements. A precondition (for this example: the (H < 24)) works as a filter, only allowing H values with values in the range 1 – 23, it discard all tests for which the precondition does not hold and runs the tests for which the precondition is valid. Second element is the Property (for this exmple: everything from ?IMPLIES(M < 60)… and onward).

For this code piece, we see that there is one Pattern and generator, three natural numbers, then 3 preconditions, ensuring that only sane values are given for (H)ours, (M)inutes and (S)econds. When the sifted values are accepted, the last part can be thought of as the property: The function call hms2s/1 must return the amount of seconds for the 3-tuple. Here we are doing a crucial thing, which is basic to quickcheck. We are comparing the function value with our model.

The clock.erl module is totally blank except for the module declaration, and the Make file holds

A very interesting thing we see from the above code is that when doing property driven development is that in developing the model for the property, one may find the solution to the implementation. In contrast to EUnit tests, this is one step closer to the implementation.

You can take my word for that this will pass the tests, or try it yourself with make test. Next is the addition and subtraction functions. They will allow us to subtract and add seconds, minutes and hours from a given time() 3-tuple.

First, we write a test, based on the previously (seemingly) okay functions, for the internal model

Here we see that QuickCheck generated a minimal failing case with Hour = 1, Minutes = 0, Seconds = 0 and Subtracting 1 second fails. Using these values with the model (in the console)

1> clock:s2hms((clock:hms2s({1,0,0}) - 1)).
{0,59,59}
2>

we see that the model seems sane. Removing one second from an hour produces 59 minutes and 59 seconds. All okay. Now what about my broken implementation? Well, obviously it did not account for “lending”. Since we want the actual implementation to be quicker than the model, it would be cheating to copy the model.

Yes. Nice! But how sure are we that this is not running trivial tests? Well, just check WHAT it is testing, by collecting the testdata! Note the added eqc:collect/2 function for the line after the last precondition.

with A LOT of lines, so if you don’t trust your model OR your implementation, you are welcome to check out the used values by hand. For now, we are good.

I hope this first post on QuickCheck shows that you CAN do TDD with QuickCheck and also wet your appetite for QC if it was your first encounter with it. Next time I will continue on this clock lib and try to show more QC stuff.

2 Comments

Andre

Having read this I thought it was extremely informative.
I appreciate you taking the time and energy to put this information
together. I once again find myself spending a lot
of time both reading and leaving comments. But so what, it was still worthwhile!