Search This Blog

Property-based testing with ScalaCheck - custom generators

In the previous article we examined how Spock can be used to implement property-based testing. One of the "hello, world" examples of property-based testing is making sure absolute value of arbitrary integer is always non-negative. We did that one as well. However our test didn't discover very important edge case. See for yourself, this time with ScalaCheck and ScalaTest:

ScalaCheck tells us that our property is not met for input = -2147483648. What's so special about this number? ints aren't symmetric, Integer.MIN_VALUE = -2147483648 while Integer.MAX_VALUE = 2147483647. It's not possible to represent 2147483648 in Int:

To stay with the spirit of functional programming, our Bank implementation is immutable (accounts is of scala.collection.immutable.Map type), as well as Account and AccountNo. Every time we call Bank.transfer(), new instance of Bank is created, almost exactly the same, but with from and to accounts modified. This greatly simplifies coding in multi-threaded environment. Code is quite simple: take amount of money from one account and put it on another. Assume we have few example based tests and we are confident this code works. But to be extra safe we are going to build property based test. What is the property that will be satisfied, no matter how many transfers we perform? The most important one is that the total money in the bank should remain the same, no matter how many intra-bank transfers are executed. After all, we don't want money to disappear or appear from nowhere.

Our test should prove that any bank, with any number of arbitrary transfers has the same total amount of money before and after executing transfers. We start with simple:

What we are saying is: for any bank and any List of transfers, totalMoney before and after should remain the same. We must foldLeft() because Bank is immutable and every transfer must be applied on a Bank instance returned from a previous one. ScalaCheck can generate random Ints (as we saw in AbsSuite) and other primitives, strings, etc. - and collections of these. But ScalaCheck has no idea how to create random Bank or Transfer:

What the compiler is telling us is that it can't find a type class org.scalacheck.Arbitrary[T] type-parameterized with Bank. There are instances of this type class for primitives or collections, but obviously not for our Bank. There are actually two abstractions we need to provide: Gen implementation and Arbitrary type class wrapping it. Let's go through it step by step. accountNoGen generates random AccountNo with values ranging from 100000 and 999999. Gen is like a stateless stream of data, it produces possibly infinite number of random values. You might wonder, why not just use Math.rand()? We can, but this way ScalaCheck can instrument all generated random data and e.g. allow replying it later, when the same random seed is used.

We are now ready to generate random Bank. It takes an arbitrary number (Gen.containerOf[List, Account]) of arbitrary accounts (accountGen), but we don't want to generate empty banks or banks with too many accounts:

The last piece is a random Transfer. This part is actually more complex. In order to generate arbitrary transfer we need two random accounts from a bank. But we don't know accounts yet, since bank with accounts was generated randomly. Thus our generator must be parameterized with a bank that was earlier randomized. The difference between accountNoGen and accountNoInBankGen is that the latter picks an existing account number from a given bank, rather than an arbitrary, random number. In arbitraryTransfer we don't have to pass bank explicitly because it is marked as implicit:

Unfortunately check((bank: Bank, transfers: List[Transfer]) won't work. Bank and List[Transfer] are generated "at the same time", so there is no way to pass generated bank to transfers generator. We have to go deeper, using different ScalaCheck syntax (forAll), abusing it slightly:

Money doesn't add up! Looking carefully we see that the test failed with just one account and one transfer. By repeating the test we can easily find the pattern: single transfer with the same source and target account (664482 this time)! Scroll back to our implementation and try to figure out why (remember about immutability):

Be sure you understand why the two code snippets are fundamentally different. Hint: compare accounts(to) and accountsMinusAmount(to). OK, it works, but I see way too many identifiers and noise, let's go more functional:

Private Bank.update() modifies one account by applying a custom function on top of it. We call this higher-order function twice, once to modify from account, later to modify to - but this second application works on top of already modified Bank instance.

One thing we haven't covered is shrinking (noticed // 1 shrink comment in test failure message?) ScalaCheck produces random, sometimes really large input, for example very long list of random transactions. Imagine just one transaction in hundreds causes error. If ScalaCheck finds such a list and reports it, discovering which particular transfer caused bug can be a challenge on its own. Thus ScalaCheck, using various heuristics, tries to shrink generated input in order to find the smallest one, still exhibiting erroneous behaviour. In our case it's a matter of selectively removing transfers from an input list ("shrinking" it), until we find the smallest subset still exposing a bug. This time-saving process is called "shrinking". More importantly we can customize it, for example telling the framework how to shrink Bank to a smaller, still problematic instance.

As you can see property based testing can be useful. It doesn't replace example based testing. Moreover, every time you find a bug using ScalaCheck, you should start with writing an example test that fails (and fails all the time, not from time to time). Remember that property based tests are randomized so they will not always find all bugs - and even worse, sometimes they will find bugs much later. Such tests are valuable, but they will never replace ordinary, predictable tests.