A journey to ScalaCheck

From the Spanish good weather to the Dutch every-possible-weather-in-one-day, from Waterfall to Agile, from just testing to property based testing. The path I took when joining Lunatech was an interesting one.

I want to share some of that and show how my journey to ScalaCheck started. I will prove that it’s not complicated to start with and it can uncover deeply hidden bugs in your code.

As developers, we need to be sure that we create code that performs exactly how it is meant to. This should be true in every possible scenario. However, how can we prove that our codebase actually does this for a wide range of data? Sometimes it is just not feasible to write innumerable amount of test cases for a specific function. We need to find a way to somehow prove our function works as expected in every possible case.

Property-based testing provides another way of thinking, that was new to me, about writing tests. Sometimes it is better to prove that a function satisfies a specific property, rather than to write a number of tests which try to confirm it is working fine. One way of proving is to generate an appropriate amount of data and apply these data to your test suite. These generated data should all have the same specific property, hence the name property-based testing.

As an example, imagine we want to test String concatenation. To do this we need to be sure that: For all given two strings, str1 and str2, the result of concatenating both strings must satisfy: str1.length + str2.length >= str1.length

In this small piece of code, we declare a property ("Concatenation length of two strings …​"), that holds forAll possible cases of concatenating 2 strings (s1 and s2). This seems reasonable but how can we prove what this property holds true. One way is by creating a lot of tests. And that is where Generators come in handy.

Generators

To generate this input data, ScalaCheck provides us with a wide range of generators available in objects Arbitraty and Gen.

The org.scalacheck.Arbitrary module defines implicit Arbitrary instances for common types, for convenient use in your properties and generators:

returns an arbitrary generator for the type T

The org.scalacheck.Gen uses Arbitrary and offers various generators:

alphaLowerChar, alphaUpperChar, alphaNumChar

identifier, alphaStr, numStr

negNum, posNum, chooseNum

listOf, listOfN, nonEmptyListOf

choose, oneOf, someOf

const

Some examples using arbitrary/generators:

id ← arbitrary[Int]

married ← arbitrary[Boolean]

age ← choose(0, 120)

currency ← const("euro")

description ← arbitrary[String]

However, most of the time we do not want to check such a general data type. For this, ScalaCheck also offers the possibility of defining custom generators where we can establish what the input data should look like.

Let’s use a simple example to understand the usage of custom generators. Imagine we are a Benelux bank that wants to verify that their Dutch customers who have a negative balance in at least one of their accounts, should be notified by email. For simplicity, we define customer and bank account as below:

Something which is worth mentioning at this point is the usage of .suchThat. It is recommended not to write very restrictive conditions in this filter, because ScalaCheck first generates all input data, and filters it later based on the condition provided. If the condition is too restrictive, it may end up with too many inputs discarded and the tests will not run.

To conclude with generators, let’s have a look to a sample of our Dutch customer with at least one account with negative balance:

This shows us that maybe we should add some conditions to the accountId or the balance, because it is not normal to deal with such values in real life. This was for example one of the reasons to create scalaCheck-datetime

Writing tests

Now that we are familiar with properties and generators, it is time to write tests. We have good examples in the Scala community, because ScalaCheck is used by many Scala open source projects (like Akka or Play).
In this case, we will continue with our concatenate example.

Our properties file can be as simple as that, or we can make it as complicated as we need. We can also integrate it with ScalaTest or Specs2

Running ScalaCheck tests

Using sbt, we run ScalaCheck tests in the same way we run ScalaTest tests: sbt test:compile test. If our code is correct and all the tests generated by ScalaCheck are successful, we can see the following as output:

By default, ScalaCheck generates 100 tests per property, which must be satisfied for the test to pass.

In case a property is not satisfied by the generated test data, ScalaCheck yields an error. And not only shows the input data which makes the property to fail, but it also simplifies as much as possible to show you the minimum value which makes the test to fail. This helps us a lot when going back to the code and applying a solution to fix the wrong implementation.

How ScalaCheck helps with finding bugs

If you are not yet convinced we’ll give you another example of code that looks fine at first glance, but will not meet the requirements.

What ScalaCheck is showing is that the property fails for input = -2147483648 Then, we realize that Int numbers are not symmetric Int.MaxValue = 2147483647 Int.MinValue = -2147483648 So, when trying to apply abs to Int.MinValue, we get Int.MinValue.abs = -2147483648 which does not satisfy the condition of input.abs >= 0.

It is very likely that we write our code without thinking about these kind of corner cases, because we probably never expect an input with value -2147483648 But since -2147483648 is valid input our code will accept it and will crash if we do not add conditions to prevent it.

ScalaCheck focuses mainly on corner cases, where our functions are more sensible to fail. So for Int values, it will first test with MIN_VALUE, MAX_VALUE and 0; for String values will test with symbols and non-roman alphabet.

Useful links to get started

Summary

When you feel you are adding many tests based on input data, stop for a moment and think twice about the possibility of translating the functionality into a property that ScalaCheck can test for you.

If we can write properties for a given function, ScalaCheck provides an easy and very intuitive way of writing tests, which automatically generate large amounts of data for us, mainly focusing on corner and special cases.It is very helpful that ScalaCheck shrinks test cases to the minimal case.

ScalaCheck does NOT substitute ScalaTest or Specs2, but it complements them with property testing.Don’t forget that ScalaCheck is generating a finite number of tests, which means that there is always a chance that within this randomized set of tests, a bug might not be found (although it does exist in your code). However in case your input type is more constrained e.g. Byte, it can even generate all possible input data.

I started with ScalaCheck soon after I started with Scala and it changed the way I look at tests. Be always open to explore and try new options, because from all of them you will always learn something useful.