Random Test Inputs – part 2

If you saw my last post Random inputs in unit testing you’ll remember that I was advocating the benefits of using random test data in your unit tests.

One of the bits of feedback that I had (and seems to be the main complaint of most people who are against this) is that you need reliable test input so that you can recreate failures.

I don’t think these two concepts are mutually exclusive. Let me explain.

A decent test framework will help you recreate failures

I use testtools as my preferred test framework for Python. One of its benefits is that its so-called Matchers will output useful information for failures, including the data used for input. As a very basic example:

Thus, if you get a failure from a random input, it’s easy to see what data caused that.

Making your data recognisable

Once you start randomising a lot of input data, it starts to become hard to identify which input is which when you have a failure for code that has many inputs. Take this for example, which I have taken from my days as the MAAS engineering lead: