Random inputs in unit testing

I recently had an upstream reviewer telling me that I should not randomise my test input because “randomness does not provide a useful input to the test and sets a bad example of practices for writing tests”.

I am going to explain here why this is wrong and it’s actually good practice to randomise inputs. Let me start by saying that random test failures are not the same thing as spurious test failures. I’ll come back to that later.

Consider this simple code under test, it’s a contrived example but you will get the idea:

Here, I am using a fixed input of “foo” as many people like to do in tests as a way of saying “this value is irrelevant”.

The bug should be obvious here — the test passes when it should not because the code under test is returning the same value as used in the test. As I say a fairly contrived example, but it illustrates the point that tests should never assume anything about code under test.

(A further improvement could be to generate a random string, but I’ll leave that for a future blog entry.)

Here, we’re generating a random input and asserting that the returned value is the same as the input. This not only avoids the bug above but it is far better at demonstrating test intent. It will also never fail unless the code under test is buggy, and that brings me back to the point above about random vs spurious test failures.

A random test failure is good. It means you found a bug! A spurious test failure is one that indicates you’re not testing properly – an example of this is where you depend on some network connectivity to complete your test; as networks are inherently unreliable this is a bad test and will create spurious test failures when the network fails.