jeudi 1 mai 2014

You say that a value is not used, prove it

Introduction

Some time, we are doing tests by considering that some values are not relevant for the test so we put default value. But some time we actually rely on those default values to have our tests working. It can lead to hidden knowledge and false passed tests

For instance, if I create a class OldWomen. For my test instances It's reasonable to put 70 as a default value. It's an age that anyone (at least any developer) will consider as old.

Let say I have a medicine dispenser to test. I can have a story like this

Given a old woman with a cancer
when medicine dispenser is turned on
then the old woman remains in life

But for any reasons we decide that Women older than 80 always forget to plug in the medicine dispenser.

...our test will continue to work

but in fact if the old woman is older than 80, she will die. (Indeed it's a simplified world)

Your test is not relevant and can make old women died.

Improving your default values

We are all using continuous integration, a same test in a normal team will probably be run more than 10 times a day (Most of the time just to say "Hello I'm here and I'm working fucking well").

We cannot test all cases

We haven't the brain enough big to foreseen all impacts of our changes (If we could, we basically won't do any test).

So let's make the machine do the job. Let's implement a default value generator !!!

By randomly generating default value you will probably failed one day (and probably the first day) If your code isn't safe for all values.

Obviously this approach has some limitation and some corner cases will not be threated.

Some ideas to implement those generators

Create a generic class RandomValueGenerator<T>. You will be able to handle many cases this way

When you create String value take care of alpha numeric/ASCII characters

Generate nullable value if it's relevant (for example 50% of the values returned will be null ). You can create a decorator to do that

You need to find a good way to call this newTest method on every tests. It's your job here !!!

Non regression testing

At this point we tested that unexpected change in GIVEN values has no impact on THEN values but we also want to be sure that only expected values has changed on a test. Let's take again our old example:

Given a old woman aged of 90 with a cancer
when medicine dispenser is turned on
then the old woman will die

But it's not only that the old woman will die that changed, there is also a good news, the electricity bill will not increase dramatically before people discover the old woman.

When you code do you always have in mind this kind of change ? It why non regression testing are so important. Although, I don't want to change my test just to handle that case (It probably doesn't worth it) but I want to acknowledge that this change in behavior is ok.

There is no easy solution to solve this problem, but here some ideas :
- You need to compare state of the system at the end of the test with the expected state. It could be difficult or simple depending on your system. If your are in a db oriented project you may store the resulting database after the test.
- For the first run you cannot test anything (It's non regression testing... ) but you should store the results of this first test in the source base of the project.
- For non regression testing always use the same seeds. You can use lot of different seed but for each seed you should expect to have the same result.
- Reuse the seeds that have in the past broken your test.