Clearly the code disagrees with the test name. Since the name of the test is more likely to correspond to the rule I have in my head that I believe the system follows, this test could be extremely destructive to my system.

Test Absence is when a test is absent, but we assume behaviour anyways. This is a logical mistake on our part, but it leads to bugs in the system.

Now the Queen and I are, in fact, two distinct people. But 2 = 1, which means the Queen and I are one. So you see, I am in fact the Queen of England.(Paraphrased from memory)

Honestly, I probably would have chosen to prove I was Metallica (the entire band), but if one admits that 2 = 3, the rest of the logic is difficult to argue against. With one false “fact”, we destroyed our entire reasoning system.

I’m sure many of us can relate this back to software. Try to recall a time when you received a bug complaint from a user, and after tracking down the bug you stared in awe at your computer screen. “That shouldn’t even be possible…”

The real problem? These “glitches” in our reasoning systems are inevitable. We’re all humans, we will make a mistake. One way or another, you will eventually have a bug in a test, or assume behaviour exists that really doesn’t.

How do we Address Test Fallibility?

This brings me back around to the original topic – how do we avoid this fallibility?

We automate our test suite.

It seems strange to say automation will “cure” all of our problems reasoning about systems, and that’s because it won’t. It’s simply the best way to combat the problem.

Automation addresses Test Absence by:

being cheaper (time) than manual testing

eliminating any reasons to skip tests

growing as a regression suite over time as we think of new test cases

If you forget a test, simply add it and it will never be forgotten again.

Automation addresses Test Fallacies by:

being repeatable

being correctable (you can’t prevent humans from re-making mistakes)

being easy to read, examine, and reason about

If you make a mistake in a test, simply fix it and the mistake will not reoccur.

In summary, automating your tests doesn’t guarantee that your tests will be good, but it allows you to confidently improve your tests over time. Good test allow you to reason correctly about your system.