Testing versus Checking

I stumble upon this blog post Testing vs. Checking by Michael Bolton which I found very interesting and a fundamental for software testers. I tried to summarize it and place it in a table to see the distinction clearly.

Checking

Testing

confirming existing beliefs

finding new information

a process of confirmation, verification, and validation

a process of exploration, discovery, investigation, and learning

focused on making sure that the program doesn’t fail

focused on “learning sufficiently everything that matters about how the program works and about how it might not work.” – James Bach

We are checking when:
* we’ve made a change to the code and we want to make sure that everything that worked before still works. When we have an assumption that’s important, we check to make sure the assumption holds.
* developers write and modify their code, creating automated routines that they run frequently to check to make sure that the code hasn’t broken

We are testing when:
* we configure, operate, and observe a product with the intention of evaluating it, or with the intention of recognizing a problem that we hadn’t anticipated
* when we’re trying to find out about the extents and limitations of the product and its design, and when we’re largely driven by questions that haven’t been answered or even asked before

A check provides a binary result—true or false, yes or no

A test has an open-ended result

is all about asking and answering the question, “Does this assertion pass or fail?”

is about asking and answering the question “Is there a problem here?”

checking can prove the presence of bugs, but not their absence

the process of finding out whether our checks have been good enough

machine-decidable and are, in and of themselves, value-neutral

requires human, which makes it a sapient process

when done by a programmer, is mostly a quality assurance practice. The quality assurance angle: a programmer helps to assure the quality of his work by checking it.

is not quality assurance, but acts in service to it; we supply information to programmers and managers who have the authority to make decisions about the project.

Checkers

Testers

A person who needs a clear, complete, up-to-date, unambiguous specification to proceed

Does not require the certainty of a perfect specification to make useful observations and inferences about the product

A person who needs a test script to proceed

His job is to discover information; often that information is in terms of inconsistencies between what people think and what’s true in reality

A person who does nothing but to compare a program against some reference

A tester’s task might be to reveal problems that occur when our excellent code calls buggy code in someone else’s library, for which we don’t have a specification

I also liked how James Bach describes testing, “questioning a product in order to evaluate it.” Evaluation of a product doesn’t assure its quality, but it can inform decisions that will have an impact on quality.

Though people may or may not agree with these, it made me understand where to draw the line. =) Thanks Michael!