One of the definitions you must learn in preparation for your ISEB Foundation or ISTQB Foundation certification exam is the notion of the pesticide paradox. It states that if we ran the same tests over and over again, we would probably find the amount of new defects identified would decrease over time. The concept is related to the law of diminishing returns.

This paradox states that running the same tests over and over again would not show any new defects because all defects found using the initial test cases would eventually all be fixed. However, there are other possible reasons for the diminishing returns as well. One explanation is the fallacy of exhaustive testing. Simply put, except for the simplest of software applications it is simply not possible to identify and test every possible scenario, so running the same tests over and over would result in a large overlap of previously identified scenarios. Another reason is that a software program’s functionality may change over time, which again would invalidate previously-executed test cases.

As you can see, there is a common theme here, and that is that using the same test cases over and over will result in diminishing returns relative to those test cases. Therefore, to avoid the pesticide paradox and other causes of diminishing returns, all tests should be regularly reviewed to ensure that all expected areas of functionality are covered. This could involve adding new scenarios as well as discontinuing ones that are no longer valid. Additionally, new tests can be written to exercise the code in new or different ways to highlight potential defects. Finally, use a mix of formal and informal testing techniques to achieve more holistic testing results.

The bottom line is that test case creation is not a “set it and forget it” activity. No matter how comprehensive your initial set of test cases and scenarios might be, they must evolve over time and thus must be continuously reviewed and improved to optimize long term results.