For those of you who have used pairwise test design methods (or similar combinatorial software testing methods) in an attempt to zero in on a manageable number of unusually powerful tests from among a very large number of potential tests that could be executed...

Have you ever applied these test design methods and achieved poor results? If so, what poor results did you observe? What factors led to the poor results?

Have you thought about applying these test design methods and concluded that a pairwise (or similar) test design approach would not work in a given situation? If so, what factors led to your conclusion that pairwise or other combinatorial test design methods could not be used to achieve desired results?

3 Answers
3

The underlying assumption for those methods is that any two parameters are independent from each other, so any two parameters that do have some kind of dependency will show poor results, i.e. your coverage will not be as good as expected.
When I tried to use pairwise the first step was to find independent enough parameters, I couldn't find any that will also save run time. On top of that I got loud complaints from the developers that using combinations of parameters will make theirs life harder finding and debugging problems.
I test complex embedded systems BTW.

Rsf, Thanks for sharing your experiences. FYI, many pairwise test design tools have constraint handling features that mitigate the first problem (e.g., when some parameters are dependent upon others). Also, about the debugging challenge issue, while a valid point, (a) in my experience, I'd rather have the problem of "How do we debug this reproduce-able defect?" if the alternative would be "Why didn't we detect this defect during testing?", and (b) you can often quickly eliminate the vast majority of possible causes by looking at passed tests that contain similar combinations of test inputs.
–
JustinMay 18 '11 at 19:08

Actually, the underlying heuristic for combinatorial testing of multiple input parameters is that they are interdependent and the specific values assigned affect on a common output condition or state.

Based on the concept of testing various combinations of input variables that affect a common output condition or state, combinatorial testing may not be suited for testing

sequentially ordered inputs

state transition or business flow

customer scenarios

etc

I have also discovered that conbinatorial testing may also miss single mode errors where the order in which the parameter variables are changed trigger a problem, but if the order is different the single mode failure may not be detected.

Upvoting this comment. For those who are interested, Dr. Atif Memon at the University of Maryland and his collaborators have published some papers about the application of combinatorial testing to user interfaces. They talk about the issue the commenter described in the third paragraph.
–
user246May 27 '11 at 12:35

@user246 Why don't you just give us link to the papers? And BTW, what comment you are talking about? There is no other than yours at time I am reading it...
–
Alois MahdalMay 17 '12 at 8:57

The short answer is situations when a bug can occur only when three or more states must be set a certain way.

The long answer ...

Pairwise testing good technique that actually applies some pretty sound logic. That said, it is fairly easy to understand what the potential tradeoffs are .. let me illustrate using an example I have used previously.

Let’s assume that we are building an application that we use determine the cost of a flight.

We can choose from Sydney, Melbourne or Brisbane as destinations. We can choose from Business, Economy or Budget Economy as our type of ticket. Finally, we can choose Virgin Blue, Qantas or Jetstar as our airline.

Let’s assume that there is a bug which occurs when Business and Jetstar are selected together, because Jetstar don’t have any business class seats. With all 27 possible combinations, we would get three test failures, as shown below:

So we found our bug in this case, so we didn't loose anything. So what is the trade-off with using pairs?

Well if a bug only occurs when three or more things occur, such as a Qantas business class flight to Melbourne, then you are not guaranteed of finding it. You may get lucky if your test case just happens to match, but you are not guaranteed to find it, that is the tradeoff with pairwise.

You will get this kind of tradeoff with any testing approach, not just pairwise. My example shows what happens with 2 variables, when moving into combinations with more than just pairs, you can increase the coverage.

There are also some tool limits. AllParis (which I used) can only do pairs, whilst other tools will have less limitations.