If They Say This About Taguchi Testing Then Run Don’t Walk Away

OK, that title got your attention! What I am talking about? Well, some people promoting Taguchi methods of testing suggest that you can eliminate under performing test runs during the test. Here is a quote from the guy that runs StomperNet’s test software:

Lets say you are running a L18 — this has 18 different landing page recipies. While the test is running you will naturally see a couple of recipies that are doing significantly worse than the others. As this happens you can shut off these recipies. You can keep doing this until you are down to two recipies — i.e. - a split test which will confirm the results from the teguchi test.

Hey it’s a quote so I did not correct the spelling

Recall from my Why Use Taguchi Methods post on orthogonal arrays and how they are used in the Taguchi Methods. In order to have an orthogonal array and maintain statistical independence, you cannot eliminate test runs because that leads to an unbalanced array.

Here is the L8 orthogonal array:

Experiment

Columns

Number

1

2

3

4

5

6

7

1

1

1

1

1

1

1

1

2

1

1

1

2

2

2

2

3

1

2

2

1

1

2

2

4

1

2

2

2

2

1

1

5

2

1

2

1

2

1

2

6

2

1

2

2

1

2

1

7

2

2

1

1

2

2

1

8

2

2

1

2

1

1

2

Since this array allows for 7 factors each having two levels, the total number of combinations is 128. Obviously 120 of those are being left out in the L8 array. A few of those would be:

Columns

1

2

3

4

5

6

7

1

1

1

1

1

1

2

1

1

1

2

2

2

1

2

2

2

1

1

2

2

2

2

2

2

2

1

1

2

1

2

1

2

1

1

2

1

2

1

1

2

1

2

2

1

2

2

2

1

2

2

1

2

2

1

2

However, since the factor columns in an orthogonal array are statistically independent and each factor can be evaluated in isolation, all combinations can be accounted for in the final design. Eliminating rows in the array during the testing process makes this impossible and results in completely ignoring the other 120 combinations.

This is something left over from full factorial test, such as that used by Google Website Optimizer, and is a completely valid thing to do in that design since every combination of test factors and levels are used. In the Taguchi Method the array allows us to represent the full test space with a limited number of test runs that are carefully chosen. They are chosen in a way that makes each factor independent of the others. Elimination of a test run breaks that and confounds all the results.

So as the title says, if they tell you to do this then run because they do not understand Taguchi Methods or test design in general. Using their approach will cost you.