Implement Or Discard?

Our Practical Takeaways For Running A/B Tests

1. Testing Time Is Precious - It's Ok To Stop Tests Early

Perhaps the biggest suggestion I might make from playing this game (and running a/b tests) is to take an agile approach to testing. That is, some people might estimate their testing duration based on some effect prediction and set-forget for a couple of weeks. The problem with this is that the effect prediction is just a prediction and often the real effect is different - way different. In such a case time is wasted if the test is bound to continue no matter what (if the variation is losing badly or has an effect larger than predicted). Hence we advise to look at your results often and stop tests when they begin to show strength.

2. Making The Right Decision Is More Difficult Earlier On

It might become obvious quite quickly that there are three factors at play which make the discarding-vs-implementation decision more difficult. These include:

The lower the baseline conversion rate, the more difficult the decision

The fewer the number of visitors, the more difficult the decision

The smaller the relative effect, the more difficult the decision

The more time we give a test to run, with a greater sample size, the effect ranges of A and B tighten, making the right decision (implementing or discarding) easier.