How long does a test take?

It is impossible to know ahead of time how long it will take to get a test’s results. This depends on:

The site’s volume,

The percentage of traffic you decide to assign to the test,

The test itself (the changes and impact on your conversion rate),

The objective you are tracking.

Generally speaking, we strongly recommend you wait until at least 5000 unique visitors are tested per variation, and a hundred or so conversions. Also, we recommend that you let a test run for at least seven days in order to take into account the different behaviors between work days and weekends.

There is a statistic reliability index in AB Tasty (Khi-2 index) which lets you know whether your test is statistically reliable. We recommend that you attain 95% statistical reliability before making any firm and definitive decisions.

We also strongly recommend that you leave a test active for at least the amount of time of your purchasing cycle. This corresponds to the time required for a user to go through all the steps of the purchasing process. This time may be estimated at several days for classic shopping sites (browsing, verification, purchasing), but it may take several weeks on less traditional sites (ex: B2B activities, large purchases…).

Not every test will give results, and you may sometimes have to stop some of them due to low statistical reliability, because the tested hypotheses simply have no impact on your conversion rate. The elements that indicate a low reliability rate and little chance to see a clear trend emerge are:

Reliability rate below 90%,

Reliability reached very quickly (ex: 1 or 2 days),

A very small difference between the original’s conversion rate and that of the variation,

Results that are chaotic time-wise (the average conversion rate curves cross each other regularly from the start of the test).