Welcome

Welcome to Josh Baker's Practical Advice for Optimizing Your Internet Marketing blog. Here you will find internet marketing optimization and online strategy articles full of tips, tricks, discussions, and thoughts to help you take your marketing and business to the next level of success.

How to Interpret the Estimated Conversion Rate Range in Google Website Optimizer

Interpreting the Estimated Conversion Rate Range Properly in GWO is of Key Importance!

It’s easy to get initially excited when you see that one of your test panels in Google Website Optimizer has a higher Estimated Conversion Rate than that of your control panel as presented on the Combination Report page. This may even lead to you believe you can end the test (prematurely I may add).

Unfortunately, just looking at the Conversion Rate number given to you by Google in the bold type font isn’t enough, you most certainly need to do a little bit of visualization to really have a better understanding of what is going on and how they are performing against each other.

Whether you are running an A/B test or a multivariate test, this is important to know for either – the number they give you is a conversion RATE RANGE. Many people mistakenly look at just the number given and do not visualize the full conversion range given along with it (done so with simple addition and subtraction of the number given next to the estimated conversion rate after the plus and minus sign). This range is based on the observed conversion rate of during the experiment thus far. Not factoring this in can lead to many people ending or wanting to end tests before they are truly ready to be ended. For example,

Estimated Conversion Rate

Test Panel – 6.0% +/- 1.0%

Control Panel – 5.5% +/- 1.0%

Reading and interpreting this correctly would actually tell you that the:

Test Panel is converting in the range of 5.0% to 7.0%, and the

Control Panel is converting in the range of 4.5% to 6.5%

This being true, their conversion rate ranges are overlapping each other. Visualizing this information shows you the overlap much more clearly as shown below:

So, is the test panel really converting better than the control? Potentially the control panel could still be converting better – it’s not clear enough to make a decisive or correct decision at this point.

Now think of a scenario such as two panels having a higher conversion rate and much larger conversion rate range:

Test – 55.0% +/- 8.0% (conversion rate range of 47.0% to 63.0%)

Control – 52.0% +/- 7.0% (conversion rate range of 45.0% to 59.0%)

Some may feel that the test panel is the winner and so it’s ok to stop the test and roll out with the test panel…BUT…

As the test goes on for a longer period of time, you should find the conversion rate range getting “tighter” and the above test potentially ending in such as a way as:

Control panel converting at 58% +/- 1 .0%, and the

Test panel converting at 47.0% +/- 1.0%.

In this ending scenario the control panel is actually the winner not the test panel as we saw mistakenly earlier on. Ending the test too early certainly changes the story drastically as does the results could differ as well!

I know sometimes it’s exciting to see your test panel winning, but patience is certainly of high importance in A/B and multivariate testing in order to make sure your tests and results are always moving forward correctly.

1 Comments

Patience is indeed obligatory !
More interesting question is what confidence accorded to the improvement seen (with a confidence level very high) over time?
Three months later, uplift will it be the same?