Why an A/B Test Winner May Damage Your Profits (and how to avoid it)

5 Steps for Building a CRO Strategy

[Infographic] How to Use Conversion Optimization to Battle Shopping Cart Abandonment

We have brought thought leaders, influencers, visionaries and veterans to our tribe. Now it’s your turn. If you have something worthwhile to share with a large community of savvy testers, go ahead and pitch your post idea. We’re listening.

Why an A/B Test Winner May Damage Your Profits (and how to avoid it)

Not long ago I attended a Digital Analytics Association Symposium in San Francisco where Krista Seiden, Analytics Advocate from Google, delivered a talk on Building a Culture of Optimization. She told a story about how she once rushed to her boss’s office to declare a winner from an A/B test – and you would probably expect me to be surprised when she then revealed that it ultimately turned out to be a total flop due to inaccurate data. But honestly, I wasn’t.

I have seen so many conversion rate optimization efforts on ecommerce websites, which fail to deliver good results due to data inaccuracies, that it has stopped shocking me. Do you want to be sure that you are not going to be tricked by the next A/B test results that you analyze? Then I have got a bunch of tips for you.

Do the Homework — Ensure Data Quality

Are you 100% sure that your digital analytics tools are collecting accurate enough data for you to act on? Do you completely believe in the figures that your A/B testing software is showing you? Well, I don’t. Unless I validate them.

At Mavenec we always follow 3 steps when we are building an A/B testing ecosystem:

Typically, anything between 10-20% of transaction data gets lost due to users opting out of tracking, javascript errors and so on. I was horrified when (for the first time) I had a client whose Google Analytics reported only 60% of the transactions. How could anyone act on such data?

If you want to avoid this kind of mistake (and you certainly do) compare the figures between your financial records and Google Analytics (or Webtrends, AT Internet, Adobe Analytics or any other analytics tool that you use) in order to be sure that you are acting on reliable figures.

I just love using Google Analytics data to analyze segments after the test is conducted. Sadly, hardly ever am I able to find there all of the data that I need to evaluate a test.

When analyzing A/B tests on e-commerce websites I would usually like to know some additional metrics that are missing – with the average margin on every challenger as a great example. Another is the return rate, which may vary between challengers. Typically, you won’t find those in digital analytics tools.

Following these 3 steps will ensure that you have the right data in place and that you can rely on it. But the right data is only half of the success. What else could go wrong you may ask? Well, you may have the right data in place, but your analysis could mislead you if you rely too much on one metric of success.

Do not Focus Solely on the Conversion Rate

Way too many A/B tests on e-commerce websites focus only on the conversion rate. The conversion rate is just a part of the story. By generating more transactions, it’s possible you could still generate lower revenue, if for example the average order value goes down. So, what other metrics should you take into account when assessing your next A/B test?

1. Average order value

The first metric – aside from the conversion rate – that you should take a look at is the average order value. If you manage to lift the conversion rate but impede the average order value you may end up with more work to be done in order to ship the orders but with the same level of revenue.

Not all of the products an online merchant sells are equal and certainly, not every one of them produces the same money. Never, ever should you forget about the profit margin.

Checking this metric is extremely useful if you conduct tests on free delivery. There have been numerous examples where slashing delivery costs to zero skyrocketed the conversion rate and boosted the average order value. But this strategy also affects the profit margin.

Analyze this additional metric cautiously, to be sure that this strategy earns its keep (hopefully if you have followed one of the steps from the previous list and integrated your test data with your financial systems you can calculate it without much more hassle).

3. Return rate

Pushing users to complete transactions with psychological tricks may sometimes backfire. These users may not be satisfied with their purchases and you will then have to cope with a jump in the number of returned orders. This will create additional costs and hamper your profits. Always check if the return rate hasn’t gone up.

Before determining your next big winner in an A/B test on an e-commerce website, take note of the two steps mentioned above. Firstly, make sure that you are collecting the right data that you can rely on. Secondly, pair the conversion rate with additional metrics. Forgetting about these steps may lead you into proclaiming a winner, which boosts your conversion rate and sales, but leaves you with less money in your pocket.

Do you want to come up with great testing hypotheses for your next A/B tests? I bet you do! If I bet right, you should get our Ultimate 115-point Ecommerce Optimization Checklist which we use with our clients to dramatically increase their conversion rates and sales.

We have brought thought leaders, influencers, visionaries and veterans to our tribe. Now it’s your turn. If you have something worthwhile to share with a large community of savvy testers, go ahead and pitch your post idea. We’re listening.