The 3 most common A/B testing mistakes beginners make

Having done over 200 A/B and MVT tests I have gained testing experience the hard way. I have had my fair amount of success and failures. Working with companies that are just exploring the A/B testing possibilities I have noticed reoccurring mistakes that is based on inexperience.

If you are a newbie in this area I would recommend you read this post. If you are experienced I would suggest you read about the top 3 most converting headings. If you have any questions do not hesitate to ask.

1. Low traffic

The absolute most common factor I have encountered is that the traffic to the test is way too low. It is important to have the right amount of traffic to be able to draw accurate conclusions. With accurate I mean statistically correct conclusions. You need statistical confidence otherwise your test is inconclusive. In other words it means that the change can be due to chance. For example Google does AB and MVT tests on 5% of their searches, but since they have so many searches they reach statistical confidence within a very short time (probably one day).

How do you determine that you have the right amount of traffic? I suggest you look into your web analytics package and on page level. Check how many visits you have a day and then calculate the amount of conversions. A rule of a thumb is to have around 300 conversions on each recipe you try before you conclude the test. If your conversion rate is 10% then you need 3000 visits to have the right amount of traffic.

2. To many recipes

AB testing an idea is always the way to go instead of having the HIPPO (Highest Paid Persons Opinion) dictate what the users want, which is a sure path for disaster. The second most common testing mistake is to have too many testing ideas that you try at once. This means that your traffic will be split into many recipes forcing you to run the test for a very long time. A recommendation is to test more than two recipes only when you have more than a 600 visitors to your testing page. Review your ideas by looking at user insight, my next point.

3. No clear idea or possibly the wrong idea

Testing an idea is always good but how do you know if you are testing the correct idea or if you are just wasting time and resources? Well this is where your user insight comes into place. You need to know what your users want to be able to compile the correct test. If you don’t you will make many tests that will be more or less useless. For example a well known poker grand tested 6 different versions at once. Three of them were very similar and the testing results were inconclusive.

User insight is derived by many different tools and methods. User studies such as, surveys, focus groups, user testing, eye tracking, keyword analysis, surfing patterns, interviews etc shall all be used to gain the knowledge of what your users want. This method is a never ending story and distinguishing the winners from the losers. I am a strong believer of doing continually user studies, just like Google does.

Of course you might also be very lucky hitting that idea that will give you a fantastic increase.

So now that you know the most common A/B testing mistakes you are ready to go. You can also read the more hands on post on how you start an A/B test.