A Beginner’s Guide To A/B Testing: An Introduction

A/B testing is a fantastic method for figuring out the best online promotional and marketing strategies for your business. It can be used to test everything from website copy to sales emails to search ads. And the advantages A/B testing provide are enough to offset the additional time it takes.

Well-planned A/B testing can make a huge difference in the effectiveness of your marketing efforts. Narrowing down the most effective elements of a promotion, and then combining them, can obviously make your marketing efforts much more profitable and successful. Here is the first in a series of posts covering A/B testing that we’ll be publishing in the coming weeks.

How Do You Plan an A/B Test?

The first thing to do when planning an A/B test is to figure out what you want to test. Are you running an on-site test, or an off-site test? If you’re running an on-site test, you’ll want to think of all the sales-related pieces of your website, and then figure out which elements you want to test.

With off-site tests, you’re probably testing either an ad, or a sales email. Testing ad copy to see which ad brings in more converting visitors can help you focus your advertising efforts. Once you know your ad is converting as well as possible, it’s easier to justify spending more money on it. The same goes for emails. If you send out two versions to your list (randomly selecting which half gets which email), and then track which one converts better, you can send only that version the next time.

Once you know what you’ll test, make a list of all the variables you’ll test. For example, if you’ve decided to test your call to action, you might test:

the location of the call to action

the exact text used

the button color or surrounding space

It’s a process, and it’s common for multiple A/B tests to be carried out prior to making a final decision or final change.

Make sure that before you start testing you have a clear idea of the results you’re looking for. You should already know your baseline result, which is the results you’re currently getting. You want to test option A and B against each other, but you also want to know that whichever one does better in the test is also doing better than your current results. Alternatively, you can use A as your control (leaving it whatever you’re currently using) and then use something new for B.

Tests need to be run simultaneously to account for any variations in timing. You can’t test one variation today and the other one tomorrow, because you can’t factor in any variables that might have changed between today and tomorrow. Instead, you need to split the traffic seeing your variations at the same time.

A/B Testing Can Improve Your Bottom Line

Accurate A/B tests can make a huge difference to your bottom line. By using controlled tests and gathering empirical data, you can figure out exactly which marketing strategies work best for your company and your product. When you figure that one variation might work two, three, or even four times better than another, the idea that you would conduct promotions without testing starts to seem a bit ludicrous.

A/B testing, done consistently, can improve your bottom line substantially. If you know what works and what doesn’t, and have evidence to back it up, it’s easier to make decisions, and you can often craft more effective marketing materials from the outset. Just remember to keep testing regularly, since the effectiveness of anything can change over time.

Things to Test

You can test virtually anything in your marketing materials: headlines, calls to action, body copy, images, etc. If you can change it, you can test it. But that doesn’t mean you should necessarily spend months testing every little thing. Instead, focus on the things that are most likely to have a big impact.

On your website, this likely includes:

the headline

your call to action

any graphic you use in direct correlation to your sales efforts

the sales copy or product descriptions

In an email, it probably includes the same. In an ad, especially a text ad (like a search ad), you have fewer things to change, and so likely you’ll test either the main headline or the offer itself.

Testing different offers is also important. Just make sure that you have methods in place to ensure that each person is always offered the same promotion. For example, if a free gift is offered to group A, and a discount is offered to group B, then you want to make sure that group A always contains the same visitors, as does group B.

You can also test things in conjunction with each other. For example, you might want to test newsletter A with landing page A, and newsletter B with landing page B. And then, later, you may want to test newsletter A with landing page B, and vice versa. This can give you a more concrete result if you’re getting mixed results, or if your results are very close.

Devoting Time to Testing

A/B testing is not an overnight project. Depending on the amount of traffic you get, you might want to run tests for anywhere from a few days to a couple of weeks. And you’ll only want to run one test at a time for the most accurate results.

Giving a test insufficient time can mean skewed results, as you don’t get a large enough group of visitors to be statistically accurate. Running a test for too long can also give skewed results, though, since there are more variables you can’t control over a longer period. Make sure that you stay abreast of anything that might effect your test results, so that you can account for any statistic anomalies when reviewing your results. If you’re in doubt, it’s perfectly reasonable to retest.

Considering the impact A/B testing can have on your bottom line, though, it’s worth taking a few weeks to properly conduct tests. Test one variable at a time, and give each test sufficient time to run.

Can I Test More Than One Thing At a Time?

There are two approaches to this question. First of all, let’s say you just want to test your headline, but you have three possible variations. In that case, running a single test and splitting your visitors (or recipients in the case of an email) into three groups instead of two is reasonable, and would likely still be considered an A/B test. This is more efficient than running three separate tests (A vs. B, B vs. C, and A vs. C). You may want to give your test an extra couple of days to run, so that you still have enough results to base any conclusions on.

Testing more than one thing at a time, such as headline and call to action, is a multi-variate test, and is more complicated to run. There are plenty of resources out there for multi-variate testing, but we won’t be covering that when talking about A/B testing.

Be sure to check out our previous article on website testing: A Beginner’s Guide to Google Website Optimizer. Also, we’ll be covering a number of other A/B testing topics as part of this series, including website copy, email campaigns, pay-per-click advertising, and SEO landing pages.

On a similar note if you are using WordPress and want to use Google Websiteoptimizer I’ve made simple plugin that supports both A/B splittest and multivariate. Also implemented linkclickconversion support.
Find download link here:
andreasnurbo.com/3480/testversion-of-new-gwo-4-wordpress
Currently it only covers pages and posts. Got some ideas for making it possible to test all areas of your WP site.

Thanks @ Andreas for the book recommendation “always be testing..”. I have just begun to focus on conversion optimization . It has been a shift from traditional SEO disciplines and I think that conversion optimization is simialr to SEO in that its an art and a science.

Awesome write up. Im in the middle of setting up some split testing with a wordpress install. You’ve given me some very helpful tips here…thnx! I love that we can take the guessing out of what works and doesn’t. Design and layout usually end up being driven by the decision makers perception about what will work and not. This way it will be straight up statistics that can decide.

Yes. there are tools in place to help with split testing. I just wanted to add, that we need to keep in the back of our minds about mobile and video split testing as well, coming from a business owner, internet marketer or SEO perspective. I enjoyed the post! If you curious about a video split screen tool check out easyvideoplayer2.hypenewproducts.com