A/B Testing? Multivariate? Which Should I Use? When?

The concept of testing variations of a web page to increase conversion might seem to be a highly technical and obscure activity. Relax: though it is somewhat scientific, it’s by no means rocket science. It’s actually very easy to understand and do.

Scientific Method

Do you remember the story from middle school science class about Isaac Newton and the apple? History has it that Newton, a renowned late 17th Century English mathematician, observed that an apple, falling from a tree, always fell to earth in a straight line; it never curved or went sideways or upward. Newton, who had been developing theories about gravity and motion, questioned if this force might also exist at greater distances … perhaps as far away as the moon.

Newton tested his question (called a hypothesis) and found he could predict the orbital motion of the moon using a mathematical formula he’d developed to explain gravitational force. Other scientists later used his formula to predict the existence of the planet later named Neptune.

Newton’s discovery was the result of the use of the scientific method, a simplified version of which follows:

Observe an action or phenomenon.

Pose a hypothesis.

Test the hypothesis.

Whether marketers realize it or not, A/B testing is based on the scientific method, which enables you to gather knowledge, test your hunches and potentially predict your marketing outcomes.

How It Applies to Your Marketing

In your marketing, the observation is usually something along the lines of, “Our conversions aren’t high enough.”

Your hypothesis would usually be expressed as, “Maybe I can increase conversions if I change on my landing page/sales page.” (Before you can fill in the blank, you need data. This will be covered shortly.)

Then you test your hypothesis and observe again, to see what kinds of results you get. From there, you may form a new hypothesis, test it, etc.

To get a hypothesis about which part of your page to change, review the page analytics. For instance, you might see your page has a high bounce rate. That’s definitely a problem, but that datum alone is not enough with which to form a hypothesis.

Looking at the time on page can give you a clearer picture of how people are interacting with the page: a high bounce rate but low time on page tells you that something near the top of the page is turning people off—perhaps your headline. A high bounce rate and long time on page indicates that the problem is near the bottom of the page—perhaps the call to action.

By reviewing your page analytics, you can get a clear picture of where the problem is and thus have information on which to base your hypothesis.

When analytics don’t provide a clear picture of what’s hurting your conversions, you will need to go directly to your users with a short survey about your page and use their feedback to determine where your page needs improvement.

When you’ve got your hypothesis, there are two basic ways to carry it out: A/B testing and multivariate testing.

A/B Testing

In A/B testing, you’re comparing the results of two versions of a page to see which brings a higher percentage of conversions.

“A” is the current version of the web page, also called your “control.” “B” is a variation of “A” in which you’ve changed only one element. An “element” is defined as a discrete part or section of your page’s copy or design and could be any of the following:

Headline or subject line.

Body copy.

Call to action.

Photos or video.

Colors.

Forms.

That seems like a lot of potential variables, but the review of your analytics (or, alternately, feedback from your users) will help you isolate the general area of the problem so you can make an educated guess about which element to change.

You can vary an element in several ways. For instance, with a headline, you can:

There are three potential outcomes. After changing an element and sending equal amounts of traffic to each page, you will find:

A significant increase in conversion. Great! You hit the bullseye.

No difference in conversion. Good. You probably didn’t change the right element. Make a new hypothesis and test it.

Your “B” version gets a lower response. Okay. You definitely want to stick with “A” for the time being. Review your analytics, make a new hypothesis and test it.

The advantage of A/B testing is speed and simplicity. A single change can be made and tested with as little as 100 visitors. These single changes can create dramatic increases.

Multivariate Testing

Multivariate testing is similar to A/B testing except that rather than changing only a single page element, you change several elements, thus creating multiple versions, which you supply to viewers dynamically. For instance, if you test three different headlines, three different calls to action, and three different photos, a visitor to your page would end up seeing one of 27 different versions.

Because of the larger number of versions in most multivariate testing, you need a much greater amount of traffic to achieve conclusive results.

It’s a fine-tuning step for pages that are already performing well, so you would only use it if you already have large amounts of traffic.

The advantages of multivariate testing are that it’s more comprehensive and it gives you a clearer picture about how the various combinations of elements work together.

Drawbacks of multivariate testing include possible lack of feasibility to make major layout changes to pages and the reduced effectiveness of tracking tools caused by use of dynamic content.

After the Test

If you conduct A/B or multivariate testing for any length of time, you will likely find an optimum page—the one that converts the best (so far).

But the test is not over. Monitor the behavior of your new leads or customers and see if they are indeed the right audience. If you find that they are not, that’s your observation.

It should be followed by the hypothesis that you need to change something in order to attract the correct public, which will likely require an overhaul of your messaging in general.

Keep a Record

What ever method you use, keep a written record of your hypotheses (the page changes you make) and your results (percentage of increase/decrease in conversion … or no change) so that you can go back and see where you succeeded and where you failed.

For each entry in your record, use a descriptive name for the test, such as “Lead page #1, headline #2” or the like, which will still make sense when you look back on it a year from now.

Final Thought

Marketers have done split tests of their marketing material since long before the internet to create new, higher-performing controls. You can maximize your existing list or market and produce remarkable increases in conversion for the comparatively small effort involved in testing.