What is Multivariate Testing?

Multivariate testing is, in a sense, a more complex form of testing than A/B testing, where A/B testing is fairly straight forward.

Multivariate testing is a technique for testing a hypothesis in which multiple variables are modified. The goal of multivariate testing is to determine which combination of variations performs the best out of all of the possible combinations.

Websites are made of combinations of changeable elements. A multivariate test will change multiple elements, like changing a picture and headline at the same time. Three variations of the image and two variations of the headline are combined to create six versions of the content, which are tested concurrently to find the winning variation.

What are the Benefits of Multivariate Testing?

Using multivariate testing can be helpful when multiple elements on the same page can be changed in tandem to improve a single conversion goal: sign ups, clicks, form completions, or shares. If conducted properly, a multivariate test can eliminate the need to run several sequential A/B tests on the same page with the same goal. Instead, the tests are run concurrently with a greater number of variations in a shorter period of time.

What are the disadvantages of Multivariate Testing?

The most difficult challenge in executing multivariate tests is the amount of visitor traffic required to reach meaningful results. Because of the fully factorial nature of these tests, the number of variations in a test can add up quickly. The result of a many-variation test is that the allocated traffic to each variation is lower. In A/B testing, traffic for an experiment is split in half, with 50 percent of traffic visiting each variation. In a multivariate test, traffic will be split into quarters, sixths, eighths, or even smaller segments, with variations receiving a much smaller portion of traffic than in a simple A/B test.

What is A/B Testing?

A/B testing is the act of running a simultaneous experiment between two or more pages to see which performs or converts the best. Despite the name (A/B testing), the experiment can be conducted with as many pages as desired.

Once you have decided what to test on your landing page (e.g. headline, call to action, photography, adding a video etc.) – you can create new versions (or variants) of your page to enter into the experiment.

What are the Benefits of A/B Testing?

The process of A/B testing leads to improved website content because you must discuss which content is valuable and why. As you generate the variables to be tested, you create a list of potential content improvements. A/B testing can also cause organizational shifts that increase profits. Organizations using A/B testing value data over opinions. Resources are spent on implementation, not debate. These companies are nimble–they quickly adapt to changes in the marketplace and changes in data. Decisions are made quickly and mid-course adjustments are expected. A/B testing reduces risks inherent in big decisions like pricing and strategy. Testing customer behavior before making decisions increases the likelihood of success. A/B testing helps companies target their resources for maximum results, increasing the ROI, whether the return on investment is defined in short-term success or long-term customer loyalty.

What are the Disadvantages of A/B Testing?

A/B testing requires sacrificing some conversions as part of the experiment.

A/B testing measures only quantity, not quality. A/B testing measures conversions based on the occurrence of events, a binary value. Just because one version leads to more sign-ups or even sales does not mean that the other version has less value.

Things change. Just because version A performed better over version B one year ago does not mean that it will still perform better now. You may see different outcomes when conducting the same test twice during different times, which can be frustrating to anyone.

For beginners, keeping things as simple as possible is undoubtedly the way to go. Develop a solid A/B testing plan (with well reasoned hypotheses) and focus on a number of experiments quickly and frequently.

Once you feel that you’ve exhausted your A/B testing, then you should consider multivariate testing provided that you have enough traffic to be able run such experiments.