What Can Be Tested?

A/B testing is great because you can test nearly everything, and Umbraco makes it easy to make changes (more on that soon). The caveat is that not all changes are created equal. Some require more work, time, or money to complete, depending on the structure of your site and Umbraco itself. Sometimes it helps you and other times it stands in the way. With Umbraco, you can think of changes in 3 tiers:

#1) Simple content changes

Relatively easy to do, content changes are just tweaks to the content on your page managed through Umbraco. This category includes:

Copy

Images

Headlines

Banners

Links

Social proof (reviews, client logos, etc)

Pricing

Promotions

Delivery options

#2) Simple template changes

Changes to the template itself shouldn’t be a problem for a developer, but they are a step up from the first due to the fact that you actually need some outside help and they take more time. Examples are:

Layout

Colour

Button placement

Above/below the fold placement

Personalization (made easier with plugins as described in Endzone’sawesome article)

Anything not content managed in the CMS

#3) Large development tasks

Larger development tasks are usually changes that span several pages or actually affect the inner workings of the site itself. They generally require a lot of time and investment up front. Some might be:

Different processes (checkout, onboarding)

UX issues that require changes to the fundamental aspects of a site (E.G, changing the navigation menu to make it easier to use)

Restructuring your e-shop

Changes to highly interactive pages

As you can see, there’s pretty much no limit to what you can test; in fact, one of hurdles that most companies struggle to overcome is figuring out just what exactly to test in the first place. You can’t just jump right into things, though.

Randomly testing changes on a hunch will get you nowhere. You’ll spend a lot of time and money on making changes that could easily have the opposite effect that you had hoped for. Good A/B testing requires forming hypotheses, stating goals, and analyzing the results after enough traffic has come through the doors. We’ll cover how to choose what to test in more detail later.

Multivariate Testing

Sometimes, the complexity of your website may call for running more than one split test at a time. Whenever multiple elements of a page can be modified to affect an experience with a site, a multivariate test can be run. The upside is that they have the ability to pinpoint the most effective changes to a page by running multiple A/B tests concurrently and in a much shorter time. The downside is that it is difficult to really glean anything with certainty without a heavy dose of visitor traffic. The very nature of multivariate testing means that it is a lot more complex. It’s important to have deep experience with it and know exactly what you’re setting out to achieve before initiating one.

During a multivariate test, different variations of the original page are shown to the visitor, this time with multiple differentiations from the original. A multivariate test might show a different heading and a different call to action at the same time. For example, if you wanted to test 2 headlines (A and B) and 2 calls to action (C and D), then the visitor will see 1 out of 4 possibilities (A + C; A + D; B + C; B + D)

An example of multivariate testing

Imagine that you’re a political candidate running for office, and your website is designed to get more people to donate to your cause. Your visitors click-through and land smack dab on a page showing a picture of you in a Hawaiian shirt with a button that says “PAY ME”. Since your donations are embarrassingly low, you’ve decided to run a multivariate test.

First, two groups of changes are made: the picture and the button (an alternative of you with your family and a new button that says “DONATE”)

Having the ability to test multiple calls to action, colours, placements, images, etc, all in different combinations with each other gives companies unprecedented evidenced-based insight into possible changes to make that will increase the effectiveness of their websites.