Articles in this section

A/B Testing | Step-by-Step Guide

October 15, 2019 21:13

Updated

Revinate gives you the ability to utilize A/B tests in your email campaigns so that you can learn more about what works and does not work with your email marketing strategies. For an overview video of what A/B Testing is, use the following link: A/B Testing Overview

Creating your A/B Testing Campaign

When you first create your campaign, you’ll see the button (Step 1) that allows you to enable A/B Testing for that campaign.

Turn it on and you’ll see two options for your A/B Test (Step 2).

You can do a Subject Line Test or a Content Test.

A Subject Line Test is good if you want to find out what kind of subject lines will potentially drive the more opens for your campaign.

A Content Test will give you the ability to test anything in the email body of the campaign to find out what potentially results in the most clicks.

Some variables that you might want to test in a content test are different hero images, the wording of your headline, the color, placement or wording of your Call To Action button, or even the kind of promotional offer you are providing. For a content test, it is highly recommended that you only test one variable so that you can attribute the differences in performance to that variable.

Steps for Testing

In this example, we will demonstrate A/B testing a subject line.

Create your campaign as you normally would and enter your subject line for Version A. For this example, we are going to test a straightforward subject line that identifies the offer inside.

You’ll notice that you can rename this version if you’d like. You could name this version “Standard SL” or you can just leave it as “Version A.”

You also have the ability to copy this subject line to Version B. This is useful if the subject line for Version B is almost identical. In this case, Version B is a completely different subject line so we will not click on it.

When you are done with Version A (including the content of the email), you can click on “EDIT VERSION B”

You’ll now be taken to Version B of your campaign. You’ll notice that the Subject Line field is the only field that is editable. Once you enter your Version B subject line, you can save and click on "Next" to move onto the next stage.

In the preview tab, you’ll be able to toggle back and forth from Version A to Version B to see your different subject lines. You can even send tests for one version or both so that you can examine this in your inbox.

Content Testing

Content Testing follows the same basic steps as Subject Line Testing, but differs in that you have more variables that you can choose from to conduct a test. For this example, we will test the wording of the Call-To-Action (CTA) button. Version A will say “Click Here” and Version B will say “Book Now”.

Once your content is complete, you can then copy the content to Version B so that all you have to do for Version B is to change the variable, which for this example is the wording of the CTA button.

After you have clicked and copied your content to Version B, scroll to the bottom and click on “EDIT VERSION B.”

Once you're in the editor for Version B, make the change to your variable. In this example, we changed “Click Here” to “Book Now.”

Click on Save and then click on Next to move to the preview screen. Here, you’ll see that you can toggle between Version A and B and send tests for one version or the other or both.

Segmentation and Test Distribution

It is important to choose your segment wisely for your A/B Testing targeting. We recommend that you send to a large enough audience (minimum of 1,000 total recipients) so that you can get a large enough sample size and have meaningful results. That being said, you should still follow best practices regarding deliverability and segmenting.

After you select your segment, you’ll see a different page than what you normally see in a standard campaign.

You’ll see a horizontal bar that is split up into 3 sections with a slider separating the first two sections from the third. This is your distribution bar and it represents your targeted audience for this campaign.

**In the default split, 15% get A, 15% get B and the remaining 70% will receive the winning version.

50/50 Testing

This slider bar determines what kind of test you can do. If you slide the bar all the way to the right as shown below, you are doing a straightforward 50/50 split of your audience. In this kind of test, half your targeted recipients will receive Version A of the campaign while the other half will receive Version B. Your campaign will deploy to your entire list at your scheduled time and you can view the resulting statistics later on to find out which version performed better.

25/25/50 Testing

If you chose to split up your audience into 3 sections as shown in the previous screenshot, you will be designating a test period where a certain percentage of your targeted audience will get Version A while an equal amount of your audience gets Version B. After the test period is over, a winner is determined and the rest of the campaign audience will receive the “winning” version.

Determining A/B Testing Winner

You will also need to select the metric you want to use to determine the winner. You can select from Open Rate (this is the metric you should use for Subject Line Testing) or Click Rate which considers all of your links (other than unsubscribe and view in browser links). Click Rate is the metric you should use for Content Testing.

If you select the A/B/Winner option, there are some extra scheduling options to consider.

The first step is to select a time for the deployment of your test portion of your segment. Take into consideration the time of day that this will go out. Be aware that if you send your test portion of your segment during a time where there are very few opens (such as the middle of the night or on the weekend) you will get a very small sample size and your results will have a much higher margin of error.

Testing Length of Time

The next step is to determine the length of time for your test. The clock starts ticking when the test portion is deployed and at the end of the allotted time, a winner is declared. There is a 4 hour minimum and you can toggle between the number of hours or number of days.

The last step is to determine what action to take when the test is over and a winner is determined.

Option 1: Send immediately when test is complete. If choosing this option, take into careful consideration when your test will end and what time the remaining segment will receive their campaign.

Option 2: Select a time and date to deploy the winning version to the remainder of your audience. One recommended choice is to send the remainder the next day at the same time as your initial testing portion.

Option 3: Don’t send anything, email me and I’ll decide the winner. For this choice, no action will be taken and instead, you will receive an email at the end of the test duration to let you know that you can examine your results and pick the version to send to the rest of your audience.