Quick Refresher: A/B Testing

A/B testing emails is a common approach for determining which variations in content, subject lines, images, etc., drive more engagement (i.e., higher open and click rates) by recipients. Engagement is a key objective for a variety of emails, but most A/B testing tools aren’t well-suited to notifications and transactional use cases.

Good news, though! SparkPost has a new A/B Testing feature that’s been optimized for transactional email and other app-generated messages and notifications. My colleague Irina recently shared the details in her preview of the feature.

Now, we’ve released A/B Testing, and we’d like you to try it! So don’t be surprised when you see a new option next time you log into SparkPost.

I’ve put together a quick tutorial on how to get started with A/B testing emails in SparkPost in three steps.

A/B Testing in 3… 2… 1.

Let’s get started with SparkPost’s A/B Testing feature. In a nutshell, our implementation is built around the core product features you already use: templates and transmissions. Here’s how it works.

Step 1. Create an A/B Test

First things first: let’s create a new test.

Once you’ve logged into SparkPost, click “A/B Testing” in the left navbar.

On the A/B Testing page, you’ll see a list of existing A/B tests, if any. Otherwise, you’ll see a blank slate. Let’s change that!

Let’s create a new A/B Test. Click on the button on the top right labeled “Create a New A/B Test”.

Give your test a name. You’ll note that we automatically suggest a new test ID for you as well, but you’re welcome to override that with any unique ID text you wish. Be sure to note the ID—you’ll need it when you issue a transmission API call later.

We’ll leave the subaccount field alone for this tutorial.

Select the template you’d like to use as the test’s default. You’ll see this field includes a nice autocomplete dropdown that suggests stored template names you’ve already created, so finding a default template couldn’t be easier.

Click the “Create New Test” button to save this new test and go to the next step.

Step 2. Refine and Schedule Your A/B Test

Here’s where we get into the fun part: picking the preferences and variants of your test to determine what drives higher engagement!

Start by selecting the date range over which you’d like to run the A/B test.

As you scroll down, you’re given more control and options at the tip of your hands, including how you want to determine a winner.

At the bottom of the screen, you’ll be able to select the variant templates you want to use to compare against your default template.

Once you’ve made all your selections, scroll back up to the top and click on the “Finalize and Schedule Test” button in the top right corner.

Step 3. Call Your A/B Test During Email Transmissions

Now that you’ve created and scheduled your new A/B test, all that’s left to do is to call it when you inject messages during email transmissions by including
ab_test_id rather than
template_id in the content of the JSON object you submit to the Transmissions API.

Here’s an example of a JSON object you’d post to the Transmissions API endpoint:

Full A/B Testing Feature Documentation

Feedback Wanted!

SparkPost’s A/B Testing feature makes it easy to compare two or more templates to see which one performs best. But today’s just the start—and we need your help.

I’m the product manager for SparkPost’s A/B Testing feature, and I’d love to hear from you about what you think of it. In fact, I’m going to make it really easy for you to help us shape the future of this feature:

Afterwards, if you complete an A/B test and then give me feedback on how it went (hearing about how your engagement improved and which template won would be fantastic too!), I’ll give you another $50. That’s $100 up for grabs.

If free money isn’t your thing (hey, no judgement), but you still want to voice your opinion about A/B tests, tweet us or share your thoughts in our Community Slack.

We use cookies to optimize your experience, analyze traffic, and personalize content. To learn more, please visit our Cookie Policy. By using our site without disabling cookies, you consent to our use of them.