Subject Line Test: 125% more unique clickthroughs

The results are in. A few weeks ago, we asked the MarketingExperiments blog community and the Copyblogger community for help on a subject line test. Today we’ll look at the process we went through, the results, and what you can learn from it all.

Nathan Thompson, Senior Manager, Research and Strategy, MECLABS Conversion Group, was the lead judge for the entries on our blog, and Sonia Simone, CMO, Copyblogger Media, LLC, led her team’s effort (Sonia wrote about their process today on Copyblogger).

“Some were clever, some were text-book marketing, and others were, well … weird,” Nathan said.

“Our process for cutting down the list involved removing those that were factually incorrect first as we didn’t want to incite a riot among our readers once they opened the email and realized we had tricked them. Then we removed those that were too long, too confusing or unclear.”

Once the list was narrowed down to the acceptable subject lines, Nathan and the MECLABS Conversion Group team began grouping them into categories based on what the subject lines were attempting to convey:

Incentive

Curiosity

Value exchange

Newsworthiness

Etc.

“Then we voted on the subject lines we felt were the best representation of each of these categories,” Nathan said.

The 6 Treatments

The team’s testing strategy ended up focusing on how the incentive held up compared to a more value-oriented subject line without the incentive. Here are the subject lines they selected, along with the categories they were testing for with each subject line:

The primary research question was, “Which motivation represented in the subject line will result in the highest clickthrough (Delivered/Clicks) rate?”

The KPI was unique clickthrough rate of overall emails delivered. We didn’t choose open rate as the KPI, because we wanted to avoid subject lines that were click bait but didn’t tie into the motivations of customers who could benefit from attending Optimization Summit 2012. Click bait can increase unsubscribes and decrease the result we really care about: unique clickthroughs. People don’t like to be tricked.

The Results

Drumroll, please …

And the winner is — “Do your landing pages pass this test?” written by Christine Parizo.

As you can see in the below screenshot from the MECLABS Test Protocol, Christine’s subject line generated a 125.9% higher unique clickthrough rate than the worst-performing subject line of the test.

Let’s put the magnitude of this into perspective. By simply changing a few words in one line, Christine helped drive more than twice as many visitors to our landing page. Talk about return on investment. Christine, I look forward to high-fiving you in person in Denver at Optimization Summit.

But wait, let’s take a look at the results from a different angle …

Click to enlarge

While Christine’s subject line was significantly different than most of the other subject lines, there was one subject line which it was not significantly different from at a 95% Level of Confidence – “Learn 3 tips that made 10,000 landing pages extremely successful.”

That subject line was written by PaulM. We had only planned on giving one prize, but since there was no significant difference between the top two subject lines, we decided both copywriters should be rewarded. So Paul also wins a free ticket to Optimization Summit 2012 and a seat in our on-demand LPO course.

Plus … all six copywriters whose subject lines were selected to be tested will receive a 90-day free membership to Third Tribe Marketing, Copyblogger’s private community of online businesspeople.

“It should also be noted that the two top performing subject lines in clickthrough were also the top performing in terms of open rate,” Nathan said.

What can you learn from this test?

Of course, here’s the most important part of this blog post — how you can use this information to improve your own marketing efforts?

At a high level, the whole reason we ran this test was to show you just how powerful testing and optimization can be. As I said, the winning subject line drove more than twice as many visitors to the landing page. This isn’t some massive marketing automation or e-commerce cart redesign effort, it is simply changing a few words.

Second, test don’t guess. Some of the tops minds in marketing — like Sonia Simone and Nathan Thompson — were involved in this test, and while I didn’t put anyone on the spot, in working with them on setting up this test, I can tell you there was no clear winner in our minds.

Also, be open. Turn to those outside your normal cadre when crafting tests. That is the beauty of testing: you can always have one or two safe ones, plus one that really challenges the model. And when that outlier wins, it can open your eyes to a whole new approach.

For us, we had help from the savvy marketing professionals who read the MarketingExperiments blog and Copyblogger. While you might not have that access, you can still ask your customers for ideas through your content marketing, social media channels or surveys.

Also, you can ask an expanded peer group. Reach outside of the marketing department and ask for opinions from Sales, Customer Service, Product Development, IT, and even front-line, customer-facing employees. Beyond your company, ask peers in other companies, your husband or wife, the people you play basketball or squash with … anyone who can expand your horizons.

As for interpreting the results of this test itself …

The two top performers both used the specific phrase “landing pages.” The people that are interested in landing pages are interested in Optimization Summit because they understand the power of landing page optimization.

Keywords are important for more than just SEO. Google has trained us to conjure up short phrases when we want information. People think in keywords now, any time they’re looking for info, not just when they’re staring at a blank search box. When your recipients scan the subject lines in their inbox, what keywords can you provide to tap into their interests and motivation?

“The results, interesting as always, tend to indicate that incentive was not as big a player as we had guessed it would be,” Nathan said. “In fact, the two top performing subject lines didn’t mention the discount incentive at all. And even the subject line that paired the incentive with a bit of newsworthiness (Speaker’s List) was unable to outperform ‘Learn 3 tips that made 10,000 landing pages extremely successful.’”

Why wasn’t incentive effective in the subject lines? You need to sell the value of a product before anyone will find that incentive valuable. “$300 off” is not valuable, especially in an inbox cluttered with other incentive-laden subject lines. “$300 off” something your audience wants and values, now that is worthy of a clickthrough. I learned that we seriously underestimated how well our audience understands the value of Optimization Summit 2012 … or even knows of it at all.

This is a common problem many marketers make. We’ve been sleeping, eating and dreaming Optimization Summit as we have been putting it together, so it’s all too easy to overestimate how much our audience has learned about this event.

“This is a solidly specific headline – the readers know that the message will be about landing pages,” Sonia said, in analyzing the winning subject line. “Passing (or failing) tests is a common anxiety, and in a competitive economy, no one wants to have the landing pages that fail the test.”

“The word this is important (in fact, there was another entry for ‘Do your landing pages pass the test?’). That slight added element of specificity enhances reader curiosity to generate those clickthroughs.”

I’m a little surprised about “Do your landing pages pass this test?” as it seems a bit disconnected with the email content. Someone who opens the email probably expect to find a test they can apply to their landing pages to judge the quality of these pages, but that’s why we test right? My feeling about the theme of this headline is that it’s less about fear and more about curiosity. It conjures the question of what is this test I can use to judge the quality of my pages. It’s a common question but pretty hard to answer in most landing page professional’s mind so the promise of a simple test would seem enticing. That’s why I was concerned about the potential disconnect once the email is opened.

DanielB, would it be possible to share the email open rates for each of the subject line as well? Very curious to see how the consideration rate (opens/clicks) plays out for each headline! For example, does “Do your landing pages pass this test?” have a much higher open rate and maybe slightly lower click thru rate (lower consideration rate)?

Again, congrats guys, I love surprises in testing! That’s when we get to learn something new!

And excellent point about the possible disconnect of the promise. Friday’s MarketingExperiments blog post is actually about a test where the winning subject line (in terms of unique open rate) did not have the highest unique clickthrough rate, due to that disconnect once the email is opened.

These unique open rates are listed in the same order as the first chart…

Firstly I would like to say BIG thank you to all the guys and girls at Marketing Experiments, your research has helped us to optimize our emails and landing pages in a way we never thought possible.

Regarding this subject line test, I was running a similar test on email (Prospects database, members who registered but still have not ordered)

A – Top Spring Offers – up to 60% off best sellers – sent to 623k
B – Top Spring Offers – create something special with your photos 623k

Everyone in the marketing and trading department thought it was a no brainer, subject line A will win by miles. However,…

- The A test subject only got a 2% increase in open-rates
- The A test got a -5% in clicks to open and – 2% on overall click-rate (clicks to emails sent)
- The A test had a 17% increase in opt-out click-rate (opt-out clicks to opens)

Backed into the numbers and calculated the consideration rate and the performance of “…pass this test?” turned out to be more amazing than I thought. It had a 24% lift on open rate AND an 84% lift on consideration (click/open) rate. Wow, I really wouldn’t have thought that…cool!

Thanks Daniel, great to see my subject line made the Top 2 and readers responding to good old fashioned “specificity”.

As we encourage our clients @stage4solutions.com; be specific, promote/sell the value [not the feature (in this case, Optimization summit)] and provide proof point to validate claim…which holds true across all content that buyers engage with.

Interesting. Customers bahaviour and interests differ a lot today when most people get tons of email, advertisement etcetera every day.
I thought things like distribution time/date, trustworthiness and realtions with the company sending the information would have a bigger impact.
I will definitely start asking friends/ coworkers and other people what they think before sending out my next marketing email and newsletters.
BR

I couldn’t help but notice that the actual email itself seems to break a few (ok… a LOT) of the tips discussed in one of the ME seminars on crafting an engaging email message.

No headline. Takes more than 30 seconds to read. Looks like a web page. Multiple (6) calls to action. Different calls to action. No value exchange in the calls for action. No message/benefit in the video screenshot. And very busy in general as far as sight flow.