This one A/B testing case study will leave you smarter and wiser

The Company

PriceCharting.com specializes in offering current and historic prices of new and rare video games and consoles such as Atari 2600 and Super Nintendo. Basically in exchange for a small fee, it ensures you don’t overpay when buying a video game.

Background

On its product pages, Pricecharting has a call-to-action (CTA) that says ‘Download’. The accompanying text tells you the fee you need to pay to get the price list. For example, the product page for ‘Sega Genesis’ console says ‘Sega Genesis Price List for $2’. This is how it looks.

The test

JJ Hendricks, the owner of PriceCharting.com, felt that the existing CTA button looked too much like an advertisement.

“The ‘Download’ button and the price made buttons for our price guide service look too much like banner ads. We felt users probably had ad blindness and didn’t click it.”

Acting on this insight, Hendricks quickly set up a small A/B test using Visual Website Optimizer. He changed the CTA button text from ‘Download’ to ‘Price Guide’ and also chopped the price part.

This is how the Variation looked.

A/B test results

Just a week into the test, the variation emerged as the clear winner with the almost perfect 99.9% statistical significance. Changing the CTA text from ‘Download’ to ‘Price Guide’ increased the clickthroughs by a jaw-dropping 620.9%!

What worked?

1) Greater relevance

Once you hit the ‘Price Guide’ button, you are taken to a page where you can customize your own video game price guide. According to Hendricks, ‘Download’ was not a very compelling CTA for his website users as they were looking for the Price Guide.

“When people view this (product) page, they are looking for the price guide for a particular type of video game.”

So you see? Just by making the text more relevant to his users, Hendricks reaped in a 620.9% increase in clickthroughs! ‘Price Guide’ was a better message and conversation match for the users as they were looking for the prices of the old video games. See how a fitness chain similarly increased its conversions by making its CTA text more relevant in point number 4 here.

2) More specific

‘Price Guide’ has a very specific meaning for the video game enthusiasts here. As opposed to that, ‘Download’ is a little vague and confusing. One doesn’t really know what will be downloaded after they click on the button. On the other hand, ‘Price Guide’ gives a clear, well-defined direction. It focuses on the goal of the page and fastens up the decision-making process of the potential customers.

The first version starts with ‘Download’, but visitors are likely to think “Download what?” The second message starts with what is on offer and then finishes by telling visitors to click on the button.

Hendrick’s hunch had been proven right and he didn’t waste any time in implementing the changes on his website. But that’s not the end of the story. Like a true brilliant marketer, he didn’t stop testing just there. His curiosity had just started taking shape.

The second test

He ran a second test to figure out how price sensitive were his customers. It was a preliminary test to see how the customers behaved before he started a series of tests on the actual prices. So he cloned the CTA test and tracked the same goal (clickthroughs). On the previous variation (which was also the winner), he added a ‘Starts at $4’ next to the ‘Price Guide’ CTA. This is how it looked.

He set this one as the Control and tested two more variations against it. One with starting price of $2 and another with $1.

*This was an internal test and the new prices were not reflected on the checkout page.

The results of the second test

Though there wasn’t a clear winner, the $4 version did much better than the $2 and $1 variations with 91% statistical significance. However, the conversion rate of the $4 version was still less than the conversion rate of the winner in the previous test (0.70% compared to 1.03%). Hendricks did not add the ‘Starts at $4’ text on his product pages, but he got an invaluable insight that will hugely help him while deciding on his future prices.

“We will probably try some higher priced products on the landing page in the future to see if it will increase revenue.”

The fact that $4 value did better than the $2 and $1 versions means the visitors are agreeable to paying a higher price. Hence, Hendricks can safely test higher prices.

Why it worked

If one goes by the results of the preliminary test, it seems users value the products of PriceCharting.com higher than what was anticipated by Hendricks or they are indifferent to the price. In either case, it’s a win-win for Hendricks who has the required impetus to test the prices now.

One of our clients, Six Packs Abs Exercises, was also pleasantly surprised when it A/B tested its prices. An increase in price didn’t change the demand, bringing them more revenue.

Your key takeaways

1) Take it to the next level

You have got brilliant results in a test? Awesome. Do a little self-congratulatory gig, soak in the happiness and implement the changes on your website. And then live happily ever-after?

Start with the next test right away. Don’t stop. If we constantly try to make optimum utilization of our time and read all those self-help stuff to become more efficient, why not constantly try to optimize our websites?

2) Run a dummy test before testing high-impact elements

Pricing directly impacts your revenue. You are reluctant of A/B testing the price as you are scared it might affect your revenue. No worries. Run a dummy test or an internal test as Hendricks likes to call it. But do this only if you can carry it off.

A lover of the written word, I plan to be the planet's first sit-down comedian. When I am not rethinking a misplaced comma, I write about conversion optimization and website usability. You can follow me @mohitanagpal

(6) Comments

This CTA looks like a pattern where a secondary CTA “Download Sega Genesis” is placed next to a primary CTA “Download”. Along with the trend that has people removing link underlines, I completely misinterpreted what the CTA refers to.

I now see (correct me if I’m wrong) that the whole thing is meant to be one sentence with the button style merely used to highlight the word “Download”. But the button boundary signals a linguistic boundary (unlike underlining a link). CTA should not be a fragment like this.

Example: On this page, there is a primary CTA all contained within the button: “Sign up for FREE trial / No credit card required”. This is good.

Awesome post I must say and the way it has been presented by quiz. Yeah, by exploring such test we can keep CTA which will give greater conversion rate. It is very important to have such different designs, experiment, measure and then finally go with the CTA in a page which gives more clicks and leads to more conversions. Keep up the good work.

It was easy to guess the right answer. In addition to being vague, ‘download’ inevitably has some negative connotations around viruses. Even if you want the price guide, the danger surrounding downloading has to flicker across your mind for a split second, even if it is subconscious. ‘Price guide’ on the other hand skips right over that and directly conveys the value, lowering the possibility of that little flinch.

@Vlad Firstly, my huge apologies for replying so late. Secondly, yes the whole thing is meant to be a single sentence and I agree with you it is slightly confusing. May be they should run another test, this time treating the CTA button and the text as two different elements. What do you think?

@Stephen Many apologies for replying so late. So glad you found this useful. Changing CTA text is in fact one of the easiest tests to set up and can have huge impact on the clickthroughs.