Jason Cohen recently authored a post on A/B testing that deserves both broader awareness and a deeper dive. Most of us in the online marketing world are aware of the power A/B tests can bring through improved click-through, sign-up and conversion rates. Getting a higher percentage of visitors to a page to take a desired action is powerful stuff.

The process by which we hypothesize, design, create and run testing, however, is fraught with peril. And, one of the least obvious, but most insiduous potential pitfalls is actually what we choose to test.

Visualizing the "Local Minimum" Issue

It's definitely interesting and sometimes worthwhile to test individual elements of a landing page, but it's often not appropriate at the beginning of a landing page or product's life. As Conversion Rate Experts points out, the "let's throw stuff at a wall and see what sticks" approach can have a small impact. Researching the questions visitors have and answering them effectively can make a world of difference.

The problem is, it's so very tempting to be seduced by an easy answer.

The Tantalizing Tease of Testing Minutiae

It's likely that many of you have read case studies like the ones below:

In all of these, some simple change accounted for big increases in click-through or conversion rate, leading to widespread praise and sharing. The problem is - they're the exception, not the rule. In fact, that's precisely why they're newsworthy and get so many mentions. That's not to say you shouldn't read them or shouldn't take away value from the examples (you definitely should). It's just that the mentality of the small change can create a misleading mindset for marketers.

Very few websites have the experience of changing a button color or altering a headline or fiddling with some copy and seeing huge improvements in conversion rate. If you have good reason to believe you're an outlier, go for it, just be cautious - it's not just the fact that small scale changes can have less positive of an impact. They also cost time and resources that you can't afford.

Some Simple, Compelling Math to Keep You Out of the Weeds

Let's say you're pretty good at conversion rate optimization - A/B and multivariate tests are relatively easy for you to perform and you've got solid instincts around them. And let's also say that you get reasonably decent traffic to your landing/test pages - in the several thousand range each day.

Even under these ideal conditions, massive problems emerge.

Knowing that each test takes a substantial amount of time to get high confidence of accuracy and that smaller tests (with less needle-moving potential) take MORE time is a pretty convincing reason to start out with the big ideas and big changes first. But, it's not the only logic behind this. Let's say you find a page/concept you're relatively happy with and start testing the little things - optimizing around the local minimum. You might run tests for 4-6 months, eek out a 5% improvement in your overall conversion rate and feel pretty good.

Until...

You run another big, new idea in a test and improve further. Now you know you've been wasting your time optimizing and perfecting a page whose overall concept isn't as good as the new, rough, unoptimized page you've just tested for the first time.

It's easy to see how you can get lost in this process and frustrated, too. That's why my recommendation (and the advice I get from lots of talented CRO folks) is to start with the big ideas and big projects, nail down the grand plans worth testing, let your audience pick a winner and then try to tweak, tune and improve.

What You Should Be Testing

What do I mean when I say "big ideas" and "overhauls?" Luckily, 37Signals provided a terrific example yesterday with their Basecamp Homepage Redesign:

They recorded a 14% improvement from new vs. old and can now decide whether they want to try another innovative concept or start optimizing the little things on this version. And while the numbers don't sound as compelling as a few of the bigger ones from the small tests, I'd argue they're going about things exactly in the right way. Perhaps a "little change" to the old version would have improved things quite substantially, but with this new version, they've got a higher base conversion rate and can benefit from every change that much more.

Another great example is the case study Conversion Rate Experts did for SEOmoz itself. That test gave us a 52% improvement in conversion rate from the PRO landing page. As an addendum, in April of this year, we tested an in-house created, shorter, less story-like landing page that we all hoped would beat out the old long-form version. After a few weeks of testing, it lost out. Later this summer, we'll be trying something completely different in an attempt to beat our current best.

The process to follow for conversion rate optimization and testing was well described in Stephen Pavlovich's post - The Definitive How-to Guide for CRO. His PDF guide, in particular, made this dead easy:

Step 1 of 4 in the process

Follow those steps, don't get lost in the minutiae, and you'll be on your way to exceptional results - no thousand monkeys with typewriters required.

Comments
30

Any posts about CRO is very welcome... and this one is somehow a "best CRO advices repository" because of all the links you provide (I ask you to feel 'guilty' for my loss of productivity in the next few hours).

A post to be bookmarked surely.

About if it's better prioritize small scale vs. huge A/B tests, I believe it is to be considered as when a shop makes changes to its products expositions. With the premise that is always good to maintain a constant look & feeling in order to not "shock" your returning visitor, it would be good to refresh the image & visual architecture of a web site every amount of months (12/18) in order to better the CRO and prioritize the big changes.

I'm curious to read the comment by our CRO mozzers experts (Dr. Pete, this is especially for you).

I get this all the time... Let's test the button color or the size of the text or the font type... To be honest I believe that people who take the approach like this think of the visitors as some little monkeys that react on different colors or sizes.

People who visit our website are human beings, therefore complex. Yes, the change of the color of a button might change on the way that a page communicates, but the truth is that how and what it communicates has the impact on the users, not the button colors.

Therefore, I totally agree with your approach to focus on big ideas. From the way people react to tests versions you also get to learn so much about them.

As well as keeping out of the weeds, I guess there's a certain 'grass is always greener' mentality. Of course, it's natural to want to improve your conversion rates but when there are often just minimal improvements available, I wonder if it can counter-productive.

For instance, if you spend a week testing a completely new landing page design and get a 4% reduction in your conversion rates, then you try a new design again and get a 6% increase, presumably you're going to be tempted to go third time lucky and try yet another design. And another and another...

At some point, your regular visitors are going to get frustrated and confused with all the changes to your site. Every time they hit your front page they see something different. Eventually they don't bother to come back to your site. So maybe your conversion rate is improving but your actual visitor volume is dropping. Just a thought.

CRO is one of my favorite topics right now. That's one of the reasons we started doing web design and development - to get more control of the marketing and conversions. The best producer of ROI for an SEO campaign is increased leads and sales, and it is hard to get (what people consider) good SEO results without increasing ROI. SEO and conversion rates are getting increasingly connected and it's hard to offer one without the other.

Thanks for the visuals on where to expend energy. I love the basecamp example, and I have read similar stories in Website Magazine. This reminds me of the post (http://www.seomoz.org/blog/design-trends-the-single-purpose-homepage) about the single focus home pages. There is definately a trend in that direction, and basecamp is another example of knowing exactly what action you want people to take, and creating the entire page around that single focus.

I think there is a lot of branding that a company has to do before a single focus homepage will be effective, because if you don't have brand recognition, you have to use the valuable home page real estate to quickly and effectively educate your visitors as to who you are and what you do. That's slightly off the subject, but I thought it was worth mentioning :)

I think it depends. I tend to like less is more too. But, if people aren't converting because they want MORE information, then maybe less is less. Though the Basecamp change is more aesthetically appealing (to me) it also appears to better illustrate their value. I think your point is well deserved though, if clutter hides your value, then clean it up.

Ya, I totally agree with the premise of this article. When I'm reporting on A/B testing case studies for WhichTestWon.com, I'm often drawn in by the hunt for those instances where a small change made a significant (10% or more) impact on conversions ONLY because our mission is to evangelize testing... so if we can show people a small change made a big imapact, mission accomplished. But, you're right in that most of the time it's the big overhaul that gets you the highest increase in conversions. I've seen it time and again.

Logic is the driving force behind changes. It's good to see the concept of a broad change that can have a greater overall impact. Then, once the big change is made, little tweaks can be done. Thanks...

Hm, I have to agree with "furstconversion" there - the cost/benefit calculation is what one should be looking at and immediately "statistical power" and "significance level" determination comes to mind. Not mentioned even once above.

Furthermore, the stats backing the graphs are dubious and uninformative, if not misleading. "Reaching statistical significance" is NOT a valid way to determine when to stop a test (statistically significant confidence makes no sense statistically, so I presume this is what you mean).

Having such a stopping rule exposes you to errors much greater than the ones your achieved significance level tells you. Furthermore, you speak nowhere of the effect size confidence intervals which is what would be the data one should look at when determining the success of a test. I bet those would look quite horribly if you stopped your tests the first time you reached significance (specifiying your level would have been nice).

Totally agree with you if you're using some manual testing tool (like GWO) where you have to keep manually changing code and updating your website by hand (that time could be put to much better use).

But with automated tools out there (such as the Conversion Chicken) that run and improve the site by themselves and don't require any maintenance there is little point in not testing, as they don't require any time investment and may lead to much better results.

Of course its great to do massive sweeping design changes (like 37 signals), but after that big design change is done why not test all the little things to squeeze out as many sales as possible?