The problem with small traffic sites is that running split tests on them seem futile.

Let’s say, you have an existing conversion rate of 5% and the expected percentage increase in conversions is 10%. If you’re testing 4 variations, here’s how long you will have to wait to get conclusive results (95% confidence level), depending on your website traffic:

Who would wait for months or sometimes even years to see conclusive test results (95% statistical confidence)? It doesn’t make any sense.

I feel you. Your argument is right. It isn’t that easy.

But to think that there’s no way out of this situation is wrong. And this is exactly what this article aims to explain — that small-traffic sites can also split test to improve their conversion rates. Here’s how:

1. Track Micro-Conversions, Instead of Macro-Conversions

Product sale or paid signups are the ultimate goals, yes. But these macro-conversions are relatively harder to come by.

When you have traffic crunch, instead of tracking test data by macro-conversions, declare test results with micro-conversions that indirectly contribute to sales. As long as you know the conversion ratio from micro-conversions to the ultimate goal, you can calculate the effect on macro-conversion and know which test version is working for you.

2. Test Something High-Impact

May be you want to test a new pricing plan? Or replace that image slider with a great discount offer to your most popular product category? Make sure all the prominent website estate — such as site’s header, above-the-fold area and left-side of the page — is occupied by important page elements that eventually contribute to your page goal. Think about something that can give you a new insight and take you in a new direction for future tests.

Sometimes small changes have a big impact on conversion rates. The key for such results is testing something that has high-impact on the customers’ psyche. Something that can influence their decision.

Understand their concerns. Know what primary factors they consider before taking an action on your site. For a funky clothing site which targets teenage and college students, pricing and free shipping can be very important. For a luxury-clothing brand that focuses on high-end celebrities, 1-day shipping guarantee or exclusive collection section on the site might be high-impact.

Know what matters the most to your ideal customer. When you test something high-impact, the drastic difference in results reduce the time taken to declare statistically significant results.

3. Test Radical Designs

When you ask your customers to choose between a red apple and a green apple, it will be difficult to say what the majority will prefer. You will need a huge sample size to answer that. Ask them for a choice between apple or orange — and even a small sample size of audience will soon tell you what most people prefer.

The table below from TechCrunch portrays this perfectly. Drastic changes often impact the conversion rate in a big way as well. Notice that as the difference between the previous and the new conversion rate increases, the requirement for test visitors and test duration to run the test decreases:

Thus, you must do away with button color tests and the likes if you have little traffic. Drastic changes make you reach conclusive results quickly.

But the drawback to these tests is that it is difficult to get any learnings from them. You cannot know what change increased or decreased the conversions. Was it shifting the call-to-action below the fold that worked? Or is it the change in headline? Or anything else?

One great way to overcome this drawback is testing two different themes in these page-level tests rather than randomly changing multiple page elements. Getting significant customer learnings from such tests is easy and can be very useful for future tests.

Know that you’ve hit the jackpot when you get a learning that you can apply to optimize your overall marketing efforts, including your offline campaigns. When a rehabilitation facility chain tested their control page that focused on luxurious features and secluded location of their centers against the trust-focused approach on their challenger page, they got a 220% lift.

This way even when you’re changing multiple elements, you will still understand more about customer motivations which will further help you in your CRO efforts. Test your page themes when testing a radical design. You will get results sooner and will also understand your customers better.

4. Forget Multivariate Testing (MVT)

The more variables you change, the longer it will take to achieve statistical confidence. Limited traffic means you don’t have enough resources for multiple test variations. Stick to A/B split tests until you have sufficient traffic.

Traffic split between multiple variations in MVT

5. Run Site-Wide Tests

Site-wide tests give you the benefit of cumulative traffic and take only a few minutes to implement. If you are a Visual Website Optimizer customer, you can simply use this URL pattern to run a site-wide test: http://www.yoursitename.com/categoryname/*

But for this to work, URL patterns of your site must be sorted. The asterisk (*) above denotes that the test will be run on all pages of the given category.

6. Try Sequential Testing

Run one version first. Then the next version for the same time period. Compare the results in your Google Analytics from the two versions. Since you will be sending double the traffic to each website version, you will have more data than you’d have had if you followed the traditional split testing way.

This is not as scientific or accurate as split testing. But this can still work if the user motivation remains the same in both the cases. Be careful about external factors that might skew the results such as holiday season, any economic event, and so on.

Run tests on exactly the same days of the week. If the first version was tested from Monday to Saturday, do the same for next version too. Make sure that there is no change in traffic sources, social media campaigns or any PR activity during this time.

The more closely identical your environments are (while testing both the versions), the more accurate your results will be. Of course, results of sequential testing can many times be misleading because of the validity threat it faces due to the history effect — that is the effect of an extraneous factor on a test variable as the time passes. Still it is better than not testing at all.

7. Run Qualitative Tests

Make a genuine move to improve your website by knowing your customers. You can go about this in many ways:

Usability Testing: Seeing how users interact with your website, the unique website path they follow, time taken by them to complete the conversion goal, where they get stuck or frustrated is revealed all too clearly by this method.

Traffic has nothing to do with this. Watching a few individual sessions will give you so many good ideas about improving visitors experience on the site.Read this article to know how you should do usability testing to get accurate insights. For remote usability testing of your website, you can use usertesting.com.

Conduct Customer Surveys: You can either mail a small questionnaire to your current customers, or you can conduct on-site surveys. A SaaS company might ask customers on their pricing page if their pricing is clear, an eCommerce company might e-mail their customers asking if they have praised or criticized their site to a friend in the past few months and what it was about?

The insights you get from customer surveys is fascinating. Just be sure that you do not annoy them by sending surveys too often. For on-site surveys, stick to 1-2 questions per session. See how Kellogg Business School implements this on their website:

Analyze Heatmaps: Even with small traffic, heatmaps can give a pretty good idea about visitors’ interest. If there is any obscure element that is attracting more attention than it should, if a call-to-action or an important navigation element is getting ignored, until where are people scrolling on a page, you can see it all.

Fixing any flaws in placement of elements, visual hierarchy and other such concerns become a lot easier with heatmaps. Want to generate a heatmap for your web page? Signup for the free trial of our A/B testing software with built-in heatmaps.

Improve Site Load Time: Whatever your traffic volume might be, every one second of delay in the page load time reduces conversions by 7%.

According to another statistics shared by KISSmetrics, 40% of visitors abandon a website that takes longer than three seconds to load. And you might already know, page load time is a ranking factor too. Do yourself a favor and fix this money leak by following these tips.

Drive Some Targeted Traffic to Your Site for Testing

Before you start a test, I’d recommend that you calculate the number of days for which you will need to run the test. You can use this calculator for a quick workaround. If the duration seem too long, consider running a campaign or two to push some paid traffic. Or take advantage of your subscriber list to divert e-mail traffic.

Facebook, Adwords and getting the word out about your site/product on popular industry blogs are some quick ways to increase the traffic inflow on your site for testing. But if you decide to go this route, don’t forget to segment your result data on the basis of traffic sources to see which traffic converted the best for you.

(9) Comments

Thanks Smriti for an article that so many people need. Everyone wishes they were Amazon, but many of us have small customers who only have a few thousand visits per month. One other suggestion I’d add is that you can test multiple changes at once (akin to the “dramatic changes” suggestion) that will have a bigger impact on conversions. The drawback is that you won’t learn as much–you won’t understand which changes were positive & which were negative, but it can allow a low-traffic site to reach a conclusion on the test.

Very good article. Does VWO use the bandit methodology? I personally think that it is a great way of generating value while testing, especially on low traffic/conversions.

I think that the learning is not so important when it comes to low traffic, I usually focus on generating value by constantly testing and tweaking the tests as the time goes in order to increase CR, even if I don’t reach statistcial significance.

I also focus on the peak conversion days of the week to make sure my potential winner is outperforming the control on those days and then declare it as a winner.

@Lucas – That’s sequential testing. You will have to be very careful about monitoring external factors and your traffic sources to avoid any noise in the data.

@Alex – To declare a winner, number of conversions and traffic both play an important role. If your conversions are significantly high, you will get a statistically significant winning variation even with relatively small traffic.

By micro-conversions, I mean things like, ‘add to cart,’ ‘visits to the pricing page,’ ‘download brochure,’ etc. That is, something that shows the intent to buy. Since these conversions need less commitment from consumers and are easier to come by, it can help you find a winner quickly than tracking macro-conversions, like product purchase, paid signups, and so on.

As long as you know the conversion ratio of micro to macro conversions, you can calculate which version is working for you without having to wait too long.

I might be misunderstanding but I think the logic of point 6 is unsound.

Specifically:

” Since you will be sending double the traffic to each website version, you will have more data than you’d have had if you followed the traditional split testing way.”

I don’t get this.

How is running a test with 1 version getting 100% of the traffic for a week and then another version getting 100% of the traffic for the next week “more data” compared running an A/B test at 50-50 for two weeks?

Re point one “testing micro conversions”:
can we really rely on the fact that the drop off rates in the next steps of the funnel starting from a micro conversion , will remain unchanged regardless of the testing variant(s)?