Although many marketers have a good grasp of the basics, they still frequently make one mistake: ending the test without a sufficient sample size.

There are three reasons for this:

Lack of knowledge – they don’t truly understand how to calculate the right sample size;

Negligence – they’re in a rush to do something else, so they rush the test;

Over-reliance on tools – most split testing tools offer some sort of confidence metric, but they can be misleading.

The first and third can be fixed.

If you’ve ever used a tool such as Unbounce to conduct split tests, you know the analytics show you something like this:

The table gives you the current conversion rate of each page you’re testing and a confidence level that the winning one is indeed the best.

The standard advice is to cut the test off once you hit 90-95% significance, which is fine advice.

The problem is that a lot (not all) of these tools will give you these significance values before they even mean anything.

You’ll think you have a winner, but if you let the test run on, you might find that the opposite is true.

Keep in mind that conversion experts like Peep Laja aim for at least 350 conversions per page in most cases (unless there’s a huge difference in conversion rates).

You must understand sample size: The fix for this problem, and most sample size problems, is to understand how to calculate a valid sample size on your own.

It’s not very hard if you use the right tools. Let me show you a few you can use.

The first is the test significance calculator. It’s very simple to use: just input your base conversion rate (of your original page) as well as your desired confidence level (90-95%).

The tool comes up with a chart that has a ton of scenarios.

You can see that the bigger the gap between the “A” page and the “B” page, the smaller the sample size needs to be. That’s why it’s best to test things that could potentially make big differences—they speed it up too.

Here’s another sample size calculator you can use. Again, you put in your baseline conversion rate, but this time, you decide on the minimum detectable effect.

The minimum detectable effect here is relative to your baseline, so start by multiplying them together to get 1%.

What that means is that you will have 90% confidence that you’ve detected a conversion rate on your second page that is under 19% or over 21% (plus or minus 1% from the 20% of your baseline conversion rate).

That also means that if your split test results show a 20.5% conversion rate for your second page, you cannot confidently say that it’s better.

Use either of these calculators to get an idea of what sample size you need for your tests. More is always better.

3. Running a split test during the holidays

This is related to segmenting, but it’s an often-overlooked special case.

Your traffic during holidays can be very different from your typical traffic. Including even one day of that abnormal traffic could result in optimizing your site for the people who use your site only a few times a year.

You also have to consider other special days that influence the type of traffic driven to your site:

Sales

Features in press

Big events in your industry

On top of that, these special days aren’t usually one-day things. For example, when it comes to Christmas, those abnormal types of visitors may visit your website leading up to the big day and a week or two after.

The solution? Go longer: The best solution is to exclude these days from your test because they will contain skewed data.

If it’s not possible, the next best solution is to extend your split tests. Go over your minimum sample size so that you have enough data to drown out any skewed data.

It’s why your website’s traffic varies from day to day and even from month to month.

For some businesses, buyers are ready to go at the start of the week. For others, they largely wait until the end of the week so that they can get started on the weekend.

It’s not valid to say that buyers who buy at one part of the cycle are the same as buyers at another part. Instead, you need an overall representation of your customers, through all parts of the cycle.

Your first step here is to determine what your business cycle is. The most common lengths are 1 week and 1 month.

To determine it, look at where your sales typically peak. The distance between your peaks is one cycle.

Next, run your split tests until you (1) reach the minimum sample size and (2)complete an integer of your business cycle, e.g., one, two, or three full business cycles—but never 1.5.

That’s the best way to ensure that you have a representative sample.

Conclusion

If you’ve started split testing pages on your website or plan to in the future, that’s great. You can get big improvements, leading to incredible growth in your profit.

But if you’re going to do split testing without the help of an expert, you need to be extra careful not to make mistakes.

I’ve shown you 6 common ways that people mess up their split tests, but there are many more.

Any single one of these can invalidate your results, which may lead you to mistakenly declaring the wrong page as the winner.

You’ll end up wasting your time and even hurting your business sometimes. Even if you get a good return from split testing, it might not be as much as it could be.

For now, keep learning about split testing, and make sure you completely understand the 6 mistakes I showed you here. If you have any questions about them, leave them below in a comment.