I write 1-2 times a month about Product Marketing and Growth. Sign up now to be notified of new posts!

A/B Testing is Expensive

Over the last few years the startup community has really gotten behind A/B testing and hyped it up quite a bit. There is a more nuanced point about the downsides of A/B testing that needs to be understood: A/B tests are very very expensive for most startups at the time when they matter most, early in their formation.

It’s easy to run A/B tests to do full funnel optimization when you are getting hundreds of thousands of visitors a month. Prior to product-market fit you are not likely in that situation. You are probably working with very limited traffic and need to see very large gains to differentiate between tests in a statistically significant way over short – less than a few weeks – timeframes.

No Small Tests

The less traffic you have the longer tests take to run. Additionally, the further down in your funnel you go the more time tests will take to run due to the proportionally lower traffic with funnel drop-off.

You can A/B test the color of that button when you have tens of thousands of visitors a month.

If you want to see big gains you need to be more dramatic with your testing. Try two very different landing pages, or even two completely different onboarding flows. Even if you do have a ton of traffic you should think hard about whether you have done enough of this type of testing, otherwise it is easy to end up stuck at a local maximum.

Understand Human Psychology

One of the most underrated skills related to growth is ability to craft product flows with a human psychology perspective. Once you gain a deeper understanding of this you will be able to make more intelligent product decisions on user flows that will almost always lead to higher conversions. I discussed in an earlier post how important it is when working on growth to always focus on the point of highest leverage. Understanding consumer psychology is imperative if you want to have leverage.

At Lookcraft I built an onboarding flow that achieved 25% conversion through a signup flow with 5 distinct steps and 20+ questions, including an entire page survey about the user’s clothing sizes. I accomplished this without running a single A/B test. Although I did rely on some past knowledge of certain types of user flows that I had A/B tested in the past, everything else was mostly intuited from a studied understanding of consumer psychology and decision making.