Product Experimentation Pitfalls, and How To Avoid Them

Amazon, Microsoft and Facebook are all companies that proudly trumpet their ”culture of experimentation” as key to their success. But for those looking to replicate such success, it’s not as simple as waking up one day and deciding that your businesses will implement just such a culture, according to Jon Norohna, Director of Product Management at Optimizely.

In fact, a lot of companies that set out on that path never get there, because they make a lot of easily avoidable mistakes.

The problem starts, simply enough, with the fact that many companies don’t really know what they mean by “experimentation”. He gives a hypothetical example of a person in the organisation who has a bold idea — redesign the application. This putative visionary goes away and spends months coming up with designs, mock-ups, UX enhancements and so on. Then it’s handed off to the engineers to build and code. Nine months later, it’s not ready, but so much time has been spent on it that the decision is made to launch and pray for the best.

This, Norohna says, is not experimentation. This is wandering around blindfolded.

“By experimentation I’m referring to a whole bunch of behaviours. You can think of it as a cycle, or process, where you bring customer insight and validation into every stage of building a new product or flow.”

It starts with a learning phase, he says. Before you can have a culture of experimentation, you have to know how to form and test hypotheses. And before you can do that, you have to know what a hypothesis is.

“Let’s build a new checkout flow,” is an example Norohna offers of something that sounds like a hypothesis, but isn’t. “Users get stuck in the checkout flow because it’s confusing” is getting closer. “If we change the navigation to be more structured on page three, people would convert more” is, according to Norohna, a good hypothesis — it’s specific and can be quantitatively tested.

The next big pitfall Norohna sees concerns metrics — you have to choose the right metric to experiment on. A/B testing is a powerful tool for moving metrics, says Norohna, but “if you choose the wrong thing to optimise for you can make matters worse.”

He illustrates this with an anecdote from his time working on Bing image search at Microsoft. To maximise the number of searches customers performed, Bing returned minimal results, requiring further searches to refine. This moved the metrics, but dissatisfied customers.

Metrics matter

Norohna also advises companies to “try breaking down your metrics into smaller pieces. Ask yourself what is your overall goal? Is it more revenue or more subscriptions? What are the sub-pieces of that? The smaller the metric you’re testing, the easier it is to see the effect of an experiment.”

The flip side of of that of course is the risk of being too timid about experimentation. Drilling down into smaller and more precise metrics is one thing, but if you’re too conservative you can end up with “surface-level thinking,” says Norohna.

“For example, we all look to Amazon as a leader of experimentation. If you ask Jeff Bezos what he means by running experiments, he doesn’t mean changing the colour of a button on a checkout page. He means trying out an idea called Amazon Prime — what if you could actually subscribe to Amazon and get free shipping? That actually all started as an A/B test that one person proposed at Amazon. It was a test of a business model, not just of a cosmetic change.”

Look beyond A/B testing, he says, and look at testing C, D, E and even F. “Going through the process of thinking of more radical variants can actually be a huge creativity driver. It’s easy to think of one change to make. It’s really hard to think of six changes that are all different from each other. But that process of digging deep and thinking of those things can make all the difference.”

In Optimizely’s experience, that sort of boldness can pay off. The win rate, where experiments generate a positive change in the metrics, is “drastically higher in the cases where you test more than just two variations. Going from two to three is powerful and so is three to four and even four to five. Five actually seems to be the sweet spot, where you are forcing yourself to be creative, but you are not putting infinite energy into 20, or 30, or 100 variations.”

Norohna offers one final pitfall companies should avoid: hoarding. Quite often, he says, “we come up with a big, bold experiment. We have a great idea, but we don’t always share that in the rest of our organisation. That’s fine if our team is three people and we are just a startup. But if you are a real enterprise — if you have multiple teams all around the company — you are actually missing most of the opportunity of running this experiment if you are keeping it all just locked away for yourself.”

He offers a few suggestions to overcome the siloing that can happen when teams don’t share their insights efficiently. The first is to build a formal experimentation log, where people throughout the company can suggest ideas. The corollary of that is to have formal experiment reviews, where the results are also kept in a central accessible location where teams across the organisation can get hold of the insights they provide.

Axes of experimentation

And as you build your culture of experimentation, set goals for the number of people in the organisation whose role includes contributing to and benefitting from both of those sources of knowledge.

There are two “axes of experimentation” that Optimizely thinks about, he says.

The first axis is velocity — how many experiments are you running? At some companies, it’s zero. But some companies perform hundreds of experiments a year and think they’ve got a culture of experimentation. At the best companies, says Norohna, this number is higher than 10,000.

The other axis is maturity — who is actually running the experiments? How ambitious are the experiments? How ingrained is experimentation in the process of building and launching new products?

According to Norohna, at one end of the axis are the companies that rely on one visionary to come up with all the ideas. At the other are companies where every employee feels empowered to bring hypotheses to the table and, in Norohna’s words, “where every new idea is an experiment and not just a prayer at the launch”.

About the author

Matthew Powell is a former editor of Australian Macworld, and a senior writer and commentator in the Australian technology sector of 20 years’ standing. Optimizely is a corporate member of the Which-50 Digital Intelligence Unit. Members provide their insights and expertise for the benefit of our readers. Membership fees apply.

Must Reads

Advertisers will lose $42 billion of ad spend globally this year to fraudulent activities committed via online, mobile and in-app advertising. Forty. Two. Billion. The figures are contained in a new study by Juniper Research. The figure is bad, but it’s even worse when you realise that it’s accelerating. Back