Experimentation: Why Scale is Key to Sustained Success

Successful innovation is not about getting it right the first time. It’s about getting there first — creating something that is novel and has value.

And getting there first repeatedly demands experimentation at scale.

Stefan Thomke is professor of business administration at Harvard Business School, and a powerful advocate for experimentation as “the engine of innovation”. He also offers valuable insights and practical advice for organisations that want to be able to experiment and innovate.

For the unconvinced he has a simple warning: “If you don’t know how to do this, you are going to be at a major competitive disadvantage. It’s as clear-cut as this: your competitors will simply outrun you. This has already happened.

“Companies should be moving fast at high scale.”

So what does it take to become a successful business experimenter?

Scale and Velocity

Thomke explains the “10x challenge,” noting that the value of experimentation comes from scale, and that a few experiments a month will not move the needle significantly.

Scaling experiments to 10x demands strong hypothesis pipelines and rigorous program management, as well as a mechanism for results to be swiftly absorbed and acted upon.

Achieving scale and velocity requires infrastructure that avoids bottlenecks and delays. Thomke uses the analogy of a highway — at ten per cent utilisation any potholes or breakdowns do not significantly slow other traffic. At 90 per cent highway utilisation any variability makes traffic grind to a halt.

To allow for this adaptability, successful experimenters invest in “strategic slack” in their infrastructure to encourage experimentation velocity.

Culture

Thomke stresses that learning from failure is fundamental to innovating, but he stresses that failures do not equal mistakes — “There is value from failure, there is no value from mistakes.”

He says that organisational cultures prepared to accept failure as the source of new hypotheses and iterations are essential. At the same time, organisations need to pay close attention to any biases, incentives and overweening governance that can slow experimentation.

Quality

According to Thomke, that discipline and rigour are essential to successful experimentation — along with high quality hypotheses that are measurable and well-defined.

A good hypothesis is “opening stores one hour later to reduce operating costs will not lead to a significant drop in sales”. That can be tested with measurable results. A waffly or weak hypothesis such as “We can take our brand upmarket” is almost impossible to test with any rigour.

While many enterprises may focus their experiments in the online world, Thomke notes that bricks and mortar tests can also be run — as long as there are strong controls to inject discipline into the experiment.

Innovation v invention

It’s important to remember that innovation is different to invention. “When we think about innovation it is novelty plus value. That’s very different to invention, which is really about patents and has no value requirement. Invention is important to innovation but it’s not the same.”

Innovation is also broad in its application and can apply to:

Products and services;

Channel strategies;

Processes;

Technologies; and

Business models.

While there are examples of breakthrough and disruptive innovation, most established enterprises rely on incremental innovation — which is often more predictable than disruptive innovation.

For digital businesses, incremental innovation can be immensely powerful, says Thomke; “Even a one to two per cent improvement — multiply that by a billion clicks and the changes accumulate.”

But getting to that one to two per cent improvement can be challenging. “When it comes to innovation we are much more likely to be wrong than right — we are lucky if we get 10–20 per cent of things right.”

It’s why enterprises need cultures comfortable with failure as a precursor to success. They need to embrace the “designated deviant” in a team who has licence to challenge the status quo and tools to run experiments.

Why experiments? Because innovation is about endurance. It’s about managing the difficulties of planning for change and transformation, gauging customer behaviour and reaction to innovation, and navigating organisational structures and incentives which may discourage the sorts of risk that experimentation and change involve.

Thomke cautions against relying too heavily on data as a predictor of innovation success. “There are inherent limits in relying on data alone — if something is very novel then there is usually very sparse data around.”

Also any hints of correlation need to be handled with care. He offers as example the fact that ice cream sales correlate with drowning deaths — the common cause is warm weather. Without context, raw data can be confusing.

Instead, he advocates well-designed and rigorous business experiments.

Thomke quotes Amazon founder Jeff Bezos — “Our success at Amazon is a function of how many experiments we do per year, per month, per week, per day” — and Thomas Edison — “The real measure of success is the number of experiments that can be crowded into 24 hours.”

“Microsoft now does more than 15,000 online experiments a year — involving up to tens of millions of users,” says Thomke. Still only 10–20 per cent of those experiments are successful. Thomke argues that they are, however, immensely valuable.

“You want to know you are wrong early on, before you launch.” Expensive marketing campaigns for new products or services won’t remove its flaws. Experiments, however, are cheap. If they prove something works, well and good — if not, they are the foundation for more nuanced hypotheses and iterations.

As a final reminder, Thomke notes the warning from physicist Richard Feynman: “It doesn’t matter how beautiful your theory is, it doesn’t matter how smart you are. If it doesn’t agree with experiment, it’s wrong.”

About the Author

Dan Ross is the managing director of Optimizely ANZ which is a corporate member of the Which-50 Digital Intelligence Unit. Members provide their insights and expertise for the benefit of our readers. Membership fees apply

Must Reads

Advertisers will lose $42 billion of ad spend globally this year to fraudulent activities committed via online, mobile and in-app advertising. Forty. Two. Billion. The figures are contained in a new study by Juniper Research. The figure is bad, but it’s even worse when you realise that it’s accelerating. Back