But there were a few problems along the way. Projects often ran over time and over budget, management often complained that the final outcomes failed to deliver the goals described in the business plan, and by the time the new products or services arrived in the market- sometimes years later, the market – and the customers – had moved on.

These days, new approaches like Agile, Lean, DevOps and Continuous Development have collapsed development into a more iterative and rapid cycle.

Now, the smartest organisations are leveraging these new approaches to introduce a culture of experimentation. This is an important means of responding to the ever-growing expectations of customers who take their best experience in any context and apply it to every context.

According to Optimizely, ”Some of the most revolutionary new products and features were originally controversial, or counter-intuitive. Experimentation transforms the need for debate into an opportunity for discovery. With it, teams can now evaluate ideas and make decisions based on real, usable data.”

A starting assumption of the white paper is that it is not enough these days to simply optimise the performance and reliability of products.

Instead, Optimizely argue, organisations must optimise the actual benefits of their software as well.

However, in the white paper they caution, “Without sufficient experimentation product developers won’t have adequate information at hand to make the best product decisions possible.”

The best way forward, they say, is to expand experimentation capabilities and strengthen software development process.

To set the scene, many teams already embrace principles such as Agile, Lean, DevOps, and Continuous Development to ship quality software faster. In fact, Optimizely note that agile is now the norm with 94 per cent of companies saying that all or some of their development teams currently use agile methods.

By breaking applications into multiple services and shipping smaller changes more frequently, software teams are looking to mitigate risks and more rapidly to customer needs.

Many organisations have also migrated from monolithic applications to microservices architectures. Yet while all these changes are important and certainly lead to greater efficiency, they are not enough in themselves to confer genuine competitive advantage.

Instead, argue Optimizely, “[The approach] must also deliver real value to end users and customers faster and better than any other product. That’s where experimentation-driven product development comes in.”

Trailblazing

The shift to experimentation is a trail already blazed by some of the world’s most successful technology companies such as Google, Facebook, Microsoft, and Amazon.

These companies all run thousands of experiments each year and have scaled and managed their businesses by using experimentation as a core part of their product development process.

While not every aspect of their success is replicable, happily their adoption of an experimental approach contains lessons for all companies who build software to deliver better experiences.

Teams that successfully develop a culture of experimentation and execute effectively typically prepare around three principles.

Choose the right metrics: The most important step before launching an experiment is agreeing on the metrics to measure. Metrics allow teams to define the success of their experiments.

Your feature roadmap is also your testing roadmap: This is precisely what is meant by “experimentation-driven product development”. A core part of every sprint process should be to decide when to attach experiments to new features and feature updates. For riskier features, it’s helpful to experiment with a small portion of users before rolling the feature out for the entire customer base.

Test small and test often: Progressive teams are releasing smaller, incremental software iterations at a more frequent pace. While it’s useful to have some bigger, riskier experiments in the pipeline, running successive, smaller experiments helps diversify risk. Plus, these small bets sometimes pay huge dividends.

According to the Optimizely, “Adopting any new system will require some learning and additional work. But the upside is undeniable. Embracing experimentation-driven product development will make your team more innovative, your business more agile, and your customers more satisfied.”

They write, “By adopting experimentation, teams eliminate uncertainty and guesswork from their software development process. By exposing new versions of software to a portion of customers while the remainder continue with the original experience, teams can quickly assess if their new updates are improvements or regressions. Experimentation, in essence, becomes the feedback loop through which teams determine if their products are impacting customers in a positive way.”

About the Author

Andrew Birmingham is the director of the Which-50 Digital Intelligence Unit of which Optimizely is a member. DIU clients provide their insights for commentary for the benefit of our readers. Membership fees apply.

The Author

Andrew Birmingham

Andrew Birmingham is the editor-in-chief and publisher of Which-50. He is the former associate publisher of The Australian Financial Review and remains a contributing editor, and during his career he has reported on the Australian media, technology, finance, life science and related sectors over a period spanning 20 years. His work has been published by The AFR, The Australian, The Sydney Morning Herald, The Age, MIS, Computerworld, CIO, ARN, Network World, CRN Australia, and My Business.

Must Reads

Advertisers will lose $42 billion of ad spend globally this year to fraudulent activities committed via online, mobile and in-app advertising. Forty. Two. Billion. The figures are contained in a new study by Juniper Research. The figure is bad, but it’s even worse when you realise that it’s accelerating. Back