Why DevOps orgs embrace failure

DevOps transformations require embracing failure

Innovative achievements on our planet seem to manifest themselves after copious amounts of failure. Most of us are at least somewhat familiar with history’s comically hilarious attempts to create a flying machine. During the early days of flight, flying machines were invented in droves. Most of these contraptions had disastrous results.

One could at this point logically ask: ‘If innovation stems from failure, why are we humans so afraid to fail? Is this fear of failure relatable to a financial cost? OR does the fear of failure relate to a cost in credibility?’ Let’s explore.

While we look at the flying machines now with a level of jovial humor, the cost of failure at the time was actually very high. During the quest for flight hundreds were injured and even killed, multitudes of financial investments were lost in a search for what some thought was impossible AND even worse the people attempting to fly were hysterically laughed at. Logically speaking the cost of failure in attempting fly was about as high as it gets. Yet people continued to try. Eventually the Wright brothers came a long and the whole pitch changed. All humor aside, the point to this is that the achievement of manned flight was only successful after attempting, failing and then trying something new. Regardless of cost.

Failure is simply an opportunity to begin again, only this time more intelligently — Henry Ford

In engineering enterprises us software engineers have the luxury of a lower fatality rate as a result of failure (This is dependent on the country you reside in).. yet in some ways it seems we are more afraid of failure than ever. In todays busy enterprise the context of the word ‘failure’ might be an idea that didn’t work, a creation that no one uses, a product that didn’t sell, a set of flaws in a software product that caused customers to leave or even a failure of imagination. From this perspective one might say we experience failure all the time and don’t even realize it. Enter DevOps.

A DevOps Journey

Organizations decisively embark on a DevOps journey for any number of reasons. In some cases the journey becomes necessary due to slow and inefficient development and release processes. In others it might simply be because DevOps is a good buzzword [BTW thats a really terrible reason to embark on this quest]. Whatever the reason, once the journey has begun there are a number of challenges that will need to be overcome in order for the quest to be successful. Some of the more common ones I have encountered ones are listed next:

People – DevOps is first and foremost about collaboration between traditional silos. Everyone in the ORG (management, sales etc.) will need to be on the same page on what DevOps is, and the cultural impacts

Process – Building feedback loops and delivery pipelines are only valuable if the processes implemented are structured in such a way where they are efficient, well defined and standardized.

Automation – We can only automate what is well defined and known about. Any automation written can be viewed as a codified version of manual processes that were once in place. Get the process down first, then automate it.

Ownership & Pride in the Craft – In a DevOps environment, Developers, Quality Assurance Personnel, and Operations must have alignment on goals. Silos MUST be torn down as a result and highly collaborative environment put in place.

Velocity & Pivots – Velocity in this context is the release throughput of a delivery pipeline. Once delivered the idea is to pivot development efforts based on feedback from customers. These two points go hand in hand…. what’s the point of pivoting on customer demands if it takes 2 years to get a requested feature delivered back to them.

Pivots, Velocity & Failure (The essence of Agility)

The Agile manifesto (you know those 12 pesky philosophy points created by those old guys that somehow changed software in the modern day) adequately identified a multitude of basic DevOps tenants. One of the more DevOps appropriate ones being #2:

One way of looking at a changing requirement is to consider a failed experiment. A failed experiment (even late in the game) would mean that the dev team needs to go back to the drawing board and figure out what worked and what didn’t. In this sense the organization would need to pivot the direction of development after failing or faltering. Here is where velocity and failure fit into Agility.

Failing Strategically instead of clumsily

Failing publicly (With the exception of SpaceX [as all of you would agree]) is not good for business. Instead of failing publicly why not try failing privately. This could be accomplished by implementing a replica or your production environment. Once implemented the org privately delivers changes to prior to physically delivering them it to prod.

Taking a strategic approach to failure (maybe only fail in front of a couple users instead of 300 users) is another strategic way to approach failure. Through the modernization of A/B testing this is now pretty easy to do.

An additional strategic way to fail is to develop a feature toggle solution. This allows the org to enable/disable features through the flip of a switch. Thus your developers can deliver something to production enable it for a short time (when it’s ready), and then disable it if needed. This is a much better approach to delivery than roll backs. It also provides a tool marketing can use to release customer facing features in a highly strategic way.

Delivering Efficiently

The only true way to complete a DevOps journey is through delivery pipelines. A pipeline is a well defined and highly automated approach to building, deploying, testing and accepting software changes in an incremental fashion. This allows the development processes to become efficient and ambiguity to be eliminated. As a result QA can process smaller chunks and customers can realize value incrementally. The more efficient your delivery process is the easier it will be to pivot. In reality this is a good measure of how agile you actually are.

Embracing Failure (Conclusion)

So what is the take away from all of the above. If there is one thing you take away from this piece its that failure is not to be avoided as thats 100% impossible. Instead fail strategically and make intelligent choices when organizing an engineering team. Look for ways to predict failure and see if there are ways to fail less publicly and in turn fail smarter. Then pivot and learn.