Three Golden Rules for Continuous Delivery

What are the three major golden rules for continuous delivery? Why are they important? In addition, how can these help an organization and team deliver software? In this blog post, I will be covering how the three golden rules for continuous delivery address and answer these questions.

Applies to

Three Golden Rules of Continuous Delivery

Golden Rule 1. “Don’t Break Your Customer.”

The objective of Continuous Delivery (CD) is to achieve a continuous and reliable flow of features into production for our customers (internal or external). The CD process provides the customer with a predictable cycle of feature delivery that is expected in today’s software development. Customers do not want the new features to break the existing ones. Even worse is to cause an interruption to the business workflow that the customer trusts. Imagine the impact on customers if Bing, Google, or Facebook, went down for 1 hour. Breaking your customer can be extremely costly!

To give another example of “breaking your customer,” even in a small way can have a large impact. Consider how disappointing it is when you buy something and it only works for a short period of time and then breaks. As a result, those type of devices and software quickly gain the reputation of a throwaway or cheap solution. With social media in full swing, that new reputation created by a bad user experience can be widespread within hours. (That is if Bing, Google, and Facebook are not down for that same hour.J) Something I am sure of, is that you do not want your hard work to gain a cheap, throwaway reputation.

Today we all understand that software development is not perfect – science and Murphy’s Law unfortunately will happen. It is how we react to the unexpected mishaps that make the difference in keeping our customer’s trust, which is very hard to gain but very easy to lose.

We all know what happens when we try to check-in a large number of changes into source control to find that the large check-in broke the build or tests. It might take hours of troubleshooting and debugging to find that two lines of code broke everything. All the while, the development team is blocked from checking code in. All the while, you are thinking: ‘I am never going to do a big check-in like that again’.

This is at the heart of the golden rule 2 “Don’t Break the Code Base, Builds, Tests, and Production.” The first three might seem straightforward and most of us may already do it; however, I believe that this rule stops short when it comes to production. The example above is the same when doing a big-bang release. As more features and code are deployed in a single release, the higher the risk that something will break and the longer it will take to fix it.

We all have seen the cost model of fixing a bug in production, (requirements stage costs 1%, development stage costs 10%, testing stage costs 100%, and production stage costs 1000%). Production bugs are going to happen and some bugs might only be found in production. This is why having a consistent delivery pipeline and matching testing infrastructure is key to not breaking production.

Let’s dive a bit deeper as I share a real life experience that unfortunately displays this perfectly. A few years back, the company I was working for at the time had a big release of a new product that was going to sunset the current product. The development and test team worked for 18 months on the new product. About 6 months before release, the teams started to engage the operations team to see if they could do a beta release for the production system. Well, you can most likely guess what is coming up next. To make this oh so common story short, I am going to cut to the good stuff. The operations teams were only able to create the platform to meet the release date. And on the release date, the production systems had to be updated with new framework and third-party software which was only in development and test environments. This broke production guidelines and best practices that the team had work so hard to adhere too. It took operations months after the big-bang release and the following critical releases to stabilize the environment. After the big release was complete, the operations team was pissed-off and did not want to do anything to improve the system.

What could have been done differently was to deliver smaller units to a production environment earlier. Small or big releases should not be a major event! Releases should be something that can be done, even at or during the celebration party. It also would have provided critical training on how to get features out and how to debug the system iteratively, resulting in a quick fix.

Coincidentally, rule 2 “don’t break you code base, builds, testing, and production” follows rule 1 “don’t break your customers” because as we all know production is the customer for the QA and development teams.

Golden Rule 3. “Don’t Break the Delivery Flow.”

Breaking a delivery flow can be done in a number of ways: trying to push more through the delivery flow than it can handle, changing the delivery flow for the one-off “Ninja Features”, or not having a consistent delivery pipeline from development to production. The hidden cost of breaking a delivery flow can be exposed by asking how fast you can deliver what is in development today. The hidden costs can range from extra time in task switching, to total chaos and stress during the release of a piece of code into production.

The other question l like to ask is how fast can you rollback a change in production and does your current model support rollback? This is somewhat of a trick question. You should always roll forward, never backwards. Rolling back is disaster recovery and a release should not be a disaster.

Another question is, how mature is your delivery model and flow? If your delivery model is fragile, then you are not able to react to changes to the environment, customers, or business. Risks are higher, and cost more both in time and frustration.

Developing a solid delivery model should fall into the same process as all other agile practices. The continuous delivery is a bi-product of developing a mature delivery pipeline and good engineering practices. It should not be Voodoo and invoking the Dark Arts to get a feature deployment to ANY environment. For teams from development, operation, to the help-desk, an organization’s delivery pipeline should be well known, automated, and trusted.

Throughout my many years of experience, I have created two phrases: “no supersize release” and “no surprise release”. These phrases drove my team to do smaller feature releases, making sure we did not break the delivery flow.

Conclusion

The three golden rules for continuous delivery is only the beginning of a very complex process. Continuous delivery is not easy or inexpensive to achieve. The cost of not having a delivery mature model in today’s market can be even more expensive and have a larger impact on the business and teams. The human factor, constraints, and hidden costs of delivering software is the difficult part to change and find within an organization. These three golden rules of continuous delivery provide a framework to achieve:

· Customer satisfaction and trust
· A stable and trusted delivery system
· Reliable model to react quickly to changes, with less risk