Pages

August 20, 2014

Enthused About: The Phoenix Project

I recently read The Phoenix Project, a short fictional book about DevOps. (I then deeply contemplated my life, and what it had become with me reading fictional books about my day job ....) But without hesitation, I think you should read The Phoenix Project too, and I'll tell you my reasons why.

These are the things about this book which I now keep recalling several months later in my job as a QA Developer. Here you have them, with my usual biases in play.

A team can only go as fast as its bottleneck. The book tells the story of a CM/infrastructure/script monkey named Brent. There are tasks in the company which only Brent knows how to do. At times, all of the critical work that needs to be done requires Brent's time, so the rest of the team stalls out. Anyone involved with testing has certainly seen this play out before. If you are working 2-week sprints on a Scrum team, adding effort on feature development isn't going to overcome your weeks-long regression cycles - it's only going to make that problem worse! If you want to go faster, get rid of the bottleneck and/or protect the resources required for it. I think that automation (of development (through frameworks and reuse at a minimum), of testing, of infrastructure configuration, of operations, of anything that's in the way) plays a key role here, so that only risky or truly tedious work requires intervention. On top of that, automation done correctly reduces risk.

Do what you can to make impact minimal, measurable, predictable, and transparent. In the book, a sort of guru shows up to give a VP-level character some field trips to a factory. Many of the things which are hidden or which seem insignificant when doing development work are made obvious by seeing them from above on the factory floor. Scrum development teams are often designed to be "cross-functional," in part because a cross-functional team can see its ideas all the way through to done, and can more directly control against blockers, dependencies, etc. Pragmatically, it also makes sense to have some expertise in horizontal support teams that serve many development teams, so that development teams don't necessarily require expertise on SVN server maintenance and the like. Much like the changes we've seen in testing over the past 8-10 years (from independent gatekeeper to expert enabler), I wonder if we'll see synonymous trends in release and build engineering to a self-service type of model. Under this model, the role of the support teams is less about actually doing the work and more about making it possible for development teams to do that work themselves.

So what does all of this have to do with impact? If an activity can be carried out by a development team, its impact (in terms of time, risk, predictably etc.) can be directly understood by that team even if its inner-workings are within a black box. In addition to automation, such an approach also depends heavily on isolation, as the changes made by Team X must be guaranteed not to negatively impact Team Y - but have we not had that technology for years thanks to VMs? And what about this crazy stuff with the lightweight containers?

Commit to a process to stop the bleeding. In the book, the characters are first overwhelmed at the thought of having a series of meetings to go over 1000s of change requests, but once they get a grip on process, they can easily manage a weekly delta. Projects with poor process build up technical debt. Technical debt will catch up with a team through production issues, long test cycles, costly hard-coding and manual intervention, and plenty of other ways. Soon enough, all the team is doing is paying down that debt, like the person only paying the interest off of his credit cards.

Process sometimes requires that the throughput of a team will slow down before it speeds back up. It might mean that some people on a team will sit idle, or spend cycles doing work that is beneath them, like writing unit tests or user documentation; but all of those tasks are paying off technical debt.

Finally, business value always matters. The book tells the story of a VP-level character in charge of Security for the company. The character becomes extremely depressed and walks out of the job because no one will listen to him about SOX audits and security risks. As it turns out, there were safeguards in place which made many of "required" changes redundant or less critical than when seen from only the eyes of a security professional.

I'm not one to dog on security (my first published paper was on the topic of security), but I did think this was an important lesson to see that business value must always be a consideration. In my case, I'm in love with test automation; but there are times when fully automating something, or taking the time to "do it right" (read: over-engineer that bad boy) does not align with the direction of the project or the company. I have to learn to recognize these cases and show some restraint.