You Can’t Buy DevOps

There are organizations that want to “buy DevOps,” like it is a plugin to add to the development process. They often create a new role, team, department, or infrastructure. But you can't buy DevOps, and it's not a designated team, either. It is the idea of people working together. Here are some approaches to get you there.

Just in the past year or two, when I’ve talked to technical staff and customers in my own work, I have noticed a trend: the dreaded DevOps team.

That is to say, the organization that wants to “buy DevOps,” like it is a plugin to add to the development process. To do this, they create a new role, often a new team, department, or infrastructure. At one Fortune 500 company I worked with, I met the director of the private cloud infrastructure project, which would take the next eighteen to thirty-six months to build out a private cloud so anyone could have DevOps on tap. It sounded good—at least, on paper—but there were a few problems.

First, consider the delivery team that wants to automate their build-release-deploy process. Instead of actually doing it, the delivery team is now waiting on an infrastructure team to deliver value (in eighteen to thirty-six months) that might be an easy plug-in to what the delivery team is doing now, or it might not be.

The DevOps group needs to hire staff like mad, so it’s looking for OpenStack administrators who understand a wide variety of Linux and Microsoft operating systems at scale, except that has probably changed to Docker by now.

Those people don’t exist—or, at least, if every large IT organization tried to hire those people, it would create a bidding war. That might be good for the few people who have the experience, but for the poor members of the delivery team, it means hiring is delayed, ramp-up time is delayed, support is delayed …

Meanwhile, vendors have jumped on the DevOps bandwagon and are happy to rebrand all the testing, monitoring, and support tools you’ve always been using (or couldn’t afford) with “DevOps.”

The idea that DevOps is something you can buy and install strikes again, slowing down delivery in the name of improvement.

And it gets a little worse.

Handoffs and Performance

One of the great things about a fully skilled, multi-discipline team is that the team can work together to move development forward all the time. As soon as you encounter dependencies, such as the team needing to request an external database administrator to add an index, or test needs to wait for a slow build, you slow the team down. I have seen projects that should have taken a morning take weeks because the work required the support of several different teams at one point in time, and each external group has a one-week service-level agreement. That meant wait a week, get a fix, assign the ticket to the next team, wait a week, get a fix, and repeat.

Even if the “DevOps person” is embedded in the team, adding a new column to the work board, and thus a handoff, will create a delay. At my company, we model this in our Lean Course, simulating a factory that has completely balanced throughput but small variation. Over time, bottlenecks emerge, and the more stations you have, the more bottlenecks and the worse things get. You don’t need to take the course; I wrote a little Ruby code on GitHub to run the simulation a few thousand times and report the results.

The bottom line here is that handoffs slow down delivery. You might need to have them, but carefully consider the case.

So DevOps isn’t something you can buy and have the new hires do. What should we do instead?

An Emergent Approach

Most of the companies we work with have teams with wildly different technology stacks. Trying to create a single, external team to provide services to them is often the classic programmer’s error of premature abstraction.

Instead of one DevOps to rule them all, why not allow each team to experiment, working on whatever the blocking condition is for them right now? That might be better build management, system test tools, automated deploy, or automated provisioning—the ability to create a test server on demand. Those things all involve speeding up the build-test-deploy process using smart, modern computer science tools. To make them work, the programmers and the operations person, and possibly the tester, would sit together and work out what is done now, and what automation might bring to the table.

The next problem is funding it. There are two approaches I’ve seen for the team to get time to work on these sorts of things.

The first is for the technical staff to experiment enough to propose some solution, with some idea of the value (30 percent improvement in delivery speed) and the cost (half the team for two months) to accomplish the improvement. If management doesn’t want to fund it, that’s fine. In six months, when management is asking, “Why are we going so slow?” the team will have a reason and a potential plan in place to go faster!

Honestly, I haven’t had much success with that approach. It’s great for small, pointed improvements, but large infrastructure projects are much more challenging.

The second method is to explain the return on investment in high-level terms and to get management to commit some percentage of time to “work on the DevOps.” (Yes, I know that’s terrible, but it makes the point.) Google’s famed 20 percent time is one example. It can also be accountable; the team can track stories on the board for improvement and reflect after a few sprints on how things are going.

Growing the Seed

Let’s say the company has ten technical teams, each dedicating 10 percent of their work to infrastructure speed improvements for six months. The goal here is for the team to grow wider, embracing operations and deploy, while speeding up delivery itself.

On top of the teams, I suggest a community of practice that meets monthly to discuss what the teams are doing. Some groups may be able to share patterns, and, in some cases, even code, developing a broad, general, shared understanding of how to automate parts of the delivery process. That’s not a standard; it’s more like a guideline or a living document.

Eventually, there might be enough critical mass to create a centralized group and coordinate the efforts of various teams, connecting the pieces and creating the abstraction layer. But let the organization take three or five or ten tries at it first, at the team level. If they mess up, it will only impact one team, and more broad ideas will begin to bubble up naturally. Experiment from the bottom up, not the top down.

DevOps is not a team. It is the idea of people working together. We should treat it that way.

Appreciate the insight Matthew. As a new tester, coming from a BA background, articles such as this make things very "plain" for me. Seems like at the end of the day, it's all about communication and collaboration.

How is an organization supposed to get on the same page if every coder writes a little code in their own little way? What about compliance, standards, scaleability? And, what if the project manager or multiple Principal Architects or Engineers disappears one day? How do you get in their head about what they were thinking that day they wrote that little code? Am I way off on this?

My experiences echo's yours, Matthew. We all want to "buy" the hard stuff rather than develop it. The changes containerization force on even the Agile and Lean development teams becomes difficult to explain to non-tech's (ex. Proj. Mgr's).

I have found the percentage option in your article to be the most palatable to non-tech's provided you can "read the story" regularly on how much that percentage of time is gaining them. I recommend Toastmaster's to anyone who finds it difficult to describe good things to non-tech's.

About the author

Matthew Heusser is a consulting software tester and software process naturalist, who has spent his entire adult life developing, testing, and managing software projects. He has served as the lead organizer of the Great Lakes Software Excellence Conference, organized a workshop on technical debt, and taught information systems at Calvin College. Matthew blogs at Creative Chaos, is a contributing editor to Software Test & Quality Assurance magazine, and is on the board of directors of the Association for Software Testing. Matthew recently served as lead editor for How to Reduce the Cost of Software Testing (Taylor and Francis, 2011). Follow Matthew on Twitter at @mheusser or email him at [email protected].