Red Hat IT's DevOps Journey: harder than we thought

Back in December, I shared with you how Red Hat IT was beginning its DevOps transformation. That early post discussed our strategy, team composition, the importance of Open Source, and incremental change. It’s been about three months and I’d like to share an update with some early lessons learned.

Overall impression so far? This is harder than we thought.

To recap, our approach to jump-starting a DevOps transformation in Red Hat IT was to form a small, temporary team inside Red Hat IT made up of six individuals from development, ops, security, and program management backgrounds. We’d have 18-24 months to influence change, build tools, coach teams, facilitate collaboration, and then go back to our “normal” jobs in Red Hat IT. Today, we’re about four months in.

At the start, I thought, “awesome–we have a top-notch team of four engineers and a product owner, freed up to just fix stuff and connect teams as their day job. We will be off to the races faster than you can say ‘DevOps’ and have a string of major successes under our belt by Valentine’s Day.”

The reality, however, was that the six of us had never worked together and only half had significant experience working on an Agile team. And, we had a very broad mission to start with. It took three months to start performing anywhere near the team’s collective potential. Chalk up my unrealistic expectations to only having led teams that I slowly grew or inherited. But, I’ll share here some lessons learned that I think should help anyone approaching a similar mission of enterprise DevOps transformation.

1. It takes time to form a new team. In reflection, it took a lot of work just to get to the starting line. Our biggest challenge was the human one: to get over being polite and to begin to trust and then challenge one another. And, because of the unique nature of our mission, we had to develop outside relationships–as a team–with our stakeholders. Plus, what was our mission, and what do we need to do to accomplish it? We all had very different ideas on what “enable DevOps for Red Hat IT” actually meant. Last, we had all the “new team” logistics to sort out (meeting times, seating arrangements, electronic communications, et cetera ad nauseam). All of this took a deceptively high amount of time and effort.

2. “Transition our enterprise IT department to DevOps” is a huge problem set. It’s a serious and ongoing effort to decompose that into actual work you can take on. At our start, our team members had wildly different opinions on what that work should be, ranging from “build a platform-agnostic version of Netflix OSS’ Asgard to run on OpenStack, etc.” to “fundamentally change the way departments like Marketing, Engineering, Sales, and IT collaborate with one another.”

To begin to dissect the problem, we collected data. Lots of it. Mostly, we simply asked our few-hundred colleagues “what’s not working for you in IT today?” across several channels: an in-person “complaint fest,” a video conference session for our friends overseas, copious discussion threads on our intranet site, and simple in-person meetings with other teams and individuals. This generated mountains of input. It took weeks of analysis, debate, and iterative planning to distill that down to an initial “theme” that our team could all sufficiently agree to and rally behind.

(FWIW, that theme is to automate Red Hat IT’s RPM-based software deployments to all environments–Dev, QA, Stage, Prod–focusing first on a single product from a single development team as our proof-of-concept. And, to lead by example and visibly demonstrate DevOps values like collaboration, openness, waste reduction, and engineering rigor while doing that. But that deserves a blog post of its own.)

While we worked to collect and distill all that input, we simultaneously focused on “quick wins”–building small software tools to help out other IT teams wherever we could. So, we still got some valuable, usable nuggets from this phase of our team’s development: an open source tool for developers to self-service inspect the state of machines to which they can’t login, jsonstats / talook, and the git-branch-blacklist utility, both huge time / interrupt / headache savers for several of our IT teams.

3. Language differences are an artifact of a siloed culture, and present a hurdle for DevOps. The Inception team in Red Hat IT is diverse by design, with members from development, operations, security, and program management backgrounds. We discovered early (but not right away) that even between our few team members, the same terms often had different meaning. In one grooming session, two team members–one sysadmin, one infosec wonk–spent 10 minutes talking past each other because to one of them, the term “full stack” included the load balancers; to the other, it didn’t.

If we are having these communication issues with our small, focused, collocated team of native English speakers, then how much is lost in translation when teams send each other tickets at a rate of dozens or hundreds per week in a mid-sized IT shop? How many hours are lost, and customer-impacting errors are made, due to a lack of common language between teams?

With these language differences, how on Earth are we going to encourage better communication between teams that each work in their own domain, and span the globe? Red Hat has offices in 33 countries, and IT has a presence in many of these.

I think the answer is actually right there in the question: we encourage better communication between teams, and the semantic differences of words eventually resolve themselves. Maybe other organizations have solved this effectively with some master glossary (I’m skeptical) but I’m pretty sure this can only be solved by building stronger channels of communication between teams (though I’d love a counterexample–tweet me!)

In summary, if you’re a leader–an architect, a tech lead, an operations manager, a CTO–embarking on a DevOps transformation project in your own enterprise IT shop, give yourself a little space to form your team, break down your mission, and get everyone speaking the same language. You’ll be happier and more successful for it.

Thanks for this Bill. It’s nice to see an honest assessment of how things are going within a company as they attempt to embrace DevOps. Too often companies whitewash out the problems and it gives the impression that all of this is easy, except it’s not. Everyone is figuring it out together and it helps when we highlight not just the successes, but also the hurdles and failures.

Thanks Mark, I appreciate the kind feedback. My goal with these posts is to get the worldwide DevOps community talking about what DevOps means in the “real world,” beyond the handful of “unicorns” we all know and love, like Etsy, Netflix, Flickr, Facebook, etc. There’s plenty to learn from the unicorns, but being “born DevOps” or “DevOps by default” is very different than taking a “horse”–an existing decades-old organization–and transforming the IT culture to break down silos, embrace rapid experimentation, and raise the value of engineering rigor. I’m interested in generating conversation about the horses.

Bill, I’m curious how much you would value ability to communicate (verbal, written, etc.) on such a cross-functional team. Did you specifically hire for this skill? If not, would you next time?

It’s probably obvious I think communication is skill #1 on a cross-functional team. Also, having been recently embedded (as a DBA/Ops onto Dev teams), I find this to be a barrier. Thank goodness for whiteboards for pictures, because I feel like we are talking in two different languages. But learning, always learning.

Communication is huge for us. I wouldn’t say we picked our team with “communication skills” a conscious priority, but it has proven to be core to what we’re trying to do.

In addition to the semantics barrier you highlighted in your recent embedded role (e.g. DBAs talking to Devs), we also have the challenge of being a globally distributed organization. Humans receive most of their information non-verbally so this makes our organizational communication challenges greater.

I won’t suggest we’ve cracked the communications nut, but we have found video conferencing to be very helpful. We’ve used it successfully for both intra- and extra-team communications, and seem to be using it more as we get proficient with its technological quirks and limitations.

I’d love to hear more about how you and others are improving communications in their IT shops to break down silos and promote sharing and collaboration.

Nice article, Bill. I like to see folks underscoring the collaboration aspect of DevOps. There is much more to building DevOps capability than installing the orchestration product du jour and creating a catalog of application stacks. Picking apart the knits and knots tied in IT by a decade of ITIL and a focus on enhancing the efficiency of each technology vertical in a silo is a non-trivial effort. Rebuilding it to include the business and its application development teams is an NP-complete problem – we will know the solution when we see it, but we are all still searching for it!

Indeed, there is no blueprint for DevOps in the enterprise (though I’m looking forward to Gene Kim’s upcoming DevOps Cookbook). But, I’m confident and emerging blueprints will include a cultural focus on outcomes for the customer, rather than your particular function, and increased communication between groups.

Great article, Bill! I liked particularly the “data-driven” approach to your DevOps journey: a simple survey does wonder and helps keeps tracking of progresses and collect feedback from the ones that are most invested in the transformation. I’d say focus on the incremental wins of DevOps; even the smallest, apparently insignificant change introduced has the potential to profoundly transform the professional life of who buys in the cultural shift of DevOps. And it’s journey! In my almost two years of observation of DevOps practises I still have to see anyone that proudly claimed: “Habemus DevOps!” 🙂

Great post, and it is missing a key destructor ingredient as it should. The “we are the smartest folks in the company” ingredient that invariably invalidates any decent effort. Redhat remains my favorite linux flavor.

Thanks for the feedback. Indeed, we’re privileged to work with a lot of smart folks here at Red Hat. Our team has found success by focusing on being good listeners, learning, adapting, and staying passionate about our mission.