Building Red Hat Enterprise Linux With Our Partners

Building Red Hat Enterprise Linux is a grand departure from the days when operating systems were designed, built, tested, and shipped by one company that also sold the hardware and applications. Ten years after the debut of Red Hat Enterprise Linux, the game-changing effect of community powered innovation is well understood. By now, we’ve all realized how the collective efforts of Red Hat and our OEM, ISV and IHV partners drive benefits for our mutual customers. Together, we spearhead upstream open source development to exploit innovative silicon and hardware and design software features for physical, virtual, and cloud deployments that can reap immense benefits for customers demanding integrated solutions. Our approach extends the definition of community well beyond the open source communities like Fedora or JBoss. It extends to include our customers and partners. Among these partners are many ISVs and our extensive work with them is a story for a future blog.

It sounds pretty straightforward, right? Building Red Hat Enterprise Linux with our partners looks deceptively simple but requires tremendous stamina; strong, long-term personal relationships between our combined product and engineering teams; rigorous processes simple enough to be followed by internal and external teams, and commitment to constant, bi-directional communication. Repeatedly, we see that these relationships in the development and product team trenches are what ultimately get us over the inevitable bumps that arise during product development cycles involving many companies with competing agendas and interests. In the last decade, it’s become part of our ‘DNA’ to effectively manage competing corporate agendas, withering schedule misalignments and delays, and daunting competitive pressures. It’s these person-to-person relationships built on years of cooperation and trust that we expect will continue to see us through the next decade of Red Hat Enterprise Linux development. To the uninitiated, it might look like Rinse, Lather, Repeat. It is anything but.

The road to Red Hat Enterprise Linux product launch starts years before the product reaches a single user, with our requirements gathering process where we formally solicit feature requests from our partners. Feature requests come to us from all directions through a dizzying array of channels, including, first and foremost, customers, and OEM, IHV, and ISV partners. The challenge? To choose the requests that best balance the required hardware support, customer requirements, technical excellence, ease-of-use capabilities, and adherence to upstream Linux direction.

After a few rounds of three-way conversations with the silicon vendors and OEMs, we better understand the details behind each feature request. For example, ‘provide OS support for OEM ABC’s new XYZ400 series computer’ is revealed to be a mega feature request, behind which is a series of required kernel device drivers, PCI IDs, and installer hooks. Features requested across multiple partners are, predictably, prioritized above one-offs. We spend countless hours evaluating and prioritizing customer, partner, and internal feature requests – sometimes exceeding 1,000 – for a major Red Hat Enterprise Linux release. Each feature chosen is subjected to a careful, case-by-case analysis to assess risk vs. benefit. Ultimately, the list is pared down to something achievable by the development team.

Then the development process begins. Red Hat Enterprise Linux engineers contribute a substantial percentage of upstream kernel development. The productivity, breadth of knowledge, and ability to stay on top of a torrent of internal and external communication demonstrated daily by these world-class kernel developers is beyond impressive. And, rare in in the software industry, Red Hat invites developers from our partners to become embedded within our development and quality engineering teams. Working behind the Red Hat firewall and sharing office space, comments on our kernel development mailing lists, and snacks and cafeteria food, the presence of these on-site partner developers creates a rich team environment for building great products. Once upon a time, embedded partner teams might have looked like a wacky social experiment. Today, it’s a proven winning recipe for building and testing product on a tight time line to meet the wildly divergent needs of customers and partners.

The marathon continues. Nightly builds become weekly builds, which in turn, become alphas, then betas, then release candidates. And throughout, our partners contribute in real-time, test in real-time and work upstream in real-time to backport their upstream work to Red Hat Enterprise Linux. Did we mention the late-breaking feature requests as we approach feature freeze? And, while we’re working on this release, the next release is taking shape and the race begins again.

The result? A major release of Red Hat Enterprise Linux with a diverse set of partner and customer requirements built in from Day 1 that have been rigorously tested throughout the development cycle by Red Hat and our partners’ global test and quality teams. Deployed during high-touch betas with customers demanding the tightest possible integration of the hardware and operating system features, Red Hat Enterprise Linux is much more than a Linux distribution. Other companies may pull together a random collection of bits and bytes from the upstream open source community, slap their name on it, and call it a product. Other companies may leverage existing products and add a few bells and whistles for their specific customer base. But we take it beyond that point. Red Hat Enterprise Linux is built - no crafted through the ongoing efforts of the open source community, Red Hat, and vast numbers of partner engineers and test teams, on-site and remote.

Arguably it may, upon reflection, indeed be one of the grandest and successful social experiments.