It's a Cycle of Life Thing: Managing Linux Releases

Doing the Penguin Dance

One of the things I used look forward to about twice a year was a new release
of Red Hat (RH) Linux. I'm pretty conservative on my main desktop system, where
I write my articles and do my main development, and tend to run a pretty
out-of-the-box configuration. I leave my hotrodding for my experimental
system, where it doesn't matter if I accidentally trash a disk. Up until
recently (say, Red Hat 7.2), upgrading my system wasn't a major hassle, because
the overall rate of change of packages was relatively small and the
improvements in system performance, as well as the look and feel of the user
interface/experience, got to be more fun with each new release.

The release of the 2.4 series kernel made a lot more functionality available
to developers, and the Linux community has taken advantage of it with wild
abandon. With the release of Red Hat 7.3 (and SuSE 8.0, and most other Linux
distributions from about mid-2001), I noticed a sudden bump in the number of
applications available and a radical change in the dependencies in any given
distribution, release after release.

For many users, upgrading a system or two is not a major hassle; they put in
the CD-ROMs and go. However, the picture gets a bit more interesting if you're
an enterprise using Linux for development or, dare I say it, as the corporate
desktop standard for your users. A small change to a critical library (like
glibc or gtk) can cause applications to break. A
change in application functionality sprung on unwary users can cause support
headaches galore. One of the great things about Linux is how quickly new
software and services are becoming available--but it's also turning into a
major headache for enterprise IT support people who have to keep track of all
of these changes and software versions. Let's face it, the system itself (and
its OS) is in reality only a small part of the Total Cost of Ownership (TCO)
and one reason why end-users would rather fight than switch when it comes to
operating systems and applications.

One of the selling points and competitive levers for commercial Linux
distributions is how many applications they bring to the table for their
various constituencies. For power programmers, it's all about the depth of the
development environment. For those deploying web services, it's the degree of
support for Apache and related technologies like Tomcat. Most recently, for the
desktop user, it's been the ever-growing number of applications available on
the desktop, like StarOffice and OpenOffice.org.

It's interesting to note that in the RH7.2 release there were perhaps 75
desktop GUI-based packages and utilities available. Under RH7.3, and now 8.0, a
default, standard desktop system includes over 200 (yes, 200!) unique
applications installed and available for the GNOME and KDE desktop
environments. All of these programs have some level of dependence on other
parts of the system, and as the depth of applications grow, so does the level
of dependence.

What's Really in a Linux Distribution?

Before Red Hat developed the first serious package manager--the Redhat
Package Manager, or "RPM"--for Linux, there was a list of packages that you
needed to install in order to make a bootable system. This list consisted of
the Linux Kernel, glibc, various "userland" utilities (e.g., the
GNU binutils), networking tools like ftp and telnet,
and a few shells. Everything else was optional. Fast-forward to 2003: the
smallest possible automated installations (in terms of packages) from the major
Linux vendors look like this:

Wow, that's a lot of stuff isn't it? That doesn't include X11, or any user-level applications like OpenOffice.org or even freecell. If you
count just the basic packages that make up X11, window managers like GNOME or
KDE, support libraries, and user-level productivity tools, you can add up to 100
more packages to this list. This isn't a bad thing, but it's clear that a
minimum usable, networked system comprises a lot of software.

Imagine that you run a large enterprise and support not only the base
operating system across hundreds or thousands of machines, but also end-user
applications ranging from office productivity to CRM, or order management, or
customer-service applications. It also makes for a lot of packages for an IT
manager to support.

What do Enterprises Need?

One of the hardest things for any enterprise to deal with is change. Back
in the days of the Internet bubble when we thought (wrongly) that old paradigms
didn't matter, no amount of disruption to the workflow of an organization was
too radical. Now, we understand that change is good, but unexpected change can
damage the ability of people to work together and get things done. In
deploying IT solutions, the two hardest parts of any deployment are 1)
consistent equipment configuration and 2) training the people who will use it
in the software packages they need to get their jobs done.

Some people might suggest that this is a non-issue and that services such as
the Red Hat Network allow IT managers to tame this complexity with automatic
updating. This is true, provided you always want Red Hat's suggestions. What
if installing a new version of OpenOffice.org or some other application would
mean that your staff needs to be retrained, or a critical feature changes
in some way that negatively affects your operations? What happens then?

The ability to upgrade systems in a logical way that minimizes system-level
changes (except for bug fixes) and allows the enterprise to keep using
older/existing versions of applications to help manage user expectations and
constrain retraining is extremely important.

In the days of mainframes and mini-computers, upgrades were a lot more
logical and well-contained because an upgrade applied to a system shared by a
large number of users. With workstations and PCs, even, a small upgrade can
cause issues far beyond the scope of a single package, as anyone who has ever
had the pleasure of maintaining even a small gaggle of Microsoft Windows-based
systems can attest. One of the things that made the non-PC world manageable
was that, for the most part, hardware vendors made a very clean separation
between the operating system and the applications.

Check out what went on at the O'Reilly booth and find out who the drawing winners are.

Back in January at the NY LinuxWorld Expo, it was interesting to hear a
developer from Dell opine onstage that the current "problem" with Linux
distributions was all about the number of releases per year and that "things
were settling down to a nice 18-month release cycle." I contend that the
release cycle of distributions as a whole is less of a problem than the lack of
a clean interface for layering products onto Linux systems. There can be no
argument that the speed of Linux development--especially from a security
perspective--is a good thing; a cleaner mechanism is needed in order to
support multiple versions of software to help IT managers keep end-user-visible
changes and disruptions under control.

Chuck Yerkes, a consultant specializing in Unix and sendmail, suggests
that the Linux world could benefit from the experiences of the BSD world, where
almost all non-core functionality is installed in a non-root tree. Instead of
a package like OpenOffice living under /usr/bin, it
would live in /packages or a similar directory that allows
multiple versions of packages to be installed easily. Of course, this
"re-rooting" can be done now with RPM and other package managers, but it has to
be done manually.

One way to look at this is that, in the old parlance of Digital's VAX/VMS,
the core of the operating system itself is a separate product from the "layered
products" used by end users. It should be possible to install or upgrade the
core of the Linux operating system without forcing the upgrade of the layered
packages. In fact, it should be possible for a Linux system to support
multiple versions of a package along with whatever underlying libraries are
needed, without causing application dependencies to break. This might seem to
add major complications to Linux distributions, since vendors would have
to track and understand more details about every package they include in their
distributions, but it would make the adoption of Linux by large and medium-sized organizations a lot easier by making their support mechanisms more
clear-cut and predicable.

A Call to Distribution Vendors and the LSB

It seems to me that it's time for Linux distribution vendors, backed by the
Linux Standards Base (LSB), to break
the core OS out from the rest of the "layered products" that make up the bulk
of their offerings. Make all of the end-user applications into a separately-selectable set of installable packages and develop a way to allow for multiple
versions of packages to coexist on the same system. In the old days, under
mainframes and minicomputers, this was done with "logical names" under VMS and
virtual directories under VM and other OSes. The BSD world seems to have
navigated this shoal; it shouldn't be to hard for Linux world to do so, either.

Acknowledgements

Deep thanks to Steve Jones of Crash Computing, Chuck Yerkes of SNEW.com,
Stephen Degler, Andrew "Bobcat" Templin, and Richard Rognlie, "Sr. Lackey" for
Gamerz.NET Enterprises, for helping me sift through bits and pieces of various
Linux distributions and offering suggestions on how maintaining large Linux
installations could be better handled.

David HM Spector
is President & CEO of Really Fast Systems, LLC, an
infrastructure consulting and product development company based in New York