Version Vexations: Keeping Big Linux Operations on the Same Page

As data centers have grown in complexity, so has the task of keeping all or even most of the machines on the same page in terms of Linux version control and modeling. "Overall, the tension between 'We have to have the latest patches installed' and 'Leave well enough alone' means that a lot of different systems are at a lot of very different patch levels," Illuminata's Johnathan Eunice said.

Modern data centers with large number of Linux servers and heavy use of virtualization often don't have the tools they need to handle system version control and modeling.

"Twenty years ago, most companies had tens of servers to manage," said Erik Troan, founder and chief technology officer of rPath. "When Intel took over the data center, that grew to hundreds or maybe thousands. With virtualization increasing the number of logical systems, organizations with tens of thousands of individual systems are quite common."

That increase in scale and the ease with which virtual machines can be cloned make it critical for enterprises with large-scale Linux deployments to have version control tools.

"There is a great need for such tools," Al Hilwa, a program director at IDC, told LinuxInsider.

Not Your Gramma's System Tools

What about tools like HP OpenView and IBM's Tivoli? They seem quite adequate for managing large-scale enterprise IT infrastructures and have been used for this for years.

"When we talk about full system modeling and version control, we mean a method of specifying every piece of software installed on a machine, having a single version number associated with that set and being able to reliably move a system both forward and backwards across those versions," rPath's Troan told LinuxInsider.

"Some organizations do modeling via a Red Hat kick-start file or something, but there isn't full tooling around building the models, version controlling them or auditing them," Troan added.

Tools such as Tivoli or HP OpenView handle some aspects of these requirements, but not all, Troan said. They don't normally version sets of software, although they do let models specify a particular version of a particular set of software, he added. However, their modeling capabilities are often "rudimentary" and can't completely understand complex software dependency relationships or model the system at every level.

About RPM and Version Control

The RPM Package Manager is a command line-driven package management system that lets users install, uninstall, verify, query and update software. It's free software released under the GNUGPL and is a core component of many Linux distributions, including Red Hat Enterprise Linux, the Fedora Project, SUSE Linux Enterprise, openSUSE and Meego.

Version control systems, also known as revision control systems, are repositories of files for the source code of computer programs that track every change made to the source code along with who made the change, why they made it, and notes on problems fixed or enhancements introduced by the change.

Deploying a complete modeling and version control solution for systems management saves time, ensures consistency, provides a good reporting system, lets you audit your systems, reduces risks and lets you roll back a system to a previous iteration when a new rollout causes problems, Troan pointed out.

The Wellspring of System Management Woe

When they talk about versioning, many people only think about the operating system and possibly key middleware such as the database management system, Jonathan Eunice, principal IT analyst at Illuminata, told LinuxInsider.

However, there are "tens or hundreds of other system components, libraries and services with entirely different version numbers, patch levels and update cycles," Eunice said. Keeping track of and patching all of these is a "royal mess," he added.

One of the causes of this problem is configuration management itself, Eunice suggested.

"The traditional conservative datacenter outlook is, if it ain't broke, don't fix it," Eunice said. "That is, don't update or even touch any component if you don't have to because if it's working now and you touch it, even with the best of intentions, you may break it. Then, if your key apps don't work, it's on your head."

That attitude is diametrically opposed to the modern network-connected infrastructure, where everything depends on everything else, Eunice said.

"Overall, the tension between 'We have to have the latest patches installed' and 'Leave well enough alone' means that a lot of different systems are at a lot of very different patch levels," Eunice remarked.

The Roll-Your-Own Solution

Many enterprises with large-scale Linux deployments create their own solutions for system versioning and management. Three factors contribute to this: The lack of adequate tools, tradition and the need to control costs.

"The requirement for tools exists, but many organizations have gotten accustomed to rolling their own solutions," IDC's Hilwa said. That's because there were few version control and modeling tools for Unix and Linux and so enterprises with servers running these OSes wrote and used complex shell scripts.

These enterprises continue to use those scripts. Further, many businesses running massive farms of machines such as service providers and cloud infrastructure companies rely heavily on scripts written in-house, Hilwa stated.

"This is partly to do with IT culture issues -- they've always used scripts -- and partly to do with costs," Hilwa elaborated. "With system costs going down and Linux being open source, big IT houses or service providers are not accustomed to spending a lot of money on management tools."