Constructing Red Hat Enterprise Linux v. 3

Putting together a Linux distribution gets a lot more complicated when stacks of requirements start arriving from hardware vendors and other partners.

Late-Breaking Features

Just as the sun never sets on kernel development, our near-and-dear partners and customers don't sit still either. One of the
few constants in this dynamic environment is that throughout the
course of the release development, there is a steady influx
of must-have crisis feature additions.
We endeavor to make it a trade-off and kick out a lower
priority feature in order to keep the workload sane. In the
end, however, it never seems to work out to such a sweet balance. The ability
of the team to persevere through all these demands is remarkable.

As productive as the Red Hat developers are,
some late-breaking features always have to be deferred to a
later release or update. These situations cause no end of trauma
for our partner managers who have to be the bearers of bad news.
It's improbable that our partners ever heard the saying “don't shoot
the messenger”. There was one incident when we were two hours from
shipping the release and a delivery arrived on the loading dock.
It was a new computer platform we needed in order to be able
to develop and test support for it. The partner was incredulous that
we were unable to accommodate.

Testing

Several different levels of testing are performed
throughout the development process. It all begins with the
developers performing unit testing. This consists of manual,
hands-on testing as well as development of automated testing
programs. The set of automated tests constantly is being augmented
as new problems are addressed. These automated test programs then
are incorporated into a test grid that is managed by the QA department.
In addition to the internally written unit tests, our test grid
includes a wide range of regression tests. Examples include POSIX,
LSB conformance, LTP, crashme, gcc suite and diabolical tests
provided by our partners. Stress tests also are performed for a range
of system functions using such tests as cerberus, lmbench, bonnie,
spec and other micro-benchmarks, to name a few. These automated
tests are run nightly in order to detect regressions quickly.
This is critical in order to isolate the offending code.
In contrast, if we waited to perform monthly base-level testing,
it would be much more difficult to identify the culprit.

On top of the nightly tests, more time-consuming
test scenarios are run less frequently. Examples include
installation testing using a wide range of configuration options
across the many different languages we support. Hands-on
testing rounds out the internal QA coverage. Doing justice to the
staggering range of testing done by the hardworking QA team here
substantially exceeds the scope of a single article.

After the kernels have passed internal units tests and QA scrutiny,
we make them available to our development partners. This vastly
broadens the coverage to include external QA and development teams
in other companies. Our partners focus testing on their hardware
platforms as well as the typical server capabilities their enterprise
customers demand.

The last layer of testing includes external beta testers. These
beta testers include high-profile customers, as well as the many
people in the Linux community who respond to our open invitation
to help with testing.

The combination of all these different testing activities yields an
extremely stable and well-tested product. It also points out the huge
value of the open-source model. It is the combination of testing resources
from many different companies and dedicated individuals that scales
well beyond the resources a single company could bring to bear to
tackle an infinite testing matrix.

Figure 3, composed by Nick Carr, summarizes the development model from
requirements gathering, development and testing, culminating in production.

Figure 3. The Development Process, from Requirements Gathering to Release

Conclusion

In the end, the Red Hat Enterprise Linux v. 3 distribution did ship
on schedule in mid-October 2003. Having worked for many years for a major
IHV on a large-scale operating system engineering team, I have been
exposed to both the proprietary operating system development model
and the open-source development model. The successful outcome of
the Enterprise Linux v. 3 release clearly demonstrates the power
of cooperative open-source development. The quantity of features,
rapid development time and caliber of participants, both internal
and external, is remarkable and stands as a strong testament to
the Linux community.

When the release shipped, we all were proud to have contributed to such a
tremendous accomplishment.
But the time window to
rejoice was short indeed. As soon as Enterprise Linux v. 3 finished,
it seemed we already were late with the development of
Enterprise Linux v. 4. This reminds me of the quote, “Congratulations
for winning this battle, that earns you the privilege to come back
and fight another day.” Gotta go, more battles to fight.

Tim Burke is the director of Server Development at Red Hat. This team
is responsible for the kernel and the set of core applications included
in Red Hat Enterprise Linux. Prior to becoming a manager, Tim earned
an honest living developing Linux highly available cluster solutions and
UNIX kernel technology.

Hmmm... I must have missed the part where removing support for ISA soundcards was a requested feature... They were supported in 2.1! Or adding a new, broken bindconf that still hasn't been fixed, though it's been out and broken for five months. Or even leaving out many, many *-devel rpm packages, such as sendmail-devel, which enables one to actually add and use milters with the milter-enabled base rpm.

If only the core bits got extensive testing, then that would explain why RHEL3 is worse than RHL9 from a "working-features" perspective.

Thats because the amount of possible work is almost infinite while the staff we have at RH is finite. Ultimately we do have to make some hard tradeoffs. We can't please all of the people all of the time. For example, we pour a lot of effort into new hardware support. Things like AMD64, sata, graphics cards, new storage and networking adapters, etc. So due to new stuff, the pile just gets bigger. Something has to give. This is why sometimes some things have to fall out the other end. ISA sound cards are 1 such example.

This is because the RH AS 3 is for enterprise use.
The only thing that need to work perfect is the core, and they need to work fast to compete with Solaris, Aix and HP-UX.
What is necessary is a stable and fast kernel with support for lots of mem, lots of disks, low latency, better threading to use in database and ERP softwares.
Sound cards and video cards is a secondary target.
In most part of datacenters, these machines are controled via console, and don't have video, mouse and keyboard.
In this scenario Red Hat AS 3 really do the job.

Trending Topics

Upcoming Webinar

Getting Started with DevOps - Including New Data on IT Performance from Puppet Labs 2015 State of DevOps Report

August 27, 2015
12:00 PM CDT

DevOps represents a profound change from the way most IT departments have traditionally worked: from siloed teams and high-anxiety releases to everyone collaborating on uneventful and more frequent releases of higher-quality code. It doesn't matter how large or small an organization is, or even whether it's historically slow moving or risk averse — there are ways to adopt DevOps sanely, and get measurable results in just weeks.