Red Hat Enterprise Linux 5.6 was released last week (January 2011), nearly ten
months since the release of 5.5 in March 2010. So let's use this opportunity to
take a quick look back over the vulnerabilities and security updates made in
that time, specifically for Red Hat Enterprise Linux 5 Server.

Errata count

The chart below illustrates the total number of security updates issued for Red
Hat Enterprise Linux 5 Server if you had installed 5.5, up to and including the
5.6 release, broken down by severity. It's split into two columns, one for
the packages you'd get if you did a default install, and the other if you
installed every single package (which is unlikely as it would involve a bit of
manual effort to select every one). For a given installation, the number of
package updates and vulnerabilities that affected you will depend on exactly what you
have installed or removed.

So, for a default install, from release of 5.5 up to and including
5.6, we shipped 57 advisories to address 206 vulnerabilities. 10
advisories were rated critical, 27 were important, and the remaining
20 were moderate and low.

Or, for all packages, from release of 5.5 to and including 5.6, we
shipped 80 advisories to address 300 vulnerabilities. 12 advisories
were rated critical, 34 were important, and the remaining 34 were
moderate and low.

Critical vulnerabilities

The 12 critical advisories addressed 49 critical vulnerabilities across just 3 different packages:

An update to the Exim Internet Mailer,
(December 2010),
where an unauthenticated remote attacker could run arbitrary code as root on a
server.
Exim is not a default package or enabled by default. There is a
public exploit for this issue which worked on Red Hat Enterprise Linux 5.

Updates to correct 48 out of the 49 critical vulnerabilities were
available via Red Hat Network either the same day or the next
calendar day after the issues were public. The update to fix Exim took
3 calendar days from
the date of the report
to the Exim developers.

Overall, for Red Hat Enterprise Linux 5 since release until 5.6, 97%
of critical vulnerabilities have had an update available to address
them available from the Red Hat Network either the same day or the
next calendar day after the issue was public.

Other significant vulnerabilities

Although not in the definition of critical severity, also of interest during
this period were several kernel flaws that where an local user could gain
root privileges. The following had publicly available exploits:

A fix
for CVE-2010-2240
was provided
by RHSA-2010-0661
(August 2010). The public exploit did not work against Red Hat Enterprise
Linux 5, but it may be possible to create one that does.

A fix
for CVE-2010-3904
was provided
by RHSA-2010-0792
(October 2010). The public exploit did not work against Red Hat Enterprise
Linux 5 but it is possible to create one that does.

Previous updates

To compare these statistics with previous update releases we need
to take into account that the time between each update is different.
So looking at a default installation and calculating the number of
advisories per month gives the following chart:

This data is interesting to get a feel for the risk of running Enterprise
Linux 5 Server, but isn't really useful for comparisons with other major
versions, distributions, or operating systems -- for example, a default install
of Red Hat Enterprise Linux 4AS did not include Firefox, but 5 Server does. You
can use our public
security measurement data and tools, and run your own custom metrics for any
given Red Hat product, package set, timescales, and severity range of interest.

Hold on a second. It might be important. I'd better go and read it. Oh it's just a note
confirming some meeting for next week. Deleted. Now, what was I working on?

A few years ago, when I was analysing where my time was going, (and why I was
working 60+ hour weeks), I figured out that the context switching caused by
being unable to concentrate on a task for more than a few minutes was a major
productivity drain.

It's hard to resist a new email. My new cellphone takes great delight in having
'push' email and would really like to beep on each new message I receive. The web
is full of gmail notifier applications designed specifically to interrupt you
to some important new mail. Even my favourite command-line email client, Alpine,
likes to ping you about new mail arriving in your inbox even if you're busy
in some other mailbox or composing a mail.

Alpine ought to have some sort of "don't notify me" option, but in the meantime
I apply the brute-force patch below to disable it.

This 5-minute patch has saved me several hours of task switching
every week, and although this means it can sometimes be an hour or two between
me checking my inbox, no one has really noticed.

Between releases there are lots of changes made to improve security and we've not
listed everything; just a high-level overview of the things we think are most
interesting that help mitigate security risk. We could go into much more
detail, breaking out the number of daemons covered by the SELinux default
policy, the number of binaries compiled PIE, and so on.

Note that this table is for the most common architectures, x86 and x86_64 only; other supported architectures may vary.

Starting with Red Hat Enterprise Linux 6 we have switched to using
SHA-256 signatures on all RPM packages and to a 4096-bit RSA signing
key.

We've done this because it is current best practice to migrate away from MD5
and SHA-1 hashes due to various flaws found in them. Those flaws don't yet
directly pose a threat to package signing however, and therefore our existing
shipped products which used these older hashes will continue to use their
existing keys until they reach their end of life.

A similar switch to stronger signing was
already made
in Fedora 11. This switch involved some changes to the RPM application.

So what this means is that we used new signing keys for both the beta and
final release packages for Red Hat Enterprise Linux 6. Those keys were created
and are protected by a hardware security
module, as we've done
with previous keys.

Also in the Red Hat Enterprise Linux 6 distribution we've started
to simplify the layout of the key files in
the /etc/pki/rpm-gpg/
directory:

RPM-GPG-KEY-redhat-beta : Both the old and new beta keys

RPM-GPG-KEY-redhat-release : Both the new signing key and the auxiliary key

RPM-GPG-KEY-redhat-legacy-release : The signing key used for EL5

RPM-GPG-KEY-redhat-legacy-former : The signing key used for products before EL5

RPM-GPG-KEY-redhat-legacy-rhx : The signing key used for RHX

The auxiliary key mentioned above is for emergency use. We created it some
time ago on a new standalone machine, took a hardcopy printout of the private
key and passphrase, stored them separately and securely, and destroyed the
software copies. We've planned for many eventualities, but in the unlikely
event we lose the ability to sign with the hardware key we could retrieve the
printout, type in the key, and continue to sign updates.

For our first wedding aniversary this weekend my lovely wife bought me a new gadget, an
Akai MPK-25 midi keyboard. The last Sonik gig that I played at we used
full-sized midi keyboards hooked to real synth modules, but for our next
gig later this year we want to move to lightweight with all soft-syths. Our
140bpm tracks are too hard to play completely live, so a 2-octave keyboard
is perfectly fine for playing a lead line, and the keyboard has these great
touch pads for triggering samples. We like triggering samples, see the
latest video on our facebook page.

We've been setting up our perfect performance environment on a laptop, using
Fedora 13 as the base OS, but with a real-time kernel and some prebuilt
packages from the
Planet CCRMA repository.

Tracy wasn't sure if the keyboard was going to work okay in Linux and didn't
find any useful information with Google, even looking for it's USB ID
(09e8:0072). Fortunately the Akai MPK-25 is class compliant and works perfectly
with Fedora 13 without needing to configure or install anything at all. It's
even happy to be powered from just the laptop USB port cutting down on cables
and adaptors.

When using USB, the midi in and out connectors on the
back become extra interfaces you can use too, those extra ports you
can see shown above -- so we can have another
keyboard and a sound module connected through the Akai to the laptop and save a
midi interface.

I'll cover the software we're using for our live gigs in a later article; aside
from the actual synth VST modules we use all open source.

It came as no surprise when
Microsoft admitted to quiet security patching. We knew many years
ago that they did this: not counting extra vulnerabilities that
were found internally or by researchers contracted to work for them.
For closed source, single vendor software, this isn't too big of a
deal - it's not like the user has a choice if they need to update some
application to address one critical vulnerability or 20.

When you look back, before they admitted to this practice,
Microsoft actively used vulnerability counts in reports as a tool to
discredit the security of open source distributions. Famously even
Steve Ballmer participated in counting
vulnerabilities using candy.

In other news, the Red Hat Enterprise Linux 4 risk report we
release each year has been published
(PDF). This whitepaper looks at the state of security for the
first five years of Red Hat Enterprise Linux 4 from its release on
February 15th, 2005. It includes metrics, key vulnerabilities, and the
most common ways users were affected by security issues.

"Red Hat knew about 52% of the security vulnerabilities
that we fixed in advance of them being publicly disclosed. The
average time between Red Hat knowing about an issue and it being made
public was 22 days (median 10 days).... A default installation of Red
Hat Enterprise Linux 4 AS was vulnerable to 14 critical security
issues over the entire five years. "

Red Hat Enterprise Linux 5.5 was released at the end of March 2010,
just under 7 months since the release of 5.4 in September 2009. So
let's use this opportunity to take a quick look back over the
vulnerabilities and security updates we've made in that time,
specifically for Red Hat Enterprise Linux 5 Server.

Errata count

The chart below illustrates the total number of security updates issued for Red
Hat Enterprise Linux 5 Server if you had installed 5.4, up to and including the
5.5 release, broken down by severity. I've split it into two columns, one for
the packages you'd get if you did a default install, and the other if you
installed every single package (which is unlikely as it would involve a bit of
manual effort to select every one). For a given installation, the number of
package updates and vulnerabilities that affected you will depend on exactly what you
have installed or removed.

So for a default install, from release of 5.4 up to and including
5.5, we shipped 52 advisories to address 140 vulnerabilities. 5
advisories were rated critical, 14 were important, and the remaining
33 were moderate and low.

Or, for all packages, from release of 5.4 to and including 5.5, we
shipped 75 advisories to address 187 vulnerabilities. 6 advisories
were rated critical, 18 were important, and the remaining 51 were
moderate and low.

Critical vulnerabilities

The 6 critical advisories were for 3 different packages. Given the
nature of the flaws, ExecShield protections in RHEL5 should make
exploiting the memory flaws harder.

An update to kdelibs
(November 2009),
where a malicious web site could potentially run arbitrary code as the
user running the Konqueror browser. kdelibs is not a default
installation package.

An update to krb5, the Kerberos network authentication system
(January 2010),
where a remote KDC client could cause a crash or run arbitrary code as
root. This issue only affected users that have configured and enabled
krb5.

Updates to correct 24 out of the 25 critical vulnerabilities were
available via Red Hat Network either the same day, or up to one
calendar day after the issues were public. The update to fix Konqueror took
us 4 calendar days.

Overall, for Red Hat Enterprise Linux 5 since release to date, 98%
of critical vulnerabilities have had an update available to address
them available from the Red Hat Network either the same day or the
next calendar day after the issue was public.

Other significant vulnerabilities

Red Hat Enterprise Linux since 5.2 contained backported patches
from the upstream Linux kernel to add the ability to restrict
unprivileged mapping of low memory, designed to mitigate NULL pointer
dereference flaws. In the last risk report we mentioned it was found
that this protection was not sufficient, as a system with SELinux
enabled was more permissive in allowing local users in the
unconfined_t domain to map low memory areas even if the mmap_min_addr
restriction is enabled. This is
CVE-2009-2695
and was addressed in a kernel update in November 2009.

Previous updates

To compare these statistics with previous update releases we need
to take into account that the time between each update is different.
So looking at a default installation and calculating the number of
advisories per month gives the results illustrated by the following
chart:

This data is interesting to get a feel for the risk of running
Enterprise Linux 5 Server, but isn't really useful for comparisons
with other versions, distributions, or operating systems -- for
example, a default install of Red Hat Enterprise Linux 4AS did not
include Firefox, but 5 Server does. You can use
our public
security measurement data and tools, and run your own custom
metrics for any given Red Hat product, package set, timescales, and
severity range of interest.

And not because she's set the alarm for the wrong time, or used a
'crazy frog' sound theme, but because it had a remote root exploit.
It's fixed now.

It all started when I bought her a Chumby for Christmas. A Chumby
is a little bedside device that can act as an alarm clock as well as
running flash-lite applets. What made it especially appealing is that
you can write your own applets if you want, and the whole thing is
Linux-based and designed to be hackable: they correctly abide by the
GPL and have their sources available, you can build and install your
own software, you can even enable ssh and have a remote shell if you
want to. And with NTP the clock is always at the right time, since
I really don't like having out-of-sync clocks around the house.

So it was time to connect another device to my wireless network: a
device designed to be left on and permanently connected to the
network, and having a connected microphone, in the bedroom. A quick
look around the OS and I found that it had a web server accessible by
default, and a pair of CGI scripts, written in shell script, running
as root, that didn't correctly escape their input. (Hint: writing
secure CGI scripts in shell is non-trivial).

With a bit of careful manipulation (to get around some character
handling in the code) I had a remote root shell on a default Chumby and
could stream audio from the microphone remotely. Oops. Not too big a
deal though as it's unlikely you're going to have it directly
connected to the internet, although with some social engineering, if
you know someone with a Chumby, you could do a cunning cross-site
scripting attack and get a reverse shell that way.

I contacted the Chumby folks and they dealt with this like an ideal
vendor; acknowledging the issue, keeping in contact, and doing a security update.
Good for them. I like this device and vendor so much I'm going to buy
another Chumby, and a few colleagues from work are too.

But how many other devices do we connect to our networks without
thinking about them, and how many folks outside of the security
paranoid have properly secured and segmented wireless networks? I've
got a IP wireless network CCTV camera and a VOIP phone system both
which seem to be running Linux (and both which seem to have
vulnerabilities) to worry about next although harder since both are
closed systems which haven't released their source.

During the creation and review of the list we spent some time to see how
closely last years list matched the types of flaws we deal with at Red Hat. We
first looked at all the issues that Red Hat fixed across our entire product
portfolio in the 2009 calendar year and filtered out those that had the highest
severity. All our 2009 vulnerabilities have CVSS scores, so we filtered on
those that have a CVSS base score of 7.0 or above[1].

There were 22 vulnerabilities that matched, and we mapped each one to the
most appropriate CWE. This gives us 11 flaw types which led
to the most severe flaws affecting Red Hat in 2009:

10 of the 11 CWE are mentioned in the 2010 CWE/SANS document, although 4 of them
are on "the cusp" and didn't make it into the top 25.

This quick review shows us that 2009 was the year of the kernel NULL pointer
dereference flaw, as they could allow local untrusted users to gain privileges, and
several public exploits to do just that were released. For Red Hat,
interactions with SELinux prevented them being able to be easily mitigated,
until the end of the year when
we provided updates.
Now, in 2010, the upstream Linux kernel and many vendors ship with
protections to prevent kernel NULL pointers leading to privilege escalation.
So although 2009 was the year where CWE-476 mattered to Linux administrators, it
didn't make the SANS/CWE top 25 as this flaw type should not lead to
severe issues (as long as the protections remain sufficient).

Here is a breakdown with the complete data set to show the CVSS scores and
packages affected:

[1] NIST NVD rate vulnerabilities as "High" severity if they have a CVSS base
score of 7.0-10.0. This ends up excluding flaws in web browsers such as Firefox
which can have a maximum CVSS base score of 6.8.

[2] Red Hat Enterprise Linux 4 and 5 were also affected by this vulnerability,
but with a lower CVSS base score of 4.3, due to the extra runtime pointer
checking.

There have been quite a few stories over the last couple of weeks
about the NULL character certificate flaw, such as this
one from The Register.

The stories center around how open source software such as Firefox was
able to produce updates to correct this issue just a few days after
the Blackhat conference, while Microsoft still hasn't fixed it and are
"investigating a possible vulnerability in Windows presented during
Black Hat".

But the actual timeline is missing from these stories.

The NULL character certificate flaw (CVE-2009-2408) was actually
disclosed by two researchers working independantly who both happened
to present the work at the same conference, Blackhat, in July this
year. Dan Kaminsky mentioned it as part of a series of PKI
flaws he disclosed. Marlinspike had found the same flaw, but was
able to demonstrate it in practice by managing to get a
trusted Certificate Authority to sign such a malicious certificate.

The flaw was no Blackhat surprise; Dan Kaminsky actually found this
issue many months ago and responsibly reported the issues to vendors
including Red Hat, Microsoft, and Mozilla. We found out about this
issue on 25th February 2009 and worked with Dan and some of the
upstream projects on these issues in advance, so we had plenty of time
to prepare updates and this is why we were able to have them ready to
release just after the disclosure.