Target reached!

Thanks to 148 generous donations, I have now reached my funding target for
my summer of
FreeBSD security development. Particularly worthy of note are donations
of 1000 USD and 6500 USD, sent by pil.dk
and Pair Networks respectively; but I
am grateful for all the donations I received, whether five dollars or
sixty-five hundred dollars.

For the first two weeks of May I will be finishing my
BSDCan paper and presentation, doing
some minor cleanup work on the portsnap client, and then attending the
conference itself, where I will be talking about some of the difficult
not-entirely-technical questions involved in FreeBSD security and gathering
feedback about what everybody would like to see in the next version of
FreeBSD Update.

After I return from the conference, I'll most likely head straight into
rewriting the FreeBSD Update server code, but I don't want to overly commit
myself to any particular schedule yet -- for one thing, a security issue
could arise which I might need to spend a week investigating -- but I will
be trying to keep this site updated with my progress as much as possible.

My Very Important Response

In a recent article entitled
"Colin's
Very Important Response", Thomas Ptacek responded to my last post
here; while I'm glad that he has admitted to getting some of his facts
wrong, there are still some significant errors.

First, Ptacek now claims that I posted my paper 'months after Osvik and
Tromer published what is now "Cache Attacks and Countermeasures: the
Case of AES"'. His chronology here is completely wrong: while Shamir
famously warned of unspecified dangers inherent in Hyper-Threading in
the Cryptographers' Track of RSA 2005 -- some four months after I first
discovered this problem -- the Osvik-Shamir-Tromer paper was not written
until much later: In fact, a few days after I released my paper (at
which point it had been circulating for almost three months with only
minor changes) I received an email from Tromer describing their paper as
not yet being finished. Of course I didn't cite the work of Osvik
and Tromer -- not only had I not yet seen their work, they hadn't even
finished writing it! (On the other hand, in the version of my paper
which I submitted to the Journal of Cryptology four months later -- by
which point the Osvik-Shamir-Tromer paper was published on the web -- I
do cite their work and point out the similarities.)

Ptacek goes on to list the reasons he accused me of self-promotion. To
respond briefly:

I will not back down from my argument that servers should be secure by
default, with an option available to make them insecure and possibly gain
some performance.

I never attempted to "monopolize attention", and I have always agreed
that Bernstein's attack also needed to be addressed.

I never accused anyone of secretly working for Intel; the closest I ever
came to this was a tongue-in-cheek question on a FreeBSD mailing list when
someone made (inaccurate) comments which mirrored what Intel's PR people
had been saying. However, it is true that Intel attempted to stop me from
releasing my paper.

I never picked a fight with Linus; while I did refer to him as a
"dictator", I did so in exactly the same manner as everybody else writing
in the area of comparative open source project management.

If I had been wearing only a cryptographer's hat, I would have gone ahead
and submitted my paper to J. Cryptology in March 2006. Instead, I wore
two hats: first that of a cryptographer, and second that of a member of
the FreeBSD Security Team. The policy of the FreeBSD Security Team is that
local privilege escalation and information leakage problems are resolved
via security advisories; this policy existed long before I became involved
in FreeBSD.

Next Ptacek points out that I have not provided evidence that I did not
undertake gainful employment during the period when I was working on this
issue. I'm not quite sure what evidence he wants -- my income tax return,
perhaps? Does he want a list of the companies which asked me to interview
for jobs, and the professors who invited me to apply for post-doctoral
research positions? Of course, if I were lying about this I could easily
forge such documents, so even if I were prepared to make them publicly
available -- which I'm not, for obvious reasons -- it would serve little
purpose.

After some floundering concerning side channel attacks -- cryptography
which, by his own admission, he doesn't understand -- Ptacek concludes by
stating that "any localhost kernel of privilege escalation finding Colin
published would be more impactful". In the very narrow sense of FreeBSD
security, Ptacek is quite correct here. However, unlike most local kernel
privilege escalation attacks, which except in very rare cases only affect
a single operating system, the Hyper-Threading side channel I demonstrated
affected all SMP i386 operating systems. This wide range of
affected systems makes an otherwise less significant issue worthy of more
widespread attention; but even without that, the fact that my
paper was the first publicly available work which demonstrated the
exploitability of the shared L1 cache on Hyper-Threaded processors
makes it worth noticing.

While the Blogosphere seems to have taken over from Usenet as the home
of "Everybody is entitled to their opinion, even if they're completely
wrong", I wish people would make more effort to check their facts before
criticizing other people: Incorrect facts make the person posting them
look ignorant, while incorrect criticisms tend to make both sides look
bad.

My Very Important Knob

In a recent article entitled
"Colin's
Very Important Knob", Thomas Ptacek made a series of untrue statements
about my work. In true Internet fashion, I'm taking this opportunity to
climb onto my own soapbox to defend myself.

The first inaccuracy in his article was to state that I "cribbed" my
Hyper-Threading side
channel work "directly from Dan Bernstein". In fact, I did nothing
of the sort. I first became aware of Bernstein's work in the area of
cache-related side channels when he
posted
to sci.crypt with a URL to his work on cache timing attacks against
AES -- work which, at that time, made no mention of Hyper-Threading.
In contrast, I had at that point already been investigating the question
of Hyper-Threading side channels for weeks.
Furthermore, the attack against RSA implementations on Hyper-Threading is
very different from Bernstein's attack against AES implementations: Where
Bernstein considered the inevitable effects of a cache upon the performance
of code, I considered the effects of code upon the cache and how those could
be measured.

The second accusation made by Ptacek is that I "went on a blitz of
self-promotion". My collecting of vendor responses -- something Ptacek cites
as evidence of my self-promotion --- is nothing unusual when coordinating a
security advisory affecting multiple operating systems: When CERT coordinates
advisories, they always do this. Given the number of people who emailed me
to ask if their Windows systems were affected, I wish Microsoft had also
provided me with a response.

Ptacek goes on to claim that "there are lots of easy timing side channels if
you can run code on the same host as your target". While he correctly points
out that a local attacker can often retrieve information about a passphrase
being typed by measuring inter-keystroke timings, he makes the entirely
unfounded jump from here to concluding that any information can be
stolen in the same manner. If Ptacek is really aware of "lots of easy side
channels" which can steal RSA keys, I hope he will come forward with details
-- for one thing, I'm sure the OpenSSL developers would like to know about
them!

Next, Ptacek implies that I, as a single voice, somehow tricked FreeBSD into
disabling Hyper-Threading. Perhaps Ptacek hasn't read the papers which he
himself is citing: In an updated version of his AES cache timing paper, Dan
Bernstein argues that Hyper-Threading should be disabled, while Osvik,
Tromer, and Shamir -- who constructed their Hyper-Threading attack against
AES completely independently of my work -- came to the same conclusion.
On the operating system side, SCO and several Linux distributions followed
FreeBSD's lead; from private discussions I know that some other operating
systems were planning on doing likewise, but I do not have confirmation that
they did so.

At this point, Ptacek points to a recent FreeBSD commit, wherein I changed
the default behaviour to allow shared L2 caches -- but still not shared L1
caches -- in order to avoid halving the performance of Intel Core Duo
processors. He claims that my paper 'says L2 cache attacks are "potentially
interesting"', which is simply untrue. My paper does point out that
the L2 cache provides an interesting covert channel, but at no point
do I argue that the L2 cache provides a useful side channel, or that
any cryptographic information can be stolen using it. I stand by the comment
I made in the commit message he quoted: Nobody has publicly demonstrated a
cryptographic side channel which exploits the L2 cache.

Finally, Ptacek makes one accurate comment: We need to pursue cryptographic
implementations with data-independent timing. (Ok, he's not quite accurate
-- what we really need is data-oblivious implementations, not merely
implementations with data-independent timing -- but after everything else
he has gotten wrong, let's be generous on this point.) Indeed, I have
started working on such a cryptographic library; but it takes a long time
to build such a library from the ground up, so as I pointed out a year ago,
the only approach which could be implemented quickly was to disable
Hyper-Threading.

Still Fundraising for FreeBSD security development

On April 4th, I thought that I had reached my donations target for funding
my summer of FreeBSD security development, and asked people to stop sending
further donations. Sadly, it seems that this assessment was premature, as
it relied upon two large pledges, and it now appears that one of them will
not be arriving. Fortunately, Pair Networks -- the other large donor --
has sent $6500 US, which now brings me within $2000 of my target.

If you were intending to donate when I updated my web page on April 4th
to say that I had reached my target, please do so now. I know there were
several people in this position, so I'm hoping I can reach my target in
the next week.

As before, details about the work I plan on doing, how to donate, and a
list of the donations I have received, are at my
page at FreeBSD.org.

Side channel attacks: Intel vs. AMD

About 18 months ago, I discovered a
side channel in the
cache implementations on Intel processors with Hyper-Threading. After
three months of investigation and another three months of informing and
educating vendors about this problem, FreeBSD issued Security Advisory
FreeBSD-SA-06:14.htt, and several other operating systems (SCO and
Ubuntu quite promptly, but most Linux vendors eventually) have also
taken action.

Three weeks ago, Jan Beulich discovered that the FXSAVE and FXRSTOR
instructions on AMD processors behave differently from the same
instructions on non-AMD processors: On AMD processors, the FOP, FIP,
and FDP debugging registers are normally not saved and restored. Much
like the Hyper-Threading side channel, this allows programs to spy upon
each other and determine the code path and location of data being
accessed, even though the value of the data handled is not revealed.
This issue has now been fixed in FreeBSD, NetBSD, OpenBSD, Xen, and the
Linux kernel.

While these two issues are on first glance rather similar -- they both
concern information leakage between processes, can only be exploited by
local users, utilize low-level processor mechanics, and are specific to
a particular vendor's products -- there are some important differences.
When Intel implemented Hyper-Threading in the Pentium 4, simultaneous
multi-threading was not a very well understood area, and the problems
of information leakage were little known; while it happens that Intel
processors were vulnerable, I don't think that Intel did anything wrong
in producing such processors -- they were just overtaken by developing
research. In contrast, AMD took instructions which had been added by
Intel as part of the SSE instruction set, claimed (via the CPU feature
flags) to support SSE, and then did not fully implement said
instructions in order to make their processors faster. Imagine the
outcry if Intel's famous FDIV bug had been implemented deliberately
because division was faster if you didn't insist upon getting the right
answers!

However, a far more significant difference between these two problems
is how the vendors responded. When I first reported the Hyper-Threading
problem to Intel, I had trouble finding anybody willing to talk to me
at all. Eventually, and thanks in large part to intervention from one
of Intel's customers, I was contacted by Ernie Brickell, a respected
cryptographer who now works for Intel. Sadly, this communication was
still almost exclusively one-way: Intel wanted to know what FreeBSD was
planning on doing about the problem, and wanted to know who the other
security people were with whom I was discussing the problem, but
whenever I asked what Intel was planning on doing, Ernie told me that
he 'would need to get permission from a room full of people' before he
could tell me anything. In fact, the only thing I learned from talking
to Intel was that they didn't consider such information leakage to be a
problem at all.

In contrast, AMD's response was superb. In discussions on the
vendor-sec mailing list prior to public disclosure of the issue, they
first promptly acknowledged the issue and stated when they would make
an official response available; next, when we asked to see their
response before the issue was publicly disclosed, they agreed to this;
and finally, in their response they not only accepted that the problem
existed, but they provided considerable background information and two
different ways that they problem could be fixed -- this was
particularly helpful to FreeBSD, since I wrote our patch based almost
entirely upon the recommendations provided by AMD.

I have no intention of getting involved in debates about which company
makes the better CPUs; as far as I'm concerned, they both make great
processors. As someone working in the field of computer security,
however, I can say this: There's at least one area where Intel can
definitely learn some lessons from AMD.

Portupgrade errors after upgrading FreeBSD

When I upgraded some systems
from FreeBSD 5.4 to FreeBSD 6.0 towards the end of 2005, one of the odd
problems I encountered was portupgrade repeatedly rebuilding its
database while complaining about an "Inappropriate file type or
format".

Recently I've become aware that other people are encountering the same
problem, so for the benefit of Google (and other search engines): For
some reason there was a database format change between FreeBSD 5.x and
FreeBSD 6.x. After upgrading the base system, run portupgrade -fR
portupgrade; this will rebuild portupgrade using the new database
format, and after this is finished, portupgrade will not keep on
rebuilding its database.

I hope this post helps some people; this is a very annoying problem
until you figure out how to fix it.