On Sunday 11th of November, an intrusion was detected on two
machines within the FreeBSD.org cluster. The affected machines
were taken offline for analysis. Additionally, a large portion
of the remaining infrastructure machines were also taken offline
as a precaution.

We have found no evidence of any modifications that would put
any end user at risk. However, we do urge all users to read the
report available at
http://www.freebsd.org/news/2012-compromise.html
and decide on any required actions themselves. We will continue
to update that page as further information becomes known. We do
not currently believe users have been affected given current
forensic analysis, but we will provide updated information if
this changes.

As a result of this event, a number of operational security
changes are being made at the FreeBSD Project, in order to
further improve our resilience to potential attacks. We plan,
therefore, to more rapidly deprecate a number of legacy services,
such as cvsup distribution of FreeBSD source, in favour of our
more robust Subversion, freebsd-update, and portsnap models.

Port managers and cluster administrators have completed the
restoration of binary package building in the last few weeks.
This has brought us back the continuous updates for the old-style
binary packages on the 8.x and 9.x -STABLE branches. Note that,
as beneficial consequences, Release Candidate builds for the 8.4
release cycle can now include binary packages on the install
media, and the Project was able to add the missing binary packages
retroactively for 9.1-RELEASE on i386 and amd64 platforms.

Port managers are currently working on introducing new-style (as
known as pkgng) binary packages in the coming months,
please check the
FreeBSD ports announcements list for further gradual status
updates.

Port managers have successfully restored some of the Project's
binary package building capacity. There are some issues left
still to resolve, e.g. how to publish the resulting package sets
in a secure manner or how to build packages seamlessly for 8.x and
9.x systems on a recent 10.x system that the head node
("pointyhat") is running, but we are very close to finish with the
preparations required for providing binary packages for the
upcoming 8.4 and further releases.

Redports underwent a full security audit, and as a result could
be brought back on line. This took place on the 5th February, and
since then more backend hardware has been added to bring it back
up to full strength. On 11th February, sanity checks for ports
have been turned back on, reenabling generation and update of the
INDEX files used. The portsnap(8) service has been switched from
CVS to SVN on 25th February. The binary package building
infrastructure has undergone a major security review, and as a
result many changes have been made to the code. The review
completed on the 16th February and we are now in the process of
bringing it up on new hardware. At this point, we expect new
binary packages to be available in 2-4 weeks.

With the exception of systems relating to the building and testing
of packages, all FreeBSD.org infrastructure has now been brought
back online. A full audit of the third party package build
infrastructure code ("pointyhat") and package testing infrastructure
("redports") continues, and neither system will be brought back
online until audits are complete.

As a result, FreeBSD 9.1-RELEASE will be published with only
minimal i386 and amd64 (x86_64) precompiled package sets available,
and with no packages available for other architectures. This
package set will be available on the DVD image, and are sufficient
to install either the GNOME or the KDE desktop environment. For any
other uses, or for any packages not included on the CD, either
using the most recently available -stable package collection or
compiling ports from the ports tree are recommended. Packages
for 9.1-RELEASE will be made available at a later date. Instructions
for obtaining and updating the ports tree can be found in the
FreeBSD Handbook.

Due to the legacy third-party package build controller head
nodes being offlined pending reinstall, we have been unable to
build new package sets over the last two weeks. As a result,
FreeBSD 9.1-RELEASE has been delayed as it was felt that we
should not ship the release without at least a minimal package
set available. We are now in a position where we are once
again able to build third-party packages for both of our
Tier-1 architectures (i386 and amd64), and are planning on
releasing it within the next few days with only a slightly
limited set of packages. Please note that historically we have
also provided packages on a best-effort basis for some of our
Tier-2 architectures such as sparc64, ia64 and powerpc. We are
not currently expecting to be in a position to build any Tier-2
packages before FreeBSD 9.1 ships, so initially no precompiled
packages will be available for these platforms. We
may be in a position to provide some packages for these
architectures shortly after the release.

A few reports covering this incident on external tech news
websites have confused details relating to how this incident
was discovered. Over the last few weeks, many of our primary
cluster servers have been either physically relocated and/or
replaced with new hardware as part of work planned several
months in advance. The discovery of this incident was
unrelated to this ongoing cluster maintenance. Several
service outages in the days surrounding the incident were
correctly attributed to ongoing cluster work, and were not
related in any way to the compromise. In parallel with the
physical upgrades and relocation of servers, we are also
reworking the network layout in order to provide better
functionality, security, resilience, and to reduce any impact
from incidents such as this. Due in a large part to the
progress already made here, we were able to have full
confidence in many systems and services so quickly after the
compromised hosts on the legacy network segment were
discovered.

Although not mentioned in the original report,
CTM (another mechanism for
retrieving FreeBSD source) uses the master trusted Subversion
repository as the source of its data. Additionally, verification of
CTM-sourced trees has been completed against the Subversion tree,
confirming that there are no differences between the two. Our
experimental Git repository has been similarly verified.

Work continues on rebuilding internal infrastructure and reinstating
services taken down during the incident. Web interfaces to the old
CVS repositories (CVSweb), and to GNATS (our bug-tracking database)
have been restored amongst other services, and other internal hosts
are being examined and rebuilt where necessary. A full audit of the
package building infrastructure is ongoing.

The FreeBSD Project is investing significant effort into looking
into both medium and long term infrastructure improvements to increase
security of the FreeBSD cluster.

Newer portsnap(8) snapshots are once again available. The
generation of these had been suspended as part of the infrastructure
lockdown, however all machines involved have either been audited or
reinstalled and so we are now confident that these can be made
available once more.

The Subversion to CVS exporter is now up and running again.
Updates made to the Subversion repository will once again appear in
repositories available via csup/CVSup. Please note that the use of
these exports are still deprecated, and users are urged to move to
one of the supported methods (for example, freebsd-update(8),
portsnap(8), or Subversion) in order to obtain updates. Note also
that we are still currently unable to guarantee the integrity of
past history within the CVS repository, but are confident in the
integrity of checkouts from the top-of-tree of each branch.

Please note that due to infrastructure changes, the first update
through either portsnap(8) or csup(1) is likely to show changes to
a large number of files. This is nothing to worry about.

As mentioned in the original announcement, a package set uploaded in
preparation for the upcoming FreeBSD 9.1-RELEASE could not be verified,
and so was removed. In order to allow system integrators and end
users to verify that packages they may have downloaded are not from
this set, we have provided files containing both
sha256 and
md5 checksums
for all removed packages.

Initial details

On Sunday 11th November 2012, two machines within the FreeBSD.org
infrastructure were found to have been compromised. These machines
were head nodes for the legacy third-party package building
infrastructure. It is believed that the compromise may have occurred
as early as the 19th September 2012.

The compromise is believed to have occurred due to the leak of an
SSH key from a developer who legitimately had access to the machines
in question, and was not due to any vulnerability or code exploit
within FreeBSD.

To understand the impact of this compromise, you must first
understand that the FreeBSD operating system is divided into two
parts: the "base" maintained by the FreeBSD community, and a large
collection of third-party "packages" distributed by the Project.
The kernel, system libraries, compiler, core command-line tools
(e.g., SSH client), and daemons (e.g., sshd(8)) are all in the
"base". Most information in this advisory refers only to
third-party packages distributed by the Project.

No part of the base FreeBSD system has been put at risk. At no
point has the intruder modified any part of the FreeBSD base system
software in any way. However, the attacker had access sufficient
to potentially allow the compromise of third-party packages. No
evidence of this has been found during in-depth analysis, however
the FreeBSD Project is taking an extremely conservative view on this
and is working on the assumption that third-party packages generated
and distributed within a specific window could theoretically have
been modified.

If you are running a system that has had no third-party packages
installed or updated on it between the 19th September and 11th
November 2012, you have no reason to worry.

The Source, Ports and Documentation Subversion repositories have been
audited, and we are confident that no changes have been made to them.
Any users relying on them for updates have no reason to worry.

We have verified the state of FreeBSD packages and releases currently
available on ftp.FreeBSD.org. All package sets for existing versions
of FreeBSD and all available releases have been validated and we can
confirm that the currently available packages and releases have not
been modified in any way.

A package set for the upcoming FreeBSD 9.1-RELEASE had been uploaded
to the FTP distribution sites in preparation for 9.1-RELEASE. We are
unable to verify the integrity of this package set, and therefore it
has been removed and will be rebuilt. Please note that as these
packages were for a future release, the standard pkg_add -r
tools to install packages could not have downloaded these packages
unless they were requested explicitly.

We unfortunately cannot guarantee the integrity of any packages
available for installation between 19th September 2012 and 11th
November 2012, or of any ports compiled from trees obtained via any
means other than through svn.freebsd.org or one of its mirrors.
Although we have no evidence to suggest any tampering took place
and believe such interference is unlikely, we have to recommend you
consider reinstalling any machine from scratch, using trusted
sources.

We can confirm that the freebsd-update(8) binary upgrade mechanism is
unaffected, as it uses an entirely separate infrastructure. We have
also verified that the most recently-available portsnap(8) snapshot
matches the ports Subversion repository, and so can be fully trusted.
Please note that as a precaution, newer portsnap(8) snapshots are
currently not being generated.

If you use the already-deprecated cvsup/csup distribution
mechanisms, you should stop now.

If you were using cvsup/csup for ports, you should switch to
portsnap(8) right away. Ports developers should be using
Subversion already. Further information on preferred mechanisms
for obtaining and updating the ports tree can be found at
http://www.freebsd.org/doc/handbook/ports-using.html

If you use portsnap(8), you should portsnap fetch &&
portsnap extract to the most recent snapshot. The most recent
portsnap(8) snapshot has been verified to exactly match the audited
Subversion repository. Please note that as a precaution, portsnap(8)
updates have been suspended temporarily.

Follow best practice security policies to determine how your
organization may be affected.

Conduct an audit of your system that uses FreeBSD.org provided
binary packages. Anything that may have been installed during the
affected period should be considered suspect. Although we have no
evidence of any tampering of any packages, you may wish to consider
rebuilding any affected machine from scratch, or if that is not
possible, rebuild your ports/packages.