New FreeBSD Security Officer

A few minutes ago I sent the following email, copied here in its
entirety, to the freebsd-security mailing list:

When I took over from Jacques Vidrine as
FreeBSD Security Officer in August
2005, I had three goals: Adding FreeBSD Update to the base system as an
officially supported tool; keeping the FreeBSD Update and Portsnap build
and mirror systems running smoothly; and ensuring that we maintain the
high quality of our security advisories and patches. I've definitely
achieved the first two, and while Jacques' shoes have been hard to fill,
I hope you will agree that we haven't done badly on the third.

Most of the credit for this goes to all the FreeBSD developers who have
served on the Security Team over the past 80 months: Marcus Alves
Grando, Qing Li, Xin Li, Remko Lodder, George V. Neville-Neil, Simon L.
Nielsen, Philip Paeps, Christian S.J. Peron, Tom Rhodes, Guido van
Rooij, Stanislav Sedov, Dag-Erling Smorgrav, Jacques Vidrine, Robert
Watson, Martin Wilke, and Bjoern A. Zeeb. For administrative reasons
I've often been the person committing security patches to the tree, and
I think this misleads some people into thinking that I do all of the
work; nothing could be further from the truth.

Regrettably, like most FreeBSD developers I have commitments outside of
FreeBSD, and as my online backup
service (Tarsnap) has grown, so too have its demands on my time. I
recently came to the conclusion that I could no longer give the role of
Security Officer the time it deserved — and more importantly, the
availability to handle important issues on short notice — and as
such I decided that it was time for my term as Security Officer to come
to an end. While I'll be remaining on the Security Team to offer
opinions and advice about security issues, and I'll still be managing
the FreeBSD Update and Portsnap bits, it's time to let someone else
drive the bus.

I asked the FreeBSD Core Team to offer the Security Officer role to
Simon Nielsen, and I am happy to say that they agreed and Simon accepted
the position. Simon has been a FreeBSD committer since July 2003, a
member of the FreeBSD Security Team since October 2004, and a Deputy
Security Officer — taking responsibility for pushing out security
advisories when I have been temporarily unavailable — since August
2005; I think it's safe to say that Simon is more prepared for this
position than any new FreeBSD Security Officer has ever been.

Thank you for all the support and bug reports you've provided over the
years, and please join me in welcoming Simon to his new role.

To pre-empt any slashdot trolls: FreeBSD is not dying, and I have no
intention of leaving the project. I'm handing off the Security Officer
hat to Simon because my occupation with Tarsnap (very literally —
it's my day job) means that Simon can do a better job as Security
Officer than I can right now; but I just came back from BSDCan and a
fantastic FreeBSD developer summit and I have every intention of staying
engaged with the project long into the future.

Speeding up Portsnap via geolocation

Ever since I wrote Portsnap in 2004, I've been hearing the same question
from FreeBSD users outside of North America: "Can we have some nearby
mirrors? The US mirrors are too slow!" I've helped many companies set
up Portsnap mirrors within their networks, but most of the public
Portsnap mirrors have always been in the US, for three simple reasons:
The existing mirrors were more than enough to handle the load (even when
it peaks following a "megacommit" which touches many ports);
administering multiple similar-but-not-quite-identically-configured
mirrors is a headache (and when it comes to system administration, I
optimize for lack of headaches); and despite my attempt to improve
mirror selection by providing "us.portsnap.freebsd.org",
"eu.portsnap.freebsd.org", and other similarly "local" names with
customized mirror lists, most people still used the default
"portsnap.freebsd.org" server pool. Thanks to
Amazon EC2 and
Amazon Route 53, this has now
improved.

After some preliminary testing, the first step in this process was to
take the existing portsnap.freebsd.org DNS entries, enter them into
Route 53 — using the web console, since there was no point
scripting a one-time setup like this — and add a DNS delegation
for the subdomain into the freebsd.org zone. This probably had no
user-visible impact: Although the Route 53 servers are spread around the
world and thus often faster to access than the FreeBSD.org DNS, most
systems would need to fetch FreeBSD.org DNS first anyway.

Having confirmed that everything was working properly with the
pre-existing DNS data being served out of Route 53, I launched four new
Portsnap mirrors, in the EU West (Ireland), South America East (Sao
Paulo), Asia-Pacific North-East (Tokyo), and Asia-Pacific South-East
(Singapore) regions. For these, I used my
FreeBSD 9.0-RELEASE
EC2 images — not an officially supported operating system on
EC2, but it works just fine — with lighttpd as the web server, for
ease of setup more than anything else, since serving up small static
files (which is all the Portsnap HTTP server needs to do) is not at all
taxing. These went into Route 53 as ec2-REGION.portsnap.freebsd.org. (I
didn't add any mirrors in the three US EC2 regions, since Portsnap
already had two very well connected US mirrors, but I might do this in
the future.)

Currently all four EC2 mirrors are running on 64-bit "small" instances;
based on the amount of traffic currently being served I may upgrade the
EU West mirror to "medium" and I might downgrade the Singapore and Sao
Paulo mirrors to "micro", but I'm going to wait until I see how the
mirrors respond to a "sweeping commit" load spike before I make any such
adjustments. Fortunately with EC2 it's possible to shut down an
instance and then boot it up on a different size of virtual machine
— a much faster process than if I had to start from a fresh
install and run through the entire setup process!

The final step was to set up Route 53 to use latency based routing to
point the Portsnap client code at the nearest mirror. Here I got a bit
annoyed: I had A records pointing at each mirror, and I wanted to create
lists of SRV records with different mirror priority orderings —
e.g., for users in Japan, the EC2 mirror in Tokyo is the best one to
use, but if it isn't responding to requests, the Singapore mirror should
be tried next, then the US mirrors, etc. — but I found that Route
53 didn't support latency based SRV records — only latency based A,
AAAA, CNAME, and TXT records. After I sent out an irritated tweet about
this, I was contacted by a Route 53 engineer who provided me with some
background details about this, and I'm hoping that they'll enable
latency based SRV records at some point in the future.

I did manage to work around this limitation however: I published a
single SRV list specifying "geodns-1", "geodns-2", and "geodns-3" as the
priority 1, priority 2, and priority 3 endpoints, then added latency
based A records which mapped these to the appropriate endpoints. This
is far from ideal — in addition to the error-prone nature of
entering each IP address multiple times and the annoying need to
configure mirrors to respond to multiple Host: names rather than just
their proper names, it means that Portsnap users only see "Fetching from
geodns-1.portsnap.freebsd.org" rather than being able to see which
mirror they're actually hitting — but it's enough to feed people
the bits they need for now.

One possibility I explored was to use Amazon CloudFront for distributing
the Portsnap bits; since it's all static files served over HTTP, it
seemed that it would be a good fit. Reality proved otherwise, for two
reasons: First, Portsnap is optimized to minimize bandwidth consumption
in part by fetching lots of small patches. Like Amazon S3, CloudFront
has a per-GET cost which is usually minimal, but for Portsnap's
abnormally small average object size the per-request fees would have
dwarfed the bandwidth fees — making running my own HTTP server
inside an EC2 instance the cheaper option. Second, Portsnap relies on
HTTP/1.1 pipelining to minimize the effects of network round trip time,
and in my tests CloudFront sent HTTP/1.0 responses (I even tried forcing
request pipelining on despite the response indicating a lack of
pipelining support, but CloudFront clearly didn't like this idea —
it responded to that with TCP RST packets). With CloudFront's 30
locations compared to EC2's 7 regions (8 if you count GovCloud), it
could have been ideal for minimizing latencies; but even with only EC2's
endpoint selection, Portsnap is still much better off than when users
around the world were hitting US mirrors.

If you're outside of North America and stopped using Portsnap because
you found that it was too slow, try it again! You may be pleasantly
surprised.