Canonical Voices

What Ryan Finnie talks about

2ping 3.0.0 has been released. It is a total rewrite, with the following features:

Total rewrite from Perl to Python.

Multiple hostnames/addresses may be specified in client mode, and will be pinged in parallel.

Improved IPv6 support:

In most cases, specifying -4 or -6 is unnecessary. You should be able to specify IPv4 and/or IPv6 addresses and it will "just work".

IPv6 addresses may be specified without needing to add -6.

If a hostname is given in client mode and the hostname provides both AAAA and A records, the AAAA record will be chosen. This can be forced to one or another with -4 or -6.

If a hostname is given in listener mode with -I, it will be resolved to addresses to bind as. If the hostname provides both AAAA and A records, they will both be bound. Again, -4 or -6 can be used to restrict the bind.

IPv6 scope IDs (e.g. fe80::213:3bff:fe0e:8c08%eth0) may be used as bind addresses or destinations.

Better Windows compatibility.

ping(8)-compatible superuser restrictions (e.g. flood ping) have been removed, as 2ping is a scripted program using unprivileged sockets, and restrictions would be trivial to bypass. Also, the concept of a "superuser" is rather muddied these days.

Better timing support, preferring high-resolution monotonic clocks whenever possible instead of gettimeofday(). On Windows and OS X, monotonic clocks should always be available. On other Unix platforms, monotonic clocks should be available when using Python 2.7

Long option names for ping(8)-compatible options (e.g. adaptive mode can be called as --adaptive in addition to -A). See 2ping --help for a full option list.

Because of the IPv6 improvements, there is a small breaking functionality change. Previously, to listen on both IPv4 and IPv6 addresses, you needed to specify -6, e.g. 2ping --listen -6 -I 127.0.0.1 -I ::1. Now that -6 restricts binds to IPv6 addresses, that invocation will just listen on ::1. Simply remove -6 to listen on both IPv4 and IPv6 addresses.

This is a total rewrite in Python, and the original Perl code was not used as a basis, instead writing the new version from the 2ping protocol specification. (The original Perl version was a bit of a mess, and I didn't want to pick up any of its habits.) As a result of rewriting from the specification, I discovered the Perl version's implementation of the checksum algorithm was not even close to the specification (and when it comes to checksums, "almost" is the same as "not even close"). As the Perl version is the only known 2ping implementation in the wild which computes/verifies checksums, I made a decision to amend the specification with the "incorrect" algorithm described in pseudocode. The Python version's checksum algorithm matches this in order to maintain backwards compatibility.

One of the nice things about the Raspberry Pi 2 is it has a Cortex-A7-based ARMv7 CPU, as opposed to the original Pi's ARMv6 CPU. This not only allows many more distributions to run on it (as most armhf distributions are compiled to ARMv7 minimum), but also brings with it the performance benefits associated with userland ARMv7 code. After releasing an Ubuntu 14.04 (trusty) image for the Raspberry Pi 2, I decided to pit Raspbian (which uses an ARMv6 userland for compatibility between the original Pi and the Pi 2) against Ubuntu (which is only compiled to ARMv7). I also benchmarked a Utilite Pro, an ARM system with a faster CPU and built-in SSD, and a modern Intel server.

Raspbian wheezy was tested on both Raspberry Pi models, while Ubuntu trusty was also tested on the Raspberry Pi 2, along with the rest of the systems. All installations were current as of today. The systems were tested with nbench (BYTEmark), OpenSSL and Bonnie++.

Results

This is a hand-picked assortment of test results; for the full raw results, see below.

Test

RPi BRaspbian

RPi 2Raspbian

RPi 2Ubuntu

UtiliteUbuntu

i5-4690KUbuntu

Numeric sort

217.2

450.72

421.55

334.63

2,385.1

FP emulation

41.334

70.276

55.108

52.454

795.9

IDEA

694.72

1,308.5

1,573.3

1,315

15,059

md5 1024

37,008.46

62,628.86

69,563.39

80,632.53

670,637.40

aes-256 cbc 1024

11,969.50

18,445.31

17,295.36

20,986.47

124,509.53

sha512 1024

8,491.32

11,838.81

20,718.25

25,803.70

431,647.74

whirlpool 1024

1,584.61

2,949.80

2,747.05

2,687.46

135,009.28

rsa 1024 verify

1,540.3

2,649.6

2,630.5

2,890.8

114,074.5

ecdsa 256 verify

73.2

126.3

138.0

161.1

4,329.6

Block output

7,520

11,028

11,299

48,214

62,762

Block input

13,233

23,015

22,997

125,954

284,914

Random seeks

524.7

1,054

874.6

3,218

444.5

Notes

Interestingly, many of the BYTEmark tests on the Pi 2 were faster on Raspbian than on Ubuntu. But keep in mind that these are tests from the 1990s, and are not taking advantage of modern optimizations (like the floating point emulation test). Many OpenSSL tests performed better on Ubuntu, but not all.

Edit: The slower nbench results in Ubuntu appear to be due to a running LSM (Linux Security Module). When Ubuntu is running with AppArmor (default) or SELinux enabled, it's marginally slower than Raspbian, but with LSMs disabled, it's marginally faster than Raspbian. (The Raspbian kernel has no LSM modules compiled in.) I'm keeping these test results as they are because AppArmor is enabled by default, but keep that in mind.

Raspbian/Ubuntu aside, virtually all of the tests were faster on the Pi 2 than the original Pi.

Bonnie++ tests were roughly the same between Raspbian and Ubuntu on Pi 2, and were decently faster than the original Pi (though in this test an older SDHC card was used for the original Pi, so it's not apples to apples). The SSD on the Utilite blows them away though.

All of the CPU tests are single-threaded, and do not take multi-core performance into consideration.

This was not a controlled scientific test. I did not run multiple tests on each system and average them together, and in the Intel system's case, it was an active (but low volume) server.

All Bonnie++ tests were run with swap disabled and on the boot drive, except for the Intel system where the boot drive (an SSD) did not have enough space for a full test. (Bonnie++ requires twice the amount of RAM as disk to run. On 512 MiB / 1 GiB / 2 GiB systems that's fine, but I didn't have 64 GiB free on the the Intel system's boot drive.

I've closed comments on this blog post. If you are looking for help, please see this post on the raspberrypi.org forums. If you post there, you'll be reaching a wider audience of people (including myself) who can help you. Thanks for all of your comments!

After my last post, I went and ported Sjoerd's Raspberry Pi 2 Debian kernel patchset to Ubuntu's kernel package base (specifically 3.18.0-14.15). The result is an RPi2-compatible 3.18.7-based kernel which not only installs in Ubuntu, but has all the Ubuntu bells and whistles. I also re-ported flash-kernel based on Ubuntu's package, recompiled raspberrypi-firmware-nokernel, created a linux-meta-rpi2 package, and put it all in a PPA.

With that all done, I decided to go ahead and produce a base Ubuntu trusty image. It's 1.75GB uncompressed so you can put it on a 2GB or larger MicroSD card, and includes a full ubuntu-standard setup. Also included in the zip is a .bmap file; if you are writing the image in Linux you can use bmap-tools package to write only the non-zero bytes, saving some time. Otherwise it's the same procedure as other Raspberry Pi images.

(PS: If this image becomes popular, I should point out ahead of time: This is an unofficial image and is in no way endorsed by my employer, who happens to be the company who produces Ubuntu. This is a purely personal undertaking.)

My Raspberry Pi 2 arrived yesterday, and I started playing with it today. Unlike the original Raspberry Pi which had an ARMv6 CPU, the Raspberry Pi 2 uses a Broadcom BCM2836 (ARMv7) CPU, which allows for binary compatibility with many distributions' armhf ports. However, it's still early early in the game, and since ARM systems have little standardization, there isn't much available yet. Raspbian works, but its userland still uses ARMv6-optimized binaries. Ubuntu has an early beta of Ubuntu Snappy, but Snappy is a much different environment than "regular" Ubuntu.

I found this post by Sjoerd Simons detailing getting Debian testing (jessie) on the Pi 2, and he did a good job of putting together the needed software, which I used to get a clean working install of Ubuntu trusty on my Pi 2. This is meant as a rough guide, mostly from memory -- I'll let better people eventually take care of producing a user-friendly system image. This procedure should work for trusty, utopic, and vivid, and might work for earlier distributions.
1) Install and boot Raspbian on one MicroSD card, and get the system access to a second MicroSD card, e.g. via a USB adapter.

2) Partition and format the card. #1 must be a vfat partition (64MB should be fine), and for the rest of this post I'm going to assume #2 is a swap partition and #3 is the rest of the card as ext4 for the root partition. Mount the root partition at e.g. /mnt/ubuntu and the vfat partition at e.g. /mnt/ubuntu-boot

6) Install all the necessary packages in Sjoerd's repository. You can either add the apt repository, or just do as I did and download/dpkg install them manually. However, there's a problem. These are built for jessie, and the linux-image package requires initramfs-tools 0.110 or greater, which is not yet available in Ubuntu. But since the kernel we're using doesn't actually need an initramfs, I've put together an "initramfs-tools 0.110~fake1" package which fakes it. Simply install that package ahead of time.

7) Outside the chroot, copy /boot/config.txt from the Raspbian installation to /mnt/ubuntu/boot/firmware/config.txt. Then create /mnt/ubuntu/boot/firmware/cmdline.txt with the following:

9) Umount everything, shut down the system, swap the new card into place, and boot. Once you are successfully booted into the Ubuntu system, I'd recommend installing the ubuntu-standard metapackage to get everything not pulled in by the base debootstrap.

A year ago, I launched M29, a URL shortener with a twist. Apparently I forgot to announce it here. Whoops.

Normal URL shorteners are fairly simple. You submit a long URL. The service generates a short URL. The long and short URL are placed in a backend database. If you go to the short URL, it redirects to the long URL.

This means that the URL shortener service has a large database of URLs available to it. While 99% of the contents of this database may be mundane, it's still a large, centralized source of information. Very relevant to the recent NSA news, for example.

M29's twist is, except when serving the redirect, it does not know anything about the contents of the long URLs. This is accomplished by generating an AES-128 key, using it to encrypt the long URL, and then splitting the key in two. One half of the key is stored in the backend service, and the other half is encoded as part of the short URL itself. This means the only time the two parts of the key come together is when the short URL is requested for the redirect.

Getting from a long URL to a short URL can be done one of several ways. If you go to m29.us and have Javascript enabled, the client side actually loads an AES libary, builds the key, encrypts the URL, and sends the encrypted URL and half of the key to the server, all processed on the client side. If you don't have Javascript enabled, this task is farmed out to the server side, which generates a random key, encrypts, makes the database insert, returns the short URL, then throws away half of the key. M29 also has a featureful API which lets you do these tasks yourself. (It is also compatible with the goo.gl API, which is easy to work with and has several tools available.)

The net effect is, while I currently have a database of about 10,000 entries, I cannot read them. Source IP and URI logging are not done on the server, so the only way I can find a long URL is if I load a full short URL, which is not possible given just the backend database.

Anyway, this weekend I did some work on M29, including adding a bit.ly-style preview option (append a "+" to the short URL to get its info), among other small feature additions and fixes. It was then I realized, by going to that above short URL (the first URL generated and used in documentation) that the one-year anniversary of the service is today.

Turns out there is a bug in Steam for these platforms, where it tries to launch the Windows version of Half-Life for GoldSrc mods from within Steam. However, Half-Life can be manually launched and pointed at the mod.

I have released a new version of SteamLink as a zip file. If you would like to run Half-Life: Uplink on Linux or OS X, simply download and extract the zip, and run the installer shell script. It will determine the Half-Life installation directory, install the mod, and give you a symlink to a script to launch it.

I'm working on re-doing my home router / VM server to provide better IO. The goal is to have an SSD as the boot drive, and 4 2TB disks in RAID 10 (4TB usable total) for the VMs. I'll be using md RAID for building it, however I want to be particular about where the drives are physically, for a few reasons:

The RAID drives are all SATA 6Gbps drives, but the server's motherboard only has 2 SATA 6Gbps ports. I've got a 2-port PCIe SATA 6Gbps controller on the way[0].

I want to place the two drives in each base RAID 1 set on different controllers: one on the motherboard controller, one on the PCIe card controller. This may provide extra performance, but more importantly, having each disk on a different controller protects the array as a whole in case of a controller failure.

Linux md RAID 10 is a relatively new mode. In the past, you would manually create multiple RAID 1 arrays, then combine them in RAID 0, thereby knowing where the disks were placed, since you were doing it yourself. The mdadm RAID 10 method is easier, but I found there is literally no documentation on what drives it uses for the underlying RAID 1 arrays. Using loopback devices and some trial and error, I figured out how the arrays are assembled.

In a nutshell, the underlying RAID 1 arrays are paired two at a time, in order given during creation. If you were to do this:

So be careful in this situation, if you care about where the devices are physically laid out.

[0] Half of the order actually arrived Friday, including the SSD and a 4-port PCIe 4x SATA 6Gbps controller. The idea was to place two of the RAID drives on the motherboard SATA 6Gbps controller, and two on the new controller, plus the boot SSD (which is also SATA 6Gbps). My system has 3 PCIe 1x ports and a PCIe 16x port. The 4x card was supposed to go in the 16x port, but I learned after failure that many motherboards do not like non-video cards in the primary 16x port. Oh well. The boot SSD will now go on one of the motherboard SATA 3Gbps ports, and I've got an order in for a 2-port PCIe 1x SATA 6Gbps controller.

I'm not a programmer, honest, but lately I've had this annoying habit of releasing code. But instead of my normal rambling posts about new announcements, I'll try to keep it brief.

apache-vsl has been one of my "90/90" projects (90% done, just need to finish the remaining 90%) for many years; I believe I started it around 2006, and I finally got the urge to finish and release it to the public. It's an Apache logging daemon (managed by Apache itself), capable of intelligently handling multiple VirtualHosts while being as efficient and extensible as possible. It's specifically built for two use cases -- servers which have a large number of VirtualHosts (hundreds or more), and VirtualHosts which receive a lot of traffic -- but it's easy enough to manage that it's useful with any Apache installation.

You can define time-based log files (monthly, daily, hourly... it's all just strftime), and apache-vsl handles when to log to a new file according to your definition. It manages symlinks to both the current and previous logfiles (for example, access_log.2012-08 -> access_log and access_log.2012-07 -> access_log.old), and has support for program triggers when a rotation happens (say, for running Webalizer against the old logfile, or compressing it).

apache-vsl manages its open files efficiently, so for a server with many VirtualHosts (not all of which may be accessed very often), all of the VirtualHosts' log files won't be open at once, but for VirtualHosts which are accessed frequently, the log filehandle will not constantly be opening and closing.

apache-vsl shares many features with cronolog, but while cronolog is designed more for splitting logfiles after the fact (if you were to use it directly from Apache, you'd need to have Apache open a dedicated pipe for every VirtualHost), apache-vsl is designed to communicate with Apache with a single pipe for multiple VirtualHosts.

A lot of good documentation is available in the manpage, so if you're interested, I suggest you start there.

My Raspberry Pi, the $35 barebones computer, arrived last week. I bought it for three primary reasons:

It was cheap.

It was interesting.

It was cheap.

I ordered early on a Thursday morning, expecting 3-4 weeks before it shipped per Element14's estimates. So I was quite surprised when I got a shipment notification later in the day. I'm guessing Element14 had a batch in, and was processing the backorders from the last few weeks, and somehow my order got mixed into that. So hey, yay instant gratification!

Shortly after that, Adafruit's clear acrylic case became available again, so I ordered that as well. I was expecting a tighter fit, but instead it bounces around a bit within the case. It's still sturdy enough to do its job, and looks nice, though the top panel no longer has the etched Raspberry Pi logo like I've seen on previous photos.

I also ordered some cables and a powered USB hub from Monoprice (gotta love $5 next day shipping to California/Nevada), and a high power iPad USB charger from OWC, but that turned out to be unnecessary. The 7-port Monoprice USB hub seems to provide enough power to power the Raspberry Pi itself, along with some accessories.

In the above setup, I've got it powering the Raspberry Pi (with HDMI output, plus 16GB SDHC card), a keyboard and mouse, a 4GB USB thumb drive, and my Defcon 20 badge. Not only does the Raspberry Pi boot, but I gave it a stress test for a few hours: Running a Quake 3 Arena 1920x1080 demo on loop (GPU), while simultaneously running Bonnie++ on the USB thumb drive (CPU and IO). It successfully did that for the entire test, so I'm confident in the power performance of the USB hub.

Now, I have no real plans for it; it'll probably sit, powered on in the corner of my office, for if I need to test something on an ARM system. But I do like the concept of its primary purpose. My first computer was a Vic 20 (predecessor to the Commodore 64), and then a Commodore 128 (backwards-compatible successor to the Commodore 64). I have fond memories of those computers, and they helped shape the course of the rest of my life. I like how the Raspberry Pi has both HDMI and Composite output, meaning it supports any TV or monitor made in the last 5 years or so, plus at least the previous 50 years of TVs. (Remember RF adapters?) This really lowers the barrier of computer use for your average child.

However, there is one problem with the Raspberry Pi I really disagree with. Namely, it requires an operating system. A 440MB operating system, as well as the means to get it onto an SD card. An operating system that could easily be destroyed by curious fingers. This could be a huge barrier.

One of the things about my Vic 20 and Commodore 128 was I didn't have any cartridges or pre-bought software. Just the computer, a tape drive, and some CR-90 tapes. All value I got out of them was either by typing in BASIC programs from BYTE or Commodore magazines, or creating programs myself.

The Raspberry Pi really needs an onboard ROM with some sort of simple language on it, such as BBC BASIC, and the ability to save directly to SD cards. Upon boot, if the SD card contains a bootloader pointing to a full operating system, boot that, but if not, load the BASIC ROM.

You may remember about a year ago when I released twuewand, a TrueRand implementation. TrueRand is a hardware entropy generation technique, implemented in software. In a nutshell, it works by setting an alarm for a few milliseconds in the future, and flipping a bit until the alarm is reached. It works due to the fact that time (your computer's RTC) and work (your computer's CPU) are not linked, so the result when the alarm comes due is unpredictable.

TrueRand was invented in 1995, and had mostly been forgotten for the last decade, until I started doing research on it last year. So it was quite a surprise when I was at Dan Kaminsky's talk at DEFCON a few weeks ago, and one of the topics he brought up was TrueRand. (Go check out his presentation slides; I just want to point out that while I'll be focusing on entropy and debiasing here, he goes into a lot of other interesting topics.)

Dan came to roughly the same conclusion as I did, that entropy sources have gotten worse over time, not better, and systems like VMs are almost completely devoid of entropy. Even more worrying, a paper published this year came to the conclusion that approximately 1 out of every 200 public keys on the Internet are easily breakable, not due to weaknesses in the encryption, but by bad entropy being used when generating the keypair. TrueRand may have been forgotten, but it's needed today more than ever. Dan and I talked for awhile after his talk, and went over a few things by email in the week following. twuewand 2.0's new features are influenced by those discussions.

Dan proposed a number of enhancements for TrueRand, mostly centered around other ideas for measuring variances given only a CPU and RTC, but what caught my eye was his idea of enhancing debiasing.

Many forms of random data are random in the technical sense, but are prone to bias. As an theoretical example, take a gun placed in a sturdy mount and pointed at a target not too far away. Most of the time, shots from it will hit the same spot every time (0), but occasionally they won't (1). So you're left something like 00000001000001001100000010000010; mostly hits, but with random misses. So it's random in a technical sense, but the distribution is heavily weighted toward one side.

The simplest method of debiasing is known as Von Neumann debiasing. Bits are processed as pairs, and any pair that is both 0 or both 1 is simply thrown out. Out of the pairs that are left, {0,1} becomes 0 and {1,0} becomes 1. So in the example above, the Von Neumann debiased output would be 00101. The data is now distributed better, but as you can tell, a lot was lost in the process. This is an extreme example since the data was heavily biased to begin with, but in data without a lot of bias, you still lose at least 50% of the bits (I've found 70-75% in real-world twuewand usage).

Dan thought, "Hmm, that's an awful lot of data simply being thrown out. We can't use the discarded data in the direct output, but perhaps we can use it to better (de-)influence the final output." He came up with a method he called modified Von Neumann, which I refer to in twuewand as Kaminsky debiasing.

The incoming bit stream is still run through Von Neumann, and put into an output buffer. However, all bits (whether they pass Von Neumann or not) are fed to a SHA256 stream. Occasionally (after the input stream is finished, or a sufficient number of bytes are put into the output buffer), the SHA256 hash is computed[1], and used as a 256-bit key for AES-256-CBC encrypting the output buffer. This way, only the bits which pass Von Neumann influence the output directly, but all bits help indirectly influence the output as well.

So twuewand now supports Kaminsky debiasing, and will use it by default if Digest::SHA and Crypt::Rijndael are installed.

Now, I want to clear up a mistake I made in my last post. I said that feeding twuewand output to /dev/urandom on Linux systems influences the primary pool, increasing entropy. First, you can actually write to either /dev/random or /dev/urandom, the effect is the same. But more importantly, entropy is NOT increased by writing to /dev/[u]random. It's merely "stirring the pot". If your system is out of entropy and you are blocking on /dev/random, no amount of writing to /dev/[u]random will unblock. (Directly, that is. If you're banging on your local keyboard to do this, you're slowly increasing entropy, but you could be doing the same thing in a text editor, or to /dev/null.)

Unfortunately, there is no way to increase entropy in the primary pool via standard system command line tools or character devices. However, there is a Linux ioctl, RNDADDENTROPY, which does this. So I wrote a small C wrapper, which takes STDIN and feeds it to the ioctl. This requires root of course. The utility is called, boringly enough, rndaddentropy, and is distributed with the twuewand tarball. It will be built by `make` on Linux systems.

I must point out that this utility gives you a very excellent method to shoot yourself in the foot. The lack of command line tools to directly access the primary buffer is most likely by design, since this bypasses most of the in-kernel filters for random input. Injecting, say, the contents of /bin/ls to the primary pool would be a great way to become one of those 1 in 200 statistics. Only use this utility to feed high quality entropy (such as by twuewand, or something like an entropy key).

Dan Kaminsky will be publishing software in the future called DakaRand, which does much of what twuewand currently does, but incorporates some of his other ideas. He provided me a work-in-progress copy, which looks very interesting, but it is not available to the public yet for a number of reasons. Be on the lookout for that when it is released.

[1] In Dan's proposal, after the SHA256 hash is computed, it would then be run through Scrypt's hashing algorithm. This is not done in twuewand for two reasons. First, Crypt::Scrypt does not currently provide a low-level method to just do hashing; instead it wants to create a full digest which is unsuitable for this purpose. Second, Dan has been debating whether this step is necessary or desirable at all, and Scrypt has "undefined effects on embedded hardware".

[ This was written at 2AM, and I tended to ramble. Hopefully I'll come back tomorrow and clean it up, add links to RFCs and Wikipedia entries where relevant, etc. ]

I've been explaining this a number of times in the last few days, so I figured I'd get this down in blog form to avoid repeating myself (even more). IPv6 is much like IPv4 in most respects, but its autoconfiguration can seem rather alien to people already familiar with IPv4 (that is, DHCP).

On an IPv6 network, the default router runs a service called the Router Advertisement daemon. Its primary purpose is to send out packets saying "hey, I'm the default router for the 2001:db8:1234:abcd::/64 network".

It actually doesn't matter what the router's IPv6 address actually is on 2001:db8:1234:abcd::/64; indeed, it (usually) doesn't even tell your what it is. Instead, it sends out RAs using its "link-local" fe80:: address. Link-local addresses are specific to the LAN segment you're on, and every device is required to have one (derived from the MAC address). It's sort of a middleware layer of doing things, between Layer 2 and Layer 3. But the important thing is you can talk to other devices if you know their link-local fe80:: address (and you're both on the same segment), and you can route to them.

So it really doesn't matter if the router's address is 2001:db8:1234:abcd::1/64 or 2001:db8:1234:abcd::dead:beef/64 or if it doesn't even have an address; if you're doing autoconfiguration, your OS will most likely be routing to it based on its link-local address. For example, my home router's link-local address on the LAN side is fe80::6a05:caff:fe02:4e4d/64, which it uses to send RAs.

By default the router sends an RA every minute or so, so the client device could theoretically set up autoconfiguration without ever talking to the router. But it will also respond to Router Solicitation requests being broadcast to the network. These requests are responded to (via broadcast) with the exact same RA packet it would have sent on a timer, except this time it happened on-demand.

Now, the RA lets the client device know what the LAN's IPv6 network space is. It also sends a series of flags which can make autoconfiguration possible.

The most relevant is the Autonomous address-configuration ("A") flag. This flag provides stateless autoconfiguration, and essentially says "you are permitted to construct your own IP address in the network I mentioned". The client will then use the MAC address of the interface as a basis for constructing an IP address. In the above example, 2001:db8:1234:abcd::/64 was the network. Say my interface's MAC address is 00:19:d2:43:c0:c4. Using the recommended MAC-munging method, this becomes 2001:db8:1234:abcd:219:d2ff:fe43:c0c4/64.

The client then uses Duplicate Address Detection (DAD) to make sure nobody on the network is already using this address. (DAD is also used to make sure the link-local address mentioned above is also unique to the network segment.)

So that's it! The client now has enough information to construct IP information, and in theory it never even had to send a packet. (I say "in theory" because DAD requires sending out a multicast packet. Plus, most OSes send out a Router Solicitation to trigger the Router Advertisement.) In this example, the router, fe80::6a05:caff:fe02:4e4d, sent out an RA stating the network is 2001:db8:1234:abcd::/64, and the "A" flag was set. The client used this information to construct the IP 2001:db8:1234:abcd:219:d2ff:fe43:c0c4/64, which it gave to itself, with the default route being fe80::6a05:caff:fe02:4e4d.

Now what about DNS? In a dual-stack environment, the client probably already has an IPv4 DNS address by now, and can certainly use that to look up IPv6 (AAAA) records. But for pure IPv6, there are two methods for getting DNS servers to the client.

First up is RDNSS, a relatively recent extension to the RA standard. It's simply an extension which gives one or more DNS servers in the RA packet. Simple, but not often used today.

The RA also has a "Managed address configuration" ("M") flag. This simply means, "there is a DHCPv6 server on this network. You may go look for it for more granular control." Also known as stateful autoconfiguration.

DHCPv6 is similar to DHCPv4, but has a number of differences. I'm not going to go into it in detail, but you can do pretty much everything you can do with DHCPv4: pooled address allocations, individual address allocation, DNS servers, domain search lists, etc. It's worth noting that if you have both the "A" and "M" flags in the RA set (and you're running a DHCPv6 server), the client will use both of these methods. That is to say, it'll construct an IP address using stateless means, then request an IP address from the DHCPv6 server, and will bind both of these addresses to the interface.

Side note: IPv6 has no concept of network or broadcast addresses. In this post's example, 2001:db8:1234:abcd::/64 was the network, but 2001:db8:1234:abcd:: can also be a device. However, most networks I've seen continue the old tradition of giving the .1 (or in IPv6's case, :1) address to the default router.

Last year I bought a Netgear N750DB router as an access point to replace my Linksys WRT54GL. Turns out the N750DB will send multicast traffic from a wireless device back to itself. This breaks IPv6 Duplicate Address Detection (DAD), and I was also seeing problems on my Ubuntu laptop. So I went back to the WRT54GL, and while stateless autconfiguration started working on the laptop again, it was still not working on the iPhone and iPad.

The second part was the iOS devices needed to be rebooted. No idea why, but after a hard reboot, they started pulling RA (and DHCPv6, which I had configured in the meantime). I've since bought a Linksys E4200 v1 to replace the N750DB, and everything is now working great.

Note that it's actually hard to tell when iOS has an IPv6 address. There are literally no IPv6 options in iOS, and the IPv6 address itself is not displayed anywhere. The only hint is in the network config page. If you have a DHCPv6 server or giving out RDNSS via RAs, the IPv6 nameserver and/or domain will show up in "DNS" and "Search Domains", though will likely be cut off. The only OS-level way to get meaningful IPv6 address/route information is a utility app called IPv6 Toolkit ($2 but worth it in this case). You can also go to a web site like vsix.us (which of course I wrote) to see what your IP address is.

Note that even after fixing my AP and rebooting my devices, when I went to vsix.us in Safari, it was still preferring v4 over v6. I was able to solve that by clearing all cached info in Safari.

This weekend, I upgraded the OS on my main colo box, from 32-bit Debian lenny to 64-bit Ubuntu 12.04 precise LTS. This was an installation which I've had for over 8 years. It was originally installed in early 2004, when Debian sarge was in its long beta freeze. It had been dist-upgraded over the years, from sarge to etch to lenny, and was quite well maintained. And over the years, the hardware has been upgraded, from a Pentium 4 with 512MB memory and 80GB usable disk, to a Pentium D with 2GB memory and 160GB disk, to a Core i7 with 9GB memory[1] and 320GB disk.

However, it was still an 8-year-old installation, and I wanted to get to a 64-bit OS. Debian lenny's support was dropped in February, and Ubuntu precise seemed like a good target, so I made the decision late last year to replace the OS.

I bought two 1.5TB WD Black SATA drives and installed precise on them a few weeks ago, in a RAID1 set. My colo box in San Jose has a 3ware 9650SE 2-port SATA 2 card, and I had an identical card available at home. Thankfully 3ware cards store RAID configuration on the disks themselves, so it was just a matter of installing the OS on the surrogate server at home and shipping the disks to San Jose to be physically installed. I also bumped the memory up to its maximum of 24GB, as long as the case was open (memory is insanely cheap right now).

The precise install was a minimal install, with just networking and SSH configured. I then took a backup of the lenny install and put it in /srv/oldos. The idea was once the datacenter swapped out the disks and powered it on, I'd go in and, for example, "chroot /srv/oldos /etc/init.d/apache2 start" for all the essential services. I could then migrate the services into the new install one at a time.

With a few small bumps[2], that strategy worked well. However, a LOT of stuff is on this box, and it took me a most of the weekend to get everything right. Here's a sample of what this server is responsible for:

As of Monday, everything is now running off the new precise install. I installed this server over LVM, and have only allocated about 500GB of the 1.5TB usable space. There is a relatively small 120GB root partition, a 160GB /home partition, and a 250GB /srv partition. The idea is not much more than the OS should be on root (and should never need more than 120GB), /home is self explanatory, and all projects (web sites, etc) go in /srv. If /home or /srv need to be expanded, it can be done remotely relatively easily by umounting them, expanding the partitions using LVM, and running resize2fs. I've also left most of the space unallocated in case I want to run KVM guests some day, though I don't have an immediate need.

[1] Yes, 9GB memory. Triple channel, so it was 3x2GB + 3x1GB modules.[2] I had two significant problems with the migrations:

The problem with picking a custom third-party Festival voice for the voice of the Cybersauce Broadcasting Corporation is that it looks like it has not been updated in years, and is specific to only a handful of Festival versions. So now I have a minimal, yet 300MB chroot running (64-bit) Debian lenny for the sole purpose of providing Festival's text2wave utility with the custom voice.

I'm using two packages[3] not available in Debian/Ubuntu: a Perl module (AI::MegaHAL) and a PHP module (crack), both of which are published at my PPA. Both are sufficiently ancient and have not been updated upstream for years. Both compiled fine in 64-bit precise and at first appeared to run fine, but would then crash. Turns out both were not 64-bit safe, and required patching. Thankfully this is easy to roll out with PPAs.

Removed generation of 2ping6 symlinks at "make all" time (symlinks are still generated during "make install" in the destination tree)

2ping is a bi-directional ping utility. It uses 3-way pings (akin to TCP SYN, SYN/ACK, ACK) and after-the-fact state comparison between a 2ping listener and a 2ping client to determine which direction packet loss occurs.

One night last week, my cable internet service went down. OK, it rarely happens, and when it does, I have an iPhone with tethering. No big deal normally, I would just have my laptop connect to the tether. However, that night I was in the middle of a few things on my home LAN, and wanted my whole network to have internet access, not just my laptop. Getting wireless access on my home router/server seemed the most preferable. So I dug through the junk crates and found:

Two Orinoco Gold 802.11b PCMCIA cards

Four Orinoco Silver 802.11b PCMCIA cards

A Linksys 802.11b USB dongle which required a USB A to A cable

A Linksys 802.11g PCI card

About a dozen WRT54G (or similar) router/APs

I immediately threw away the USB dongle due to its odd requirement. I should throw away the Orinoco cards as well, but dammit, they had real value (ten years ago). The PCI card would have worked, except my new router/server actually has no "legacy" PCI slots, just PCI Express slots. I could have cobbled together some sort of bridge with a WRT54G, but it would have been too much work. In the end I toughed it out and waited for the cable internet to return.

The next day I ordered a $17 PCI express 802.11n adapter, and it arrived today. Ubuntu 12.04 "precise" (currently in beta, set to be released next week) saw it immediately, and with a little reading I was able to get it working with my phone. I'm leaving the configs here in case they are useful to someone else:

/etc/network/interfaces:

#auto wlan0
iface wlan0 inet dhcp
wpa-conf /etc/wpa_supplicant.conf

"auto wlan0" is commented out because I obviously don't want it running on boot, since this is a backup solution. That line could simply be omitted, but I like it there to point out to myself it is not brought up on boot.

iPhone-specific note: that "’" is actually a unicode character, U+2019. "iwlist wlan0 scan" was returning this, which wpa_supplicant was not accepting:

ESSID:"Ryan\xE2\x80\x99s iPhone"

In the end, I hopped on my laptop, associated with the phone and grabbed the SSID from the logs and pasted it in on the server, to preserve the unicode character.

With that in place, "ifup wlan0" worked fine. In the future, during an outage I should just be able to log into the server, run "ifdown eth1", then "ifup wlan0", and everything at home should be run through my phone.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256,SHA1
Wed, 11 Apr 2012 10:30:08 -0700
For a number of reasons, I've recently set up a new OpenPGP key, and
will be transitioning away from my old one. My old key was created
over 10 years ago, as a 1024 bit DSA key with a SHA-1 signatures, both
of which are considered inadequate today. My new key is a 4096 bit RSA
key with SHA-256 signatures.
The old key will continue to be valid for at least 90 days. It will be
revoked on or around 2012-07-15, or after the release of Finnix 105,
whichever is later. (My old key was used to manage signatures for the
Finnix project. This will be split out into a Finnix-specific signing
key, and will be announced in a separate message.)
However, I would prefer all future correspondence to come to the new
one, as of today. I would also like this new key to be re-integrated
into the web of trust. This message is signed by both keys to certify
the transition.
The old key was:
pub 1024D/203ECA25 2001-05-09
Key fingerprint = B023 7C63 DF28 70AA C3AB C54A 2996 10A9 203E CA25
And the new key is:
pub 4096R/86AE8D98 2012-04-11
Key fingerprint = 42E2 C8DE 8C17 3AB1 02F5 2C6E 7E60 A3A6 86AE 8D98
To fetch the full key (including a photo UID, which is commonly
stripped by public keyservers), you can get it with:
wget -q -O- http://www.finnie.org/rfinnie.gpg | gpg --import -
Or, to fetch my new key from a public key server, you can simply do:
gpg --keyserver pgp.mit.edu --recv-key 86AE8D98
If you already know my old key, you can now verify that the new key is
signed by the old one:
gpg --check-sigs 86AE8D98
The new and old keys' primary UIDs are both "Ryan Finnie
". This was by design, to ensure you must verify the
key signatures rather than seeing something like "Ryan Finnie (2012)
".
If you don't already know my old key, or you just want to be double
extra paranoid, you can check the fingerprint against the one above:
gpg --fingerprint 86AE8D98
If you are satisfied that you've got the right key, and the UIDs match
what you expect, I'd appreciate it if you would sign my key:
gpg --sign-key 86AE8D98
Lastly, if you could upload these signatures, I would appreciate it.
You can either send me an e-mail with the new signatures (if you have
a functional MTA on your system):
gpg --armor --export 86AE8D98 | mail -s 'OpenPGP Signatures' ryan@finnie.org
Or you can just upload the signatures to a public keyserver directly:
gpg --keyserver pgp.mit.edu --send-key 86AE8D98
Please let me know if there is any trouble, and sorry for the
inconvenience.
Thank you,
Ryan Finnie
[Much of this text was adapted from dkg ,
thank you!]
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
iQIcBAEBCAAGBQJPhcEbAAoJEH5go6aGro2YCqYQAKM2IlO3CgOLPDYIww7tdt0t
TTYgp1ng0oOkRdSKm7maavnVd8Drkys/TgO8DQD/tuf37ZES1Vid7yqQSddAx49/
da+V9EdbCaZOqqVUY0qtW5JTV8xyn67zLwhj06/L+NWf3iP/6ymCzbWrVor2jdtn
Efeylj+T+j5igLOTBkx22d4W3VU787fiMCiwLgDmytwJ66cHR4qR+jWWnsEdVuuF
AVwcs9ELRicppE0p1jMmsr/rKJJAeM0xb1+V+BL685q4XkXRvY6Fg2WC2aoTFJF/
jp94JtlodooWuCuWnNFofqVdYIuSezjki+aRy3KmCFliWaULqL8akdtVlUmA/2gM
PdZE7Acf7JU4TVH/drvY6pbK7zwFIuBA+ESbB4lJEvZFC+Ub2aM7SceDAp2CBd+i
B4+sWkv89ZSDZqGXK2ylTNFU2o2MfQLxZWKZOdq0exZJkb5NSNF22YY8WsMsXpqJ
Ydtt0mxVp57rkhc01Vx4DJ5+OKmCJEgiTj+wnef1RvZh3ayLqkS5wUTkf6S4OLwP
cJT3i+mhAU7CQVFqSnmg98ADiq1SVnWz2rsq4m1e965ST1OpNxicK4g9UO/ePUT2
yBtyEfmFCV98KCADUSdWmD0Nx3uzHxtb+0RMMulPOQszB9VDPxIcNbdcKMLzzcp+
ZwM/dc405Tvdzptf/khgiEYEARECAAYFAk+FwRsACgkQKZYQqSA+yiXbTwCggR1l
9IHQVOKCDEJmot02C8pRFFIAnjvSY/eCeLW3mjvBF8rQUCg80KRJ
=pweu
-----END PGP SIGNATURE-----

I'm at a loss here. I've got an IPv6 setup at home, with radvd giving out network information. And it basically just works. My laptop associates with the wireless network, sends out a Router Solicitation packet, the router responds with a Router Advertisement (RA), and the laptop gives itself an IP. And for good measure, the router sends out unsolicited RAs every 10 seconds or so.

But my iPhone and iPad are no longer getting IPv6 addresses. They associate with the AP, get a DHCPv4 address, then nothing. I've even tried using the Ip6config app to scan for RAs, but according to it, no RAs arrive. But I can see them from my laptop, capture them with Wireshark, and they look proper. And of course there's no way to get around this in iOS, since there are literally no user-configurable IPv6 options.

It's been a long time since I've built a computer from scratch, and even longer since I've been excited about it. For the last few years, my home office network has been mostly kitbashed together from other computers, and when I've needed to buy new parts, it's usually been begrudgingly.

However, a few weeks ago, the stars all aligned. I was preparing to upgrade the RAM and drives on my colo box in San Jose after Ubuntu 12.04 is released, and went shopping. Hard drive prices are very high due to the flooding in Thailand last year, but RAM is dirt cheap. I was able to max out my colo server's RAM to 24GB (6x4GB) for $100. That got me thinking about my home office. At any given moment, 7 desktop computers were on: my main Ubuntu workstation, a Windows gaming machine, two G4 Mini Finnix dev machines (PPC), an C2D E7200 Finnix dev machine (x86), an Athlon64 3200+ Debian sid dev machine and an Athlon64 X2 4600+ router / miscellaneous server.

The desktops are used frequently, the G4s are insignificant (they consume 9w each), but the last three servers were all relatively old, very power hungry (and hot), and all perform tasks that need somewhat significant computing power, though usually never at the same time. And at idle, the three of them draw a combined 240w. So I decided now would be a good time to build a virtualization server and combine them. I considered upgrading one of them for the task, but decided it wouldn't work: The E7200, despite being relatively new, doesn't support VT. The 3200+ is nearing 10 years old (it was the first 64-bit consumer processor released) and definitely doesn't support VT. The 4600+ does, but it's pretty old, and the worst power offender.

No, it would be best to start from scratch. I decided to focus on an Intel Sandy Bridge processor, as much RAM as I could get, and a mid-range 6Gbps SATA drive for serving the VMs, with a focus on energy savings primarily and computing power secondarily. Here's what I ended up with:

The parts arrived on Wednesday the 29th, I assembled it last weekend, and installed Ubuntu precise Beta 1 (AMD64 server). I was worried about building a Sandy Bridge system, the platform being relatively new, but nearly everything has worked perfectly so far. The CPU is well supported, the integrated graphics work, and all integrated components on the motherboard work fine. The only issue I've found so far is the system will not reliably power off after shutdown, but I can live with that. Heck, this is even the first machine I've owned where SATA hotplug works correctly.

The 6Gbps Seagate is a LUKS-crypted LVM disk holding the OS installations. The main install contains the routing / VPN / DHCP / etc services, as well as a QEMU/KVM host managed by libvirt. The Finnix dev and Debian sid dev machines have been transferred to this as VM guests. And Finnix build times have been reduced by nearly 2/3rds in the process.

Additionally, the host contains an existing 1.5TB WD Green SATA 3Gbps drive for shared media (ISOs, iTunes collection, movies, etc), and an existing 1.0TB WD Green SATA 3Gbps drive for backups of the other hosts. This drive is mounted in one of the front hotswap SATA sled trays for easy removal. (I learned this lesson late last year when I evacuated my home due to a nearby wildfire, and found that even quick-release case screws and toolless hard drive mounts were difficult to remove when you are short on time.) The other sled tray is simply for miscellaneous uses when needed.

So I replaced three servers, consuming 240w combined idle, with a beefy new server: powerful new CPU, 32GB RAM, and 3 hard drives totaling 4.5TB. I was expecting the idle power draw to be about 120w. But when I plugged it into my Kill-a-Watt, I found the power draw was really... 58w. Wow. And the CPU and case temperatures always hover around 28C. My first feeling was anger with myself; I should have spent the extra few dollars on the i5-2500 or i7-2600, but I was worried about power draw and heat dissipation. Oh well, the i5-2400 is still much faster than anything I've owned before.

I got all the functionality transferred over to the new server last weekend, but haven't done much with it for the last week. (I was pretty sick this week, missing four days of work.) This weekend I hope to make sure everything is in working order, and decommission the other servers.

A month ago, I started working at Canonical, the makers of Ubuntu. Normally I'd write a long-winded post about my experiences from the last month, how/why I started and so on, but I'll just summarize: it rocks.

I work in the IS Operations group as a system administrator, and while my Launchpad profile has the little "Member of Canonical" badge on it now, I am not an Ubuntu developer (though of course Canonical employs quite a number of Ubuntu developers). That's not to say I don't contribute to Ubuntu; I file bugs, and manage packages as trickle-down from my Debian maintainer status, but these are all contributions that any member of the public community can do. Working in Canonical IS Operations is much like being a sysadmin in any large Ubuntu server shop.

I've gone back through last year's posts and tagged relevant posts with planet:canonical, which are aggregated at Canonical Voices, a feed aggregator for Canonical employees. ("Planet" has become a generic term for feed aggregation sites, though Canonical Voices is not actually running Planet software.) These tagged posts don't necessarily have to do with Canonical or Ubuntu topics, but are a filter for posts I want to appear there; technology topics, mostly.