OpenBSD 3.9: Blob-Busters Interviewed

Freedom, openness, security -- these principles lead OpenBSD development. The song for this release, Blob!, and the newartworks that promote them.

This release, like every OpenBSD release, contains OpenBSD and its source code. It runs on a wide variety of hardware. It contains many new features and improvements. OpenBSD attempts to convince vendors to release documentation and often reverse-engineers around the need for blobs. OpenBSD remains blob-free. Anyone can look at it, assess it, and improve it. If it breaks, it can be fixed.

Federico Biancuzzi interviewed OpenBSD's team of Blob-Busters and discussed new features of OpenBSD 3.9 along with freedom (and quality!) threats.

You have been fighting against binary-only drivers and firmware for years. Would you like to share some success stories?

Jonathan Gray: With such discussions you have to be careful to define the terms involved. Firmware does not run on your computer, it runs on the device in a separate address space. Things like PCMCIA, CardBus, and USB devices may seem really small but even these have a processor, and that is where the firmware is run. Traditionally, these devices have come with the firmware already on a ROM ready to go. In recent years, some devices have moved away from this model and required the firmware to be loaded into the device at runtime before it will work. This saves vendors some amount of money required to make the hardware.

The burden of dealing with the firmware is then put into the hands of the operating system. To have hardware just work we need to be able to include this firmware in OpenBSD. What we ask vendors for is the right to freely distribute the binary images required to make their products work. Atmel, Ralink, Zydas, and others have been really helpful in this regard; while others, like Intel, don't seem terribly keen on letting people use their hardware. You can do an install over wireless with a Ralink wireless device, this is something you can't do with the current Intel situation with Intel wireless devices. You have to read their license agreement and manually locate and download the correct firmware for your device before anything will work. The person responsible at Intel had their details listed in our documentation, the phone number has since been changed.

Drivers that are binary-only or contain a binary-only portion (binary blob) run on the computer, in the computer's memory. They typically run at the most privileged security level possible due to their requirements to talk to system memory and the like. This gives them access to anything on your system, and if they screw up it can be a disaster. These drivers are unportable and typically limited to a handful of architectures. With drivers that have full source available, people can see what the driver is doing and make corrections as required, this is something that can't be done with drivers that contain binary components. People who use binary drivers become dependent on the vendors who provide them for fixing bugs and if a vendor decides to drop the driver, you're out of luck. Not running the latest hardware with the latest approved software? Sorry, too bad.

There aren't any drivers in OpenBSD with binary-only components, this is quite a contrast to pretty much everyone else out there. We ask that vendors allow us to distribute their firmware freely and do not require source code for it.

Traditionally, 99% of Ethernet drivers in OpenBSD have been based on the efforts of Bill Paul in FreeBSD to support pretty much every kind of Ethernet device under the sun. So, naturally there was a point where Bill asked NVIDIA for documentation to the onboard Ethernet in their nForce products, and it would appear they turned him down as he moved onto working on other things. Sometime later, some other people did what is depressingly more and more common in FreeBSD, they included a driver that is a thinly-veiled wrapper around a binary blob. They even had to get a license specific to FreeBSD to distribute it. This is a horrific scenario we would never consider doing in OpenBSD.

Sometime later, I had access to a machine with nForce Ethernet and started writing a driver with full source code even though we still did not have documentation. Some information as to how the thing works was derived from a reverse-engineered Linux driver. Damien Bergamini later came on board to help get it working in time for 3.9.

This kind of thing is happening all the time, we spend large amounts of time trying to get vendors to give us documentation and rights to distribute their firmware so that their hardware will function as something more than expensive paper weights.

OpenBSD 3.9 has had lot of work done to support management sensors (fans, temperature, power, etc.). Did you have any problem finding the docs to write appropriate drivers? Do they use standard protocols, or each manufacturer reinvents the wheel on his own? How do you handle hardware bugs or "improvements" to the standard made by a single manufacturer?

Marco Peereboom: This is really exciting stuff and was written by lots of people. Jordan Hargrave wrote the bulk of the IPMI code and shared his previously written ESM stuff with David Gwynne. Mark Kettenis, Alexander Yurchenko, and Theo de Raadt (many others worked on it but they did the bulk of the work) revamped the I2C stuff and we now support most I2C devices.

The I2C devices are well documented, however, that doesn't mean that everything is hooked up as expected. Vendors use those parts in different ways and sometimes that results in unexpected behavior. Theo and Mark did an amazing job working around most of the hardware issues and improvements as you call them.

IPMI is a standard, but there are vendors out there that have reading comprehension issues. Due to this we still have some quirks that need to be addressed.

Overall, we still need to improve the sensors framework itself. Currently, it's a little tedious to maintain a farm of servers. There are some developments but nothing concrete yet. I do expect this to get a lot of attention in the near future so look out for some innovation in this area.

Can you elaborate on where the sensors framework is heading in the future?

David Gwynne: The main goal of the sensors framework is to support as many different environmental sensors as possible and provide a uniform interface on top of them.

Currently, we have about 30 drivers in the tree for a wide variety of hardware on a variety of different architectures. Sensors are everywhere. The devices we support range from simple I2C and ISA devices all the way up to the sensors on SCSI enclosures and the management controllers found in servers. Despite the differences between all these sensors they are all part of the same framework and they all appear in the same way to the user. Most of these drivers were written in the last six months.

Due to the prevalence of sensors in most modern hardware and the new drivers in 3.9, a lot of people will be pleased to discover that they can now see the state of their sensors simply by going sysctl hw.sensors.

Our goal in the future is to keep expanding this device support and to refine the user interface to it. For example, there is a small daemon called sensorsd that is currently in the tree and part of 3.9, but it is ugly and hard to configure. We're hoping to replace it in this development cycle and provide a default configuration that just works. If we can do this we can enable it by default.

Some of the more intelligent hardware is integrated with a hardware watchdog, but the watchdog API and the sensors framework don't fit together well on the same hardware. Some of the same hardware also has the ability to log events. I intend to work on these two areas in the future as well.

Is there anything new on bioctl side?

Marco Peereboom: Unfortunately, no one worked any more on this and I got sidetracked working on IPMI and ACPI. We did get lots of good feedback from the community. I am hoping to work some more on this for this release.

apmd now supports automatic performance adjustment for CPUs which provide this feature. How does it choose how much and how often change the CPU frequency?

Nikolay Sturm: The idea is very simple, apmd needs to figure out when the CPU is busy and then raise its frequency to get maximum performance. Once the CPU is idle again, the frequency can be lowered to save energy and reduce the noise level.

In practice however, this is more complicated. First, you need a good metric. Some people believe the system's load is that metric, but that's wrong. The load is just the average number of processes running plus the ones wanting to run. This does not tell us anything about the actual CPU usage.

The kernel actually measures where the CPU spends its time, regular userland processing, nice userland processing, kernel processing, and interrupt handling. Another counter measures the amount of time the CPU is idle. And this is the correct value to look at.

Unfortunately, this opens a new can of worms, as you are free to look at these counters as often as you want. The more often you look and possibly adjust the CPU frequency, the better your overall performance but the less stable your CPU becomes.

One of the reasons that made me implement this feature was that other tools were adjusting the CPU frequency multiple times per second, so it was jumping around all the time.

Having a background in physics, I prefer stable systems, so I watched my computer doing stuff and trying to figure out a good-enough time scale for apmd's measurements. It turned out looking once a second was enough to still get decent performance, but does not change the frequency too often, because CPU-intensive tasks normally last either much shorter or much longer.

Figuring out how much to change the frequency was simple, as we can only set the frequency to certain discrete values, the amount of which varies from CPU to CPU. On average however, we have four or five values. In order to get decent performance, going up in 50% steps and down in 20% steps was considered acceptable.

This way, a CPU-intensive task will take about two seconds to bring the CPU to full speed, but when it does a little I/O in between, reducing CPU usage, the frequency is not dropped too much, so it can happily work on.

Bob Beck: The automatic performance mode Nikolay describes above can be enabled in two ways. apmd -A starts apmd in automatic performance adjustment mode. In this mode, the performance adjustment is done only when the system is running on battery power, or if the battery charge is low. Otherwise, the CPU is left to run at full speed. apmd -C starts apmd in cool running performance mode, which will adjust the CPU speed regardless of the power connection status.

What is the status of the G5 port?

Mark Kettenis: The iMac G5 is perhaps the best supported model now; Xorg works out of the box, and even the built-in ambient light sensor works. The only things that don't work are FireWire and the built-in Airport Extreme. It's difficult to support the Airport Extreme because Broadcom refuses to give us documentation for it.

Support for the various PowerMac G5 models isn't complete. We haven't tested all models, but it seems the dual core G5's don't work at all. On the dual CPU models, the onboard SATA controller doesn't work yet, but you can always hook up an USB disk to those. The single CPU models might just work.

The Xserve G5 also has the onboard SATA controller problem.

Although the G5 is a 64-bit CPU, OpenBSD runs in 32-bit mode. It runs the same kernel and userland as the older G3/G4 machines supported by macppc. But if the kernel detects a G5, you'll automatically get the W^X protection that the CPU provides.

Do you prefer this CPU over x86 ones? How many features of this architecture do you plan to support now that Apple dropped it?

Mark Kettenis: The x86 architecture is just plain ugly because of all the legacy code it needs to support, although the IBM 970FX/MP (the CPU Apple calls G5) has its dark corners too.

We have no plans for dropping support for the macppc platform. We still support the mac68k platform that runs on Apple machines with a Motorola 68000 CPU, the CPU used by Apple before they switched to PowerPC.

For now, Apple is still selling G5 machines. And when they stop doing that, PowerPC won't be dead. IBM's new Cell Broadband Engine is PowerPC-based and might end up in interesting and affordable machines in the near future.

Does OpenBSD 3.9 work on Intel-based Macs?

Mark Kettenis: No, but since I (and a few other developers) have one now, 4.0 probably will. ;-)

Part of the rthreads implementation has been included in the kernel, but it is not built by default. What type of development road map should we expect?

Ted Unangst: Signal handling and scheduler modifications are the next big steps. After that, whatever refinement is necessary to make the system robust.

Signal handling is rather difficult because signals are sent to the process, but the actual delivery is done to a thread. Each thread can set its own signal mask, and it's the kernel's job to find a recipient. Some signal actions, however, like suspend and continue, should be sent to all threads, since you don't want only one thread to stop.

The GENERIC kernel now includes multicast routing. How does it work?

Esben Norby: Multicast routing has been around for a very long time, but not that many people use it. I think there are several reasons for the small usage of multicasting in general. Multicasting (the ability to send the same packets from one source to several receivers) lacks a killer application. That killer application could be video streaming, radio streaming, etc.

I've been using (video streaming) the multicast routing stuff with OpenBSD for a couple of years now, but at this point it is time to clean up the whole multicast mess.

The only thing new about multicast routing in 3.9 is the ability to enable it via sysctl, no more recompiling kernels. :-)

Before multicast routing can be used a multicast routing protocol must be present. Currently, base only supports DVMRP with the mrouted(8) daemon. Mrouted(8) is old, scary, and weird and only supports DVMRP. More modern multicast routing protocols exist like PIM. DVMRP is not bad per se, I use it in a medium-sized network (about 50 routers).

Currently, work is ongoing to improve the multicast routing protocol situation. In multicast routing, the routing table is a bit different and the forwarding function acts different. The multicast routing table not only contains the outgoing direction, but may contain several outgoing directions. For example, a packet arrives on interface 1 and the destination is 239.3.4.5 (address in the special Class D range), the packet is forwarded out on interface 3, 4, and 7 because hosts on those interfaces (or other multicast routers on behalf of hosts) have requested interest in that particular group.

What is the current status of the trunk interface?

Reyk Floeter: The trunk interface has been introduced in the previous 3.8 release with basic functionality and some initial problems. Many internal changes, like VLAN-size MTUs (if supported by the physical ports) and some new functionality are available in the 3.9 release.

The most important improvement for the 3.9 release is proper multicast support. Now, it's possible to use CARP and PFSYNC on a trunk device without limitations. Personally, I use this on redundant firewalls with additional layer 2 redundancy -- redundant firewalls, redundant network interfaces and even redundant network switches connected to the firewalls thanks to trunk in the new failover mode.

The trunk failover mode is an alternative mode to the initial round-robin interface scheduler. Instead of distributing the traffic to the connected trunk ports, only one active interface is used at the same time. If the first active interface (the master port) fails, the next active interface will be used. The failover mode does not gain any performance but works great for layer 2 redundancy without the requirement of trunk support by the network switches (mcbride even tested it with simple hubs!).

What's new in hostapd?

Reyk Floeter: hostapd has been heavily improved for the 3.9 release. I did some cleanup and implemented some nice new features. An important internal change is the internal handling of wireless nodes. I modified hostapd and the entire net80211 framework in OpenBSD to use collision-proof and fast red-black trees instead of the hash tables before. This is a proactive security improvement, which also fixes some problems in the net80211 framework.

Furthermore, the daemon now supports multiple wireless interfaces and I extended the event rule syntax to allow interface-specific rules. With the benefit that the daemon now handles roaming between internal interfaces correctly.

While playing with mrouted, the multicast router daemon, I figured out that hostapd missed a configuration option to define a higher multicast TTL. Now it's possible to forward hostapd messages to other, non-local network segments. This is very cool for remote Wireless Intrusion Detection Systems (WIDS) using hostapd as a real-time sensor. Another advantageous feature in conjunction with WIDS is the new rate keyword for event rules. It allows to detect flooding attacks (like the void11" 802.11 deauthentication attack I released some years ago) and works similar to the max-src-conn-rate option in PF.

What's the status of ipsecctl in 3.9?

Hans-Joerg Hoexer: The main goal for 3.9 was to improve the interaction with isakmpd(9). For example, reyk@ added pre-shared secret authentication. Moreover, with 3.8, it was still necessary to use an isakmpd.policy file in certain situations, this limitation is now gone.

Markus@ added support for tunneling using IP in IP. This does not provide any security mechanisms but allows [you] to setup easy tunneling without using gif(4) interfaces.

Small improvements include support for transport mode when using manual keying, using interface names when specifying addresses, and a more relaxed syntax. For example, it does not matter anymore in which order you specify source and destination address.

A new version of FTP proxy has been included. Why?

Camiel Dobbelaar: The new FTP proxy (previously known as pftpx) has a couple of advantages.

It handles the passing of all data connections automatically via anchors. The user only has to setup the anchors and the rules for the control (port 21) connections, which is simple enough. With the previous proxy, the user had to setup rules for all possible data connections as well, which could be quite hard with non-standard configurations.

Lots of FTP clients expect data connections to have the same IP addresses as the control connection. If they have an FTP session with server X, they refuse data transfers to/from proxy Y for security reasons. The new proxy [uses] the NAT features of PF to make sure that the addresses are always the same and should therefore satisfy all FTP clients. The old proxy did not have NAT tricks at its disposal, making some clients fail in mysterious ways.

Those are the biggies. Other advantages are that it handles IPv6, the extended FTP modes, and that PF queuing can be used for data connections.

HTTP proxy authentication support has been added to netcat. When should we use this feature? Is this a first step toward a proxy framework for PF?

Damien Miller: No, it is a useful tool for making connections out from behind proxies through, especially SSH connections. For example, adding this to your ~/.ssh/config will make ssh connect through an authenticating HTTP proxy:

ProxyCommand nc -x proxy:8080 -P proxyuser -X connect %h %p

Not all proxies allow connections to port 22 where OpenSSH usually listens. You can always make it listen on another, high-numbered port to work around this.

It seems there are no new features in PF this time. What's going on?

Henning Brauer: Well, we don't hack on PF because we hack on PF, we solve problems or add features we miss. There weren't any big problems to solve or features we miss.

I saw that you bumped PF state, byte, and packet counters to u_int64_t. Why?

Henning Brauer: Consistency, since they were 64 bit in most places already, and overflow concerns. The u_int32_t counter we had before could wrap around. While that has no bad consequences for PF, it might lead to wrong data in accounting systems and the like using this information from PF.

What type of work have you done on Apache?

Henning Brauer: Lots of cleanup. Pretty boring to do, not visible for the end user. Except that we might have fixed a bug or two while doing so.

TRACE was disabled. Why? Is there anything else that you would like to disable or remove from the codebase?

Chad Loder: TRACE was disabled because there have been two Apache vulnerabilities related to this feature: VU#867593 and CAN-2005-2088.

We looked at various ways to support disabling TRACE using configuration options. I asked around on the mailing lists if anyone uses TRACE, nobody does, so we just decided to rip it out entirely. Nobody has complained yet, so I'm pretty sure nobody was using it.

Where has libresolv gone?

Henning Brauer: It only ever was a stub with nothing in it, because some software incorrectly assumed it always had to link libresolv. We found that these days the stub libresolv just confused some configure script.

I saw that the in-base libpcap was updated with most features of a recent version from tcpdump.org. Why didn't you just import the latest version?

Damien Miller: The tcpdump.org libpcap has a heap of portability goop in it, so we just took the important bits of the new API and implemented those, based on the tcpdump.org sources. The API is good, but there is too much abstraction in the tcpdump.org sources, so we cleaned it up a little.

Pretty much anything recent that previously needed an updated libpcap should compile against the in-tree one now.

Why did you choose to compile system libraries with debugging symbols? How much does this approach increase RAM usage? What about performance?

Henning Brauer: The added debug symbols help debugging issues, since now we get tracebacks with function names instead of addresses (that can change). It does not increase RAM usage at all and does not have performance impacts. It just costs some space on the hard disk.

What persuaded you into auditing the whole source tree to fix the wrong uses of queue(3)?

Otto Moerbeek: It all started after Pedro found a filesystem bug that happened because an element was removed from a list twice. The list was implemented using the macros from sys/queue.h. Now, I was more or less aware that it was easy to use these macros in a wrong way, but the test program Pedro made showed that it was really easy for the data structures to become corrupt: a linear list could become circular, for example. This caused a hang, because the kernel was busy running though the list trying to find the end.

So, we did a few things: one, improved the manual pages and included a warning for various things developers should not do; two, audited the source tree for wrong usage of the queue macros; and three, introduced strict versions of the macros that detect wrong usage [at] run-time. The strict macros are not enabled by default, but both developers and others are encouraged to run with them.

All this enabled us to find real bugs and fix them.

The default value for the X Aperture sysctl is now off. What's so
evil in modern video cards?

Matthieu Herrb: Loic Duflot demonstrated in an excellent paper [PPT slides] at CanSecWest that the hardware access privileges that the X server is granted by the aperture driver can be abused to gain access to kernel privileges (allowing to bypass the security level settings for example).

The X server paradigm of "userland drivers" exposes all hardware features to userland. Some of these features (Loic used the Pentium System Management Mode, but there are probably dozen of other similar features for this purpose) can be used to bypass the memory protection and thus provide the X server with full control of the kernel.

OpenBSD developers went through the effort of privilege separating the X server as its own user, but it doesn't help at all for this issue. We've also tried to design a better protection scheme with the current X design, but it appears to be almost impossible. The X server needs to be redesigned to not require direct access to hardware.

We think it is a very urgent matter for true security will never be achieved otherwise. For the time being the only advice we could give to OpenBSD users who want their system to be secure is to keep allowaperture=0 and not use the X server on those systems.

Also note that other systems have no protection at all against this attack.

I remember that just before releasing 3.8 you had to disable the new behavior of your implementation of malloc()/free() that returned SIGSEGV when accessing a freed area. You had to do this because too many ports were instable (crashing). Does 3.9 enable it by default?

Otto Moerbeek: I first have to make a correction: we do unmap unused memory, but not very aggressively. There are too many programs containing "use-after-free" bugs that would stop working if we unmapped unused memory all the time.

Apart from some code cleanup, the malloc implementation did not change for 3.9; it has been proven to be very solid. Its stricter checking and randomizations features enabled us to find evil bugs in all kinds of software. And, of course, these mechanisms also provide extra protection against heap overflow bugs that could potentially be exploitable.

While many buffer write overflows have been solved in a lot of software, read beyond end of buffer bugs are still quite common. To be able to catch these as well as use-after-free bugs, we hope to make our malloc even more strict in the future, but these steps have to be taken very carefully; if our malloc becomes too strict -- even while staying within boundaries defined by POSIX -- too many programs and users will suffer. But if we do it right, in the end everybody will gain: bugs will be found and fixed.

What's new in pkg_add and other pkg_* tools?

Marc Espie: A lot!

pkg_add -u (updates) now works. Specifically because pkg_add can recover from most FTP errors now. And the full code was activated.

There is also an interactive mode, -i, that you will want to turn on most of the time. Instead of aborting, pkg_add will ask questions when it can't find the correct answer. This goes from simple things like:

pkg_add -i screen

asking you which screen package you want, to more complicated stuff, to making sure you want to perform potentially dangerous operations.

There is also Nikolay's work to let you grab packages through FTP, when available, instead of building them from the ports tree. You just have to set FETCH_PACKAGES for that.

Aside from that, there have been hundreds of bug-fixes in fringe cases. The package tools cope much better with a wide variety of error conditions: if the md5 of installed files do not match, pkg_delete will leave you with a stub package instead of simply reporting it did not remove the package. pkg_add -u will update packages when they depend on stub library packages, instead of leaving them around. pkg_add will refuse to install packages with INSTALL scripts if it finds you the relevant partition is mounted noexec, etc.

In short, we do think that package updates are ready for production uses. Most of the ports developers have been using it for months by now.

A new tool called sdiff was introduced in the 3.9 release. Could you explain what it does and how it works?

Ray Lai:sdiff is visual comparison tool that displays two files side-by-side, with differences marked. An interactive mode can be used to merge changes, prompting the user at each difference.

One use of sdiff is to merge updates to configuration files. Configuration files are often customized for each machine. Over time, vendors add new options or change defaults in configuration files. sdiff can pull in vendor updates while keeping local customizations.

This implementation of sdiff is new, but sdiff already exists on many Unix systems. In fact, there was an sdiff in previous versions of OpenBSD, but was removed from OpenBSD when the GNU diff utilities were replaced by BSD ones.

OpenBSD and OpenSSH have been in the news lately with an appeal for donations. Could you please describe what happened?

Ray Lai: The costs of producing OpenBSD have been increasing, while the income generated has been decreasing. The percentage of people choosing to download through FTP instead of buying CDs has increased dramatically. While the number of OpenBSD and OpenSSH users has been increasing, the total number of CD sales has been decreasing!

New hardware is always needed to improve support and to replace or upgrade the infrastructure. Due to cooling requirements and electrical consumption, running this new hardware increases the already monstrous electric bill, currently at $100 per week.

The project's growth is fueled by hackathons, yearly events where developers come together and code for a week straight. OpenBSD only pays for accommodation. Tickets and other travel expenses like food are paid by the developers themselves. OpenBSD tries to pay for the tickets of poorer developers, but has very limited resources to do so. Many new developments came from hackathons, including SMP support for AMD64, SSH tunneling, improved 802.11 wireless networking, and PF.

Hackathons have been so successful that mini-hackathons have taken place, concentrating on specific areas of OpenBSD, such as the ports tree or PF development. More of these are planned in the future, so please make it possible!

David Gwynne: I agree with Ray, the hackathons are important for the project because it speeds the whole process up. A lot of OpenBSD development relies on discussion between the developers which is usually done via email. For example, at last year's hackathon I remember pascoe and krw being annoyed by mjc with lots of SCSI/umass interaction issues. There were some real fixes in both layers that improved things that a lot of people were complaining about.

The hlt hlt bug was also found and fixed at the hackathon. I'm not sure it would have been found in any other environment. Because the developers were all in the same room the discussion was sped up. Email just can't compete with a face-to-face conversation. As a result, everyone's issues were raised and discussed in minutes rather than the hours it takes to get email back from people all over the world in a variety of time zones. I'm sure that the fix to the hlt hlt issue would have taken months without the hackathon. Instead it was fixed in a day.

The hackathons also give us an opportunity to really focus. [For example], marco did a lot of hard work on figuring out how to interact with ami hardware at the last hackathon. Without this foundation bioctl might still be in development. ;)

These are just a few examples that show how the hackathons are a good investment for the community.

Federico Biancuzzi
is a freelance interviewer. His interviews appeared on publications such as ONLamp.com, LinuxDevCenter.com, SecurityFocus.com, NewsForge.com, Linux.com, TheRegister.co.uk, ArsTechnica.com, the Polish print magazine BSD Magazine, and the Italian print magazine Linux&C.