The FreeBSD Foundation is a 501(c)(3) non-profit organization dedicated to supporting the FreeBSD Project. The Foundation gratefully accepts donations from individuals and businesses, using them to fund projects which further the development of the FreeBSD operating system.

Monday, September 26, 2011

The following project update was written by David Chisnall who received a grant from us to implement xlocale APIs to enable porting libc++. We're pleased that the project is almost completed!
It's traditional to start this sort of thing by telling you who I am. I started using FreeBSD around 2001. At the time, I'd used Linux but switched to FreeBSD because it sounded like it worked correctly - I could have xmms playing music, my IM and email clients notifying me of new messages, and BZFLag making gunshot noises all at the same time. Apparently, ten years later, this still doesn't work reliably on Linux...
I got involved with clang via a somewhat indirect route. I'm a member of the core team of the Étoilé project, which aims to build a (BSD licensed) desktop environment on top of GNUstep. I grew increasingly frustrated with the level of Objective-C support in GCC, which included shipping one release with Objective-C completely broken and displaying no progress towards supporting the Objective-C 2 extensions that were about 5 years old at the time. I looked at the code, but it was an incomprehensible mess of spaghetti.

Apple had just released a new compiler front end (clang) that had Objective-C parsing mostly finished, but code generation missing. I started poking the clang code to try to support the GCC Objective-C runtime, and a few weeks later had a working Objective-C 2 compiler. I then grew frustrated with the limitations (including the license) of the GCC Objective-C runtime and wrote a more modern (MIT licensed) replacement. Clang now supports both and with the new runtime is at feature parity with Apple's implementation of the language.

While hacking on clang - which I do on FreeBSD - I fixed various FreeBSD-related bugs. This put me in contact with Roman Divacky, who had been working on importing clang into the base system. This is an important task, because FreeBSD currently uses the last GPLv2 version of GCC as the system compiler. Although this release seems less buggy than subsequent ones, it is now over 4 years old and is no longer supported upstream. It won't, for example, get any of the features of C1x or C++11.

The compiler is only half of the problem. The other half is the standard library. For C, this isn't an issue: FreeBSD has had its own C standard library implementation since before it was FreeBSD. For C++, it's a bigger problem. FreeBSD currently ships with GNU libstdc++, which has undergone the same sort of license change as GCC, leaving FreeBSD stuck with an old version.

A good candidate for replacement is libc++, developed as part of the LLVM project and available under UIUC and MIT licenses. This has a few dependencies. One is a low-level C++ ABI library, which implements the dynamic parts of C++ such as exception handling and run-time type information. I'd written an implementation of this for PathScale, and the FreeBSD and NetBSD Foundations jointly paid for it to be open sourced. I've since extended it with some additional features required to support C++11, which includes an std::exception_ptr object that allows exceptions to be passed between threads.

The other dependency is the C standard library. Libc++ was written by Apple developers (Apple is in the same situation as FreeBSD with regards to the GPLv3) and so uses some features of Darwin libc that are not portable. Specifically, Darwin libc has a convenient set of extensions to localisation: xlocale. This extends the POSIX 2008 per-thread locale APIs (missing in FreeBSD) to provide a set of _l variants of locale-aware libc functions that use a specific locale, rather than the global one.

My recent work, sponsored by the FreeBSD Foundation, has been to implement the missing xlocale APIs. This is now mostly done and pending code review. With this and the new tweaks to libcxxrt, it's now possible to build libc++ on FreeBSD and most of the tests pass.

Most of the remaining test failures are in the header. This defines a lot of complex atomic operations and requires a lot of compiler support. Eli Friedman has been working on adding this support in clang, and with his latest patch applied 25 of the 52 atomic tests pass. There are still a few remaining failures:

- 27 caused by clang not fully supporting the atomic operations yet
- 3 caused by clang not fully supporting the C++11 type-trait intrinsics
- 20 that I don't think are real failures - they're caused by the VM where I'm running the tests not having sufficiently fine-grained time reporting for the thread operation timeout tests to work properly
- 1 is caused by FreeBSD lacking the C1x quick_exit() APIs.
- 2 caused by FreeBSD lacking the uchar.h header

For comparison, Howard Hinnant, the libc++ lead developer, just sent me a list of the failures on OS X. On FreeBSD, 4271 tests pass, 53 fail. On OS X 4253 pass, 71 fail. This is looking very promising for an entirely GNU-free C++ stack.

Sunday, September 25, 2011

The FreeBSD Foundation is pleased to announce that Swinburne University
of Technology's Centre for Advanced Internet Architectures has
been awarded a grant to implement DIFFUSE for FreeBSD.

DIFFUSE (Distributed Firewall and Flow-shaper Using Statistical
Evidence) is an extension to the FreeBSD IPFW firewall subsystem
developed by CAIA. It allows
IPFW to classify traffic based on statistical properties of flows being
observed in realtime, and instantiate network actions across a
distributed set of "action nodes" for particular flows if required.

This project will tidy up and integrate the
existing DIFFUSE
prototype into FreeBSD, and incorporate a number of important new
features. Integration of DIFFUSE into FreeBSD will increase FreeBSD's
utility to designers and implementers of FreeBSD-based networking
infrastructure.

Network architects frequently require the ability to classify different
traffic types flowing across a network, typically using packet
inspection capabilities of base system tools such as ipfw and pf.
Traffic classification then enables the provision of customized service
levels to different traffic types (such as priority packet queuing and
forwarding, or allocation of specific bandwidth guarantees).

DIFFUSE uses machine learning techniques to enable robust and efficient
classification of IP traffic flows based on their unique statistical
properties in addition to traditional inspection of packet header or
payload contents. DIFFUSE also allows traffic classification to occur in
one place (e.g. in the core of a network) and trigger traffic shaping
and differentiation elsewhere (e.g. at the edges of a network). DIFFUSE
has applications in ISP, residential broadband and large corporate
network scenarios to name a few.

The FreeBSD Foundation is pleased to announce that David Chisnall has been
awarded a grant to implement xlocale APIs to enable porting libc++.

The C standard library (libc) is one of the most important parts of a UNIX
system as most programs interact with the kernel through interfaces written
in C. Porting code between platforms with similar libc implementations is
trivial and if something is supported by libc, higher-level languages can
use it without being reimplemented.

Over time, the C language has slowly evolved to modern multicore systems,
but there are still some places that are problematic. One of these is
localization as C began originally had no localization support. FreeBSD
libc and Darwin libc (used by Mac OS X) are similar, making it much easier
to port code from OS X to FreeBSD than from OS X to Linux. The libc used
by OS X supports a set of extended locale functions (xlocale) that allow
locale to be set on a per-thread basis.

Additionally, libc++, from the LLVM project, was originally developed on
Darwin, so it uses xlocale for most of the C++ locale support. The lack
of this support is the primary obstacle to porting it to FreeBSD.

Once xlocale is supported in FreeBSD libc, we can port libc++ to FreeBSD,
giving us an MIT-licensed C++11 standard library implementation. This, in
conjunction with Clang and libcxxrt, means that the entire C++ stack in
FreeBSD will be free of any GNU code. This leaves the linker as the only
significant obstacle to a GPL-free FreeBSD 10.

Tuesday, September 13, 2011

This Saturday, September 17, is Software Freedom Day (SFD). SFD is an annual global event that encourages open source software users to reach out to their local community to educate others about the benefits of using open source.

Frederic Muller, President of SFI, the non-profit organization behind Software Freedom Day, has been very helpful in encouraging FreeBSD users to participate in SFD. FreeBSD is listed as a partner on the SFD website. In addition, the FreeBSD logo is included on the cover letter and a copy of PC-BSD was included with the 210 packages that were shipped to the pre-registered teams. He also added the FreeBSD news RSS feed to planet SFD so that other SFD participants will get FreeBSD updates.

Julian H. Stacey has also been helpful in spreading the word and will be participating with the Munich team.

BSD user groups are encouraged to put together a team and to register it on the SFD wiki. If you have not participated in SFD before, take a moment to read through the Start Guide for instructions on how to promote your team. If you do register a team, leave a comment so that we can mention it on the @bsdevents twitter feed.