Raspbian makes the Pi go faster—and supplanted Fedora as the #1 OS.

Share this story

When you buy a Raspberry Pi, the $35 computer doesn't come with an operating system. Loading your operating system of choice onto an SD card and then booting the Pi turns out to be pretty easy. But where do Pi-compatible operating systems come from?

With the Raspberry Pi having just turned one year old, we decided to find out how Raspbian—the officially recommended Pi operating system—came into being. The project required 60-hour work weeks, a home-built cluster of ARM computers, and the rebuilding of 19,000 Linux software packages. And it was all accomplished by two volunteers.

Like the Raspberry Pi itself, an unexpected success story

Although there are numerous operating systems for the Pi, the Raspberry Pi Foundation recommends one for the general populace. When the Pi was born a year ago, the recommended operating system was a version of Red Hat's Fedora tailored to the computer's ARM processor. But within a few months, Fedora fell out of favor on the Pi and was replaced by Raspbian. It's a version of Debian painstakingly rebuilt for the Raspberry Pi by two volunteers named Mike Thompson and Peter Green.

It began because Thompson was ready for the next big thing. Thompson had been CTO and co-founder of Atomz, a hosted search service acquired by WebSideStory in 2005. Thompson got a "not unhealthy slice" of the $45 million or so that company sold for, and he eventually stopped working to spend a few years hanging out with his wife and kids.

A year ago, he was finally ready to get back into the technology field. Robotics is one of Thompson's primary interests, and the Pi looked like a great platform for that. It's cheap and small enough to be easily embedded into a variety of systems. But there wasn't an operating system fully optimized for the Pi's floating point unit, which is important in robotics projects and various other types of math-intensive applications.

"When I first learned of the Raspberry Pi, I was disappointed that none of the Linux distributions available for it would make use of the very fast floating point hardware that was present on the Pi," Thompson told me. "As a long-time Debian user, I was like, 'I'd rather see Debian [than Fedora]' and I wanted to see the floating point enabled because I have this longer-term interest in getting robotics working with these inexpensive boxes."

Debian had added floating point support for the ARMv7 processor, but not the ARMv6 processor used in the Pi. Debian "didn't see a product like the Raspberry Pi coming on the horizon. Even though ARMv6 in Pi has a pretty capable floating point unit, they didn't support it," Thompson said. Thus, "all the thousands or tens of thousands of software packages they built wouldn't support the Raspberry Pi."

Just as a graphics processing unit handles graphics operations very quickly, a "floating point unit performs all the math very quickly," Thompson said. "It's a peripheral that not every computer has, but when it does you really want to take advantage of it." The floating point unit is part of the PI's Broadcom BCM2835 system-on-a-chip.

If you don't take advantage of floating point capability in the hardware, a lot of mathematical operations must be performed in software, lengthening the amount of time it takes for the Pi to do useful work. This is important for robots, where complicated math operations are used for processing data from cameras and sensors, and for controlling motors with precision and speed, Thompson said. It's also important for multimedia processing, encoding music, physics simulations, or "pretty much anything that is numerically intensive."

A fruitful partnership

The path forward was clear to Thompson: rebuild Debian to run on the Raspberry Pi. This required shifting 19,000 packages from Debian to Raspbian—a monumental task.

Thompson didn't go it alone, though. He started a Raspberry Pi forum thread to talk to other people interested in bringing Debian to the Raspberry Pi. The thread caught the interest of Peter Green, a Debian developer and PhD student in the UK who went by the handle "plugwash" in the forums.

Green was a rare mix. He not only had the expertise to co-lead the project with Thompson, but he was also crazy enough to do it.

"I felt I was the only one of the people who were talking in the threads about the project that became Raspbian with enough Debian knowledge to make the rebuild a success," Green told me. "I'm sure there are other people within the Debian project who could have done it if they were interested and crazy enough, and there were many people in the Debian project who provided us with bits of help along the way."

When Thompson and Green got started, the Pi wasn't actually available yet. Even if it had been, it wouldn't be capable of rebuilding Debian in a reasonable amount of time. Thus, Thompson set up a cluster of eight Freescale iMX53 Quick Start Boards, each with 1GB of memory, a 1GHz ARMv7 processor, and (most importantly) SATA hard drives. One of the main reasons the Pi would be unsuitable for this type of work is the bottleneck introduced by USB storage, Thompson said. The Freescale boards were able to build Raspbian anywhere from four to 12 times faster than the Raspberry Pi would have.

Thompson spent nearly $3,000 on the cluster but recouped those costs through donations to the project. In addition to the single-board ARM systems, there is a separate Linux PC to serve as the repository for Raspbian builds. The repository server retrieves source packages from Debian repositories, schedules jobs for the Freescale systems, and collects the binary packages once the builds are complete. "The built packages are all staged in the repository and then synced with the external repository we maintain at www.raspbian.org where users pull their packages from," Thompson explained.

At first, Thompson used an ARM-based HP Media Vault MV5150 for the repository, but later upgraded to an Intel-based system when more horsepower was needed. Although each Freescale board has its own hard drive used for building packages, the main storage duties are handled by the repository server's 500GB drive, which is about two-thirds full now. Here's what the cluster looked like when he was first setting it up:

Enlarge/ The cluster would eventually include eight single-board ARM computers.

Mike Thompson

Thompson and Green weren't starting from scratch. Debian is already one of the most widely used Linux-based operating systems, obviously. Its ARMv7 port provided a solid foundation.

"We greatly leveraged the work done previously by the Debian Project to support floating point hardware on ARMv7 devices," Thompson said. "Other than the effort to actually build [19,000] software packages, 95 percent of the work done to port the software to support the Raspberry Pi was already done by Debian."

That's not to say the Thompson/Green work in bringing Debian to ARMv6 and its floating point unit was trivial, however. Green explained:

In Debian the compilers have default settings built into them. These default settings set things like the CPU family, the minimum CPU requirements, and the ABI [application binary interface] in use. Most packages leave those settings alone. We modified the compiler packages to reduce the default settings to ARMv6

For the majority of source packages rebuilding them with a compiler that uses our new defaults is enough to make them build ARMv6 code.

A lot of the original work was done manually, but Green eventually created auto-builder software to automate much of the process. Those auto-builders (which run in a chroot environment on top of Debian) are still running to this day, pulling updated packages from Debian repositories and automatically re-compiling them for Raspbian.

Those auto-builders also detect when there's a problem that prevents the package from being automatically rebuilt. Green explained in an e-mail:

There are several things that could cause a package to come out still containing ARMv7 code.

We hacked together a script (using readelf) to check for ARMv7 code in packages so we could tell which packages... needed further attention to eliminate it. ARMv7 contamination can come from several sources.

Static libraries [are one example]: these are taken from the build environment and incorporated into binaries. So if the static library contains ARMv7 code then it will go into the resulting binary. This was a big pain during the early days of the project as we had to use packages from Debian armhf [Debian's ARM floating point port] to break dependency cycles, but those packages could contain static libraries with ARMv7 code in and sometimes it took several attempts at rebuilding and installing different packages to figure out which library the contamination was coming from, and replace it with an uncontaminated version. However, this isn't really a problem anymore since having got past the bootstrapping problem we can nearly always build packages in a clean Raspbian environment.

Raspbian became available for download on April 20, 2012 in limited form, containing only about five percent of the Debian packages. "Enough to build a root filesystem that would boot to a command line," Thompson said.

They didn't really DO anything new though, they just recompiled packages mostly automated at that. I just don't see what the big deal is...

Thats a rather clueless statement.

Have you even looked at how much work especially testing it takes to build your own distro or fully customised image and maintain it ?

Its not that simple. Some packages port with no to minimal work, others can be heavily dependent on other arch and need a lot of work to clean them up or rewrite the build scripts and makefiles.Linking can be a real pain and getting things in the correct build order to avoid dependency problems can drive you nuts at times.

What is impressive is that its voluntary work , done in a short time period and is well maintained.

The advantage of this approach is that you don't need to worry about installing any header files or libraries on the distcc host, because the C preprocessor and all linking is handled by the caller. The disadvantage is that distcc running in this mode won't accelerate C preprocessing or linking. :-)

Unfortunately, compilation is only part of the issue. One of the other big issues is that linking very large packages such as Ice Weasel, Open Office and various other large packages requires gobs of memory -- on the order of 2GB to 3GB or more. Our build systems have only 1 GB of RAM so builds would dramatically slow down when the linker started to hit swap space. Such package builds could take 2 to 3 days because of the slowdown in swap.

Since you can't expand the memory on the Arm boards to eliminate the swapping have you tried adding an ssd equipped board to see how much that speeds up the build process for these large packages? It would probably still take a while but it could also provide a decent speed up and it shouldn't be hard to automatically direct the compilation of these large packages to a specific ssd equipped node.

Great article overall and many thanks to Mike and Peter for elucidating on why they made the choices that they made here within the forum. Kudos to you two on bringing this project from an idea into functional reality. Very admirable!

The advantage of this approach is that you don't need to worry about installing any header files or libraries on the distcc host, because the C preprocessor and all linking is handled by the caller. The disadvantage is that distcc running in this mode won't accelerate C preprocessing or linking. :-)

Unfortunately, compilation is only part of the issue. One of the other big issues is that linking very large packages such as Ice Weasel, Open Office and various other large packages requires gobs of memory -- on the order of 2GB to 3GB or more. Our build systems have only 1 GB of RAM so builds would dramatically slow down when the linker started to hit swap space. Such package builds could take 2 to 3 days because of the slowdown in swap.

One option I've had partial success with, is on a beefy PC server: a combination of qemu-arm-user and binfmt-misc. This way an x86 can execute ARM binaries directly.. chroot into your (100% arm) build env and 'gcc' is actually a native ARM compiler.

Of course this will always be slower than cross-compiling, but depending on how weak your target HW is (as noted not just CPU, but RAM amount and storage speed can play a large role in links) and how tought your codebase is to cross-compile, then this can be a nice middle-ground.

I brought up a Gentoo rootfs by unpacking their ARM stage tarball and chrooting into that on my server, and building all the stuff in there. Then installing onto an SD card for my device to run and it worked nicely.

EDIT: qemu-user is not a full machine emulator! It does emulate the ARM instructions (with all the usual basic block JIT and caching) but upon a system call will pass through to the native linux kernel, doing any 32<->64 and tweaks to other syscall arguments as necessary.

And with binfmt-misc properly setup to use qemu-arm-user as an interpreter form ARM ELFs, you can literally ./some-arm-binary from your x86 shell and it works, as does chrooting into a dir where /bin/bash itself is an ARM binary.

One option I've had partial success with, is on a beefy PC server: a combination of qemu-arm-user and binfmt-misc. This way an x86 can execute ARM binaries directly.. chroot into your (100% arm) build env and 'gcc' is actually a native ARM compiler.

Yeah, i'm aware of that option I ended up resorting to that for manually building a chromium package which ran out of address space recently. It solves the ram issue and improves the address space situation but it means you end up with a build system with the processr speed of a slow arm and the power consumption and cost of a beefy x86 so I don't think we will be going down that route for our autobuilders.

They didn't really DO anything new though, they just recompiled packages mostly automated at that. I just don't see what the big deal is...

Thats a rather clueless statement.

Have you even looked at how much work especially testing it takes to build your own distro or fully customised image and maintain it ?

Its not that simple. Some packages port with no to minimal work, others can be heavily dependent on other arch and need a lot of work to clean them up or rewrite the build scripts and makefiles.Linking can be a real pain and getting things in the correct build order to avoid dependency problems can drive you nuts at times.

What is impressive is that its voluntary work , done in a short time period and is well maintained.

Rewriting makefiles, really now? So, by that logic, I should have an Ars article written after me...

Gotta love it when the do-nothing boils on the arse of humanity come out to say "you shouldn't have created that that way!"

Go do something useful to ANYONE and report back to us.

Edited to add: just visited the Raspbian home page and donated enough for a few pints. Thanks, folks, for making something that a Pi noob and Debian OG user can feel right at home with!

Unfortunately, compilation is only part of the issue. One of the other big issues is that linking very large packages such as Ice Weasel, Open Office and various other large packages requires gobs of memory -- on the order of 2GB to 3GB or more. Our build systems have only 1 GB of RAM so builds would dramatically slow down when the linker started to hit swap space. Such package builds could take 2 to 3 days because of the slowdown in swap.

Ouch. It seems like you'd have better luck using a Nexus 4, or another phone with 2 GB RAM, and doing builds on NFS-mounted partitions via Ethernet over USB (+ the internal SD storage).

Do the Debian packages give you a choice between binutils ld and gold for linking the big packages? I know that gold was written to speed up Chrome builds, but I think it needs a lot more RAM than regular ld, so it might be a good idea to double check that your builds aren't using it accidentally, although I suppose you would've noticed in the ps output if you were using gold by mistake.

Another thing you might want to try, if you haven't already, is reducing the level of optimization in the build for these big C++ programs, to do a little bit less inlining and bring down the size of the object files you have to link. Reducing -O3 to -O2 would be a good starting point for any packages that were using -O3 (which turns on function inlining), but you could also try -Os, which is like -O2 but optimized for size.

I'm a little confused. Couldn't you just modify the make files to compile for Arm6?

The Make files will parse to appropriate hardware IF the parameters for ARM6 were written into the options files. The article indicates that support for ARM 7 was in the original Debian files but not the ARM6.

I bit the bullet and bought one, and I can honestly say, for $35, they are goddamn impressive little devices.

The community that has literally blossomed from these little things is amazing. The people around it are top notch. Its the first time in my life I've tackled programming and not felt like I was running up a hill of spikes with my head on fire.

Really, they are great devices, spend the money, set one up, they're a ton of fun

Since you can't expand the memory on the Arm boards to eliminate the swapping have you tried adding an ssd equipped board to see how much that speeds up the build process for these large packages? It would probably still take a while but it could also provide a decent speed up and it shouldn't be hard to automatically direct the compilation of these large packages to a specific ssd equipped node.

The good news is that ARM based servers with greater than 1GB of RAM are becoming available and Peter is keenly watching each new release of hardware to see if it has the magic combination of features that will help build the next version of Debian for the Raspberry Pi -- Jessie.

I myself hadn't considered SSDs, but it is something to think about as they become more affordable. Fortunately, build servers could easily get by with 64GB or smaller SSDs which could help contain the costs as Peter considers what the next build farm will look like.

Could you trick the PI into thinking it is doing a native build and using something like distcc to 'distibute' the compiles to an x86 based cross compilers?

This can indeed be done. This is a fairly common approach taken by individuals who want to build large software applications on their Pi and my understanding is that it can work pretty well. There are various instructions with examples of people doing this in the Raspberry Pi forums. However, for a full Debian port it seemed to best stick to the existing path of doing builds on native ARM hardware so we could focus efforts on resolving Armv6 vs Armv7 differences and not throw the added complexity of remote x86 cross compilers into the mix.

@mpthompson: Have you looked into using a rooted android phone for the memory hungry projects?

A nexus 4 would have 2gb of memory and a Krait CPU (ARMv7 so not any worse than what you have now and probably faster) - should work just fine and even if none of you has one, I'd think you could easily find some volunteers. Obviously in that case not a long term solution, but for things like webkit at the moment this may severely speed the compilation up.

They didn't really DO anything new though, they just recompiled packages mostly automated at that. I just don't see what the big deal is...

Well, the major issue for me was that I wanted a version of Debian for the Raspberry Pi that would make the best use of the limited resources on the device. Unfortunately, for various reasons, the folks at the Debian project weren't willing to create such a port on their own. I can fully understand this as they just finished the herculean task of creating Debian armhf (for the Armv7+ architecture) and early last year the fate of the Raspberry Pi was still enough of an unknown to justify expending resources going towards other projects within Debian. I started asking questions about what would be involved with creating an unofficial port of Debian and the project just snowballed from there. I was very fortunate that my efforts caught Peter's attention and he joined up and brought his Debian expertise to make the project possible.

I may be tooting my how horn here, but I believe the big deal is that two volunteers were able to leverage the existing open source resources of Debian and with quite a bit of our own time, effort and elbow grease able to create a distribution that is used by 100,000's of users of the Raspberry Pi. While I've been a professional software developer for 27 years, I've never done anything quite like this project before and the results have been very gratifying. The same probably goes for Peter as well. We both learned a lot and in the process were able to make a positive contribution to both the Raspberry Pi and Debian project. I sincerely hope that our efforts with Raspbian will expose 100,000's of newcomers to the Debian project and these new users will have last repercussions on Debian far into the future.

I appreciate Jon Brodkin and Ars Technica for being interested in how and why we created Raspbian. Hopefully the story of our efforts will encourage others to make positive impacts on open source projects that interest them.

@mpthompson: Have you looked into using a rooted android phone for the memory hungry projects?

A nexus 4 would have 2gb of memory and a Krait CPU (ARMv7 so not any worse than what you have now and probably faster) - should work just fine and even if none of you has one, I'd think you could easily find some volunteers. Obviously in that case not a long term solution, but for things like webkit at the moment this may severely speed the compilation up.

The problem with phones is the lack of interfaces. Usually USB is the only interface available and worse it's the same port used for charging so you can't easilly keep the phone on charge while using it to build stuff. The USB battery charging specification defines an "accessory charger adaptor" which is supposed to solve this but I've never found anywhere actually selling one.

It takes a while for new features to trickle down into affordable hobbyist boards but it is happening. I have bought an odriod U2 to check out how well it worked for building stuff and to use for manaul builds of massive packages. It's a heck of a lot faster at building stuff than the imx53 boards we use (less than a day to build webkit) but i'm wary of selecting it as our next autobuilder because of the reliance on USB storage.

There is also the arndaleboard which has 2GB of ram, a nice fast coretex A15 CPU and SATA but it's bulky, relatively expensive and seems to be somewhat backordered at the moment. The openbrix I mentioned during the interview is basically a stripped down arndaleboard and assuming everything (software support, documentation, stability, availability etc) works out it should make a nice autobuilder.

Freescale has the i.MX6 which supports 2GB of ram and has SATA but unfortunately so-far most i.MX6 boards have only shipped with 1GB of ram. However boundary devices have recently announced a 2GB version of their "nitrogen6x" board. Right now I'd say this is probablly plan B if the openbrix doesn't work out.

Ti have also been talking about a successor to the pandaboard with a better CPU and a SATA interface but I haven't seen any concrete specs, prices or availability dates yet.

Debian is in freeze at the moment so our autobuilders aren't all that busy right now. Add that to the fact that I beleive better hardware options are just arround the corner and I don't belive now is a good time to go buying our next autobuilder cluster.

P.S. I looked at the arm server hardware but the openblocks and boston stuff is too expensive for what you get, the HP and dell stuff doesn't seem to be publically available and baserock didn't respond to my enquiry (though things i've read would seem to indicate that it's also too expensive for what you get).

P.P.S If anyone is aware of any reasonablly affordable (under $300 shipped) arm boards with 2GB of ram and SATA that I have not mentioned above please do get in touch.

The problem with phones is the lack of interfaces. Usually USB is the only interface available and worse it's the same port used for charging so you can't easilly keep the phone on charge while using it to build stuff. The USB battery charging specification defines an "accessory charger adaptor" which is supposed to solve this but I've never found anywhere actually selling one.

Rampant uneducated speculation: what about the phone's Wi-Fi? I've got Samba running on my rooted HTC DINC2 so I can get at the SDcard from my PC without plugging in or fooling around with web-based GUIs. I haven't looked into android NFS servers, but it seems like it should be possible without too much hacking. Or is throughput unacceptable?

If anything, the fact that Debian does not support cross-compilation should be a huge negative against using it both as the preferred platform and especially as a development environment or target.

Essentially, that places every developer in the silly spot of having at least 2 PI's in order to get much done. This gets even worse when you consider that storage speed is typically the biggest slow down to compiling with larger programs / lots of libraries (devs limited by the speed of an SD card - WTF!). Further, why should developers be restricted to the limitations of PI, a machine specifically targeted at the "just enough" computing power paradigm, for compilation, when most will have something with 2-8 cores @ 2.5+ GHz and 4-32 GB of RAM sitting impotently next to it.

I'm not sure exactly what you're critiquing about the project but it seems like you're comparing developing on a pi and developing on a i7 machine. Yes, there will be differences, they are for different uses. The primary purpose of the pi was to be used to teach people programming, not a primary development machine.

It sounds like he's being critical of the choice of Debian, as it doesn't fully support cross-compilation, but of all the distros available to these guys, Debian was A) mostly there, and B) what they already knew.

And the fact that these guys are doing the compiles means that most end-users won't have to. That's kind of the whole point of it.

I'm not aware that anyone else is doing cross-compiles any better than Debian, so I'm not sure what else they were supposed to choose. And as far as whether improving cross-compilation was important, it strikes me that rearchitecting all of Debian that way might be harder than just manually porting everything, and then just pushing the patches upstream.

It's not like this is going to happen very often... this ARM transition is probably a very brief time period, during which the available development hardware is really terrible. In another year or two, the machines on sale should be fast enough, and have enough RAM, to support reasonable compilation times. Tearing apart and rebuilding all of Debian for an event that may, in all seriousness, never happen again is probably not the best time investment.

If there were five or six more architectures in the offing, then building the systems to make ports easy might make sense, but with only one, it just makes more sense to do it manually. Why automate something that you're not going to repeat?

Good writeup and two Mr. Green and Mr. Thompson, good job. As too the need for build hardware I hate to ask but have you thought about using emulated hardware for the build? The Pi is not very fast so maybe an I7 could emulate it at near full speed. You could use that machine to as a build machine and make a virtual PI with 4 or more gigs of ram and a super fast SSD drive. Just a suggestion. I know the closed parts might not cause issues but you don't need to emulate those just to build the OS.

Good writeup and two Mr. Green and Mr. Thompson, good job. As too the need for build hardware I hate to ask but have you thought about using emulated hardware for the build? The Pi is not very fast so maybe an I7 could emulate it at near full speed. You could use that machine to as a build machine and make a virtual PI with 4 or more gigs of ram and a super fast SSD drive. Just a suggestion. I know the closed parts might not cause issues but you don't need to emulate those just to build the OS.

Emulation was considered (as opposed to cross compilation for the reasons described above in previous postings) and that is how some of the earliest packages were built and tested for Raspbian on my desktop PC, but QEMU was still pretty slow even on decent hardware. After I got the initial iMX53 QSB system to test out some builds, it proved to be much speedier and less hassle to work with than emulation It became pretty natural to just follow in the Debian armhf footprints and replicate their work of building Debian armhf for armv7 for building Raspbian using a cluster of iMX53 QSB systems. As I believe Peter mentions above, emulators do a good job of replicating the ARM instruction set on Intel systems, but the fidelity of actually fully emulating an ARM hardware environment is somewhat questionable and there are packages that may have broke in strange ways when built in an emulator. It wasn't something worth messing with since we had the option to build on real ARM hardware.

I believe all the current work in Debian for the arm64 architecture is currently being done in emulators, but I'm almost certain they'll switch to real hardware for that architecture once it's available.

It is very gratifying to read this story here. I remeber very well when we in the community were hoping / lobbying the Raspberry Pi foundation that Raspbian would be picked up as the 'official' distro for the Pi. It is a testament to your hard work with Peter what was accomplished. Raspbian made a huge difference to the usability of the PI for the avergage user. As you said, 100,000s of users have benefited from the work what was accomplished by just a few.

Thank you for 'ride' in the first few months... I still use MATE today ;-)

I don't frequent the Raspbian forum so much anymore but it would be very nice to see an update every once in a while what you are up to.

Sorry to be so late in this discussion but I've been wondering if the work that these guys did configuring the Debian autobuilder to work with the Raspberry Pi could be used as a starting point, by normal non-heroic moderately skilled programmers, to build Debian for new boards.