I have a certain friend who will remain nameless. We know each other's wives and kids, and we have had a number of adventures together. At work, he manages a team doing mostly kernel development and optimization work. He has known about the Yocto Project from the beginning, and we occasionally chat about the work.

But since he is a friend, he feels safe to taunt me occasionally, make fun of me a little. All in good fun of course, but one of his consistent jabs about Yocto goes something like this:

"Isn't it just a build system?"

Of course, the Yocto Project is much more than the build system, but it is indeed at the core of what makes Yocto (and Poky and OpenEmbedded) such a powerful and complex system.

My friend's unspoken implication is that build systems are just not that challenging, compared with kernel work. After all, we know that the top people are doing kernel work, blah blah blah, whatever.

Indeed, he and I were both working at a company 25 years ago which did innovative kernel work and had a great cross-compiling parallel build system. Once the build system was done, it pretty much worked without any help, while the kernel needed constant attention.

But it seems to me that today the tables may be reversed, and the work on the build system in Yocto may actually be as hard as kernel work and in some cases, much harder and more interesting.

Back 25 years ago when we did cross builds, but we had only a single build OS to support and two instruction set architectures to build for. All of the source was available locally on the build machine or over an NFS filesystem, and once the compile was done, the resulting binaries and headers could be installed directly into a filesystem.

Contrast this with what Yocto must accomplish:

The build system has to pull sources from a huge number of projects all over the internet. Sources might need to be pulled out of a source control system like git or subversion. They might need to get pulled from a tarball sitting on an FTP server.

Besides pulling the sources over from other servers, we would like to be able to patch those sources before we compile them. Hey, if you have source available for your OS, it's just not as useful if you can't make changes to it, perhaps to add features or change behavior. And by saving these changes as source patches, we can often just apply them to newer versions of these sources as well. So the build system needs to apply patches after they are obtained.

The system doing the build could be running any number of different operating systems. In this case, "Linux" is really multiple operating systems, since each Linux distro has its own policy for which commands are present and what their features are.

Because of the different OS's which may be present on the build system, we need the build system to check carefully that there is no build host contamination in the resulting OS that we build. This is because we need to be sure that an archived build can be rebuilt on a different OS.

There are four instruction set architectures (ISA's) to cross-compile for, including ARM, x86, PowerPC and MIPS. Well, x86 is really both 32 and 64 bit, so I often say, "four and a half architectures." Even within a given ISA like ARM, there are actually a number of system variations which require customization to get the system to work. And there are a few common Linux commands which are extremely unfriendly to cross- compiling. (One example is PERL)

Besides building the operating system, we need to build an emulation environment to enable us to test out our new OS and apps without needing real hardware on hand.

Instead of just installing binaries in a target directory, we actually need to repackage our binaries and headers using some common Linux package format. And here as well, there is no single "right" answer, and we have to support multiple packaging formats such as RPM, DEB and IPK.

Finally, many of these things are constantly moving targets. The upstream source projects keep updating and changing, package formats are being developed, there are new hardware platforms to support constantly and Linux distros (build hosts) are all being developed and updated constantly. It's enough to drive one mad!

These are just a few of the challenges for the Yocto build system. It makes life exceedingly tricky and challenging for those who want to maintain and develop the build system.

Now, contrast this with kernel work. I don't mean to suggest for a minute that the kernel is "done", not by a long shot. But a lot of the interesting or hard problems have already been solved by Linux. So Yocto may be a more interesting challenge for folks.