Friday, October 30, 2015

Deploying prebuilt binary software with the Nix package manager

As described in a number of older blog posts, Nix is primarily a source based package manager -- it constructs packages from source code by executing their build procedures in isolated environments in which only specified dependencies can be found.

As an optimization, it provides transparent binary deployment -- if a package that has been built from the same inputs exists elsewhere, it can be downloaded from that location instead of being built from source improving the efficiency of deployment processes.

Because Nix is a source based package manager, the documentation mainly describes how to build packages from source code. Moreover, the Nix expressions are written in such a way that they can be included in the Nixpkgs collection, a repository containing build recipes for more than 2500 packages.

Although the manual contains some basic packaging instructions, I noticed that there a few practical bits were missing. For example, how to package software privately outside the Nixpkgs tree is not clearly described, which makes experimentation a bit less convenient, in particular for newbies.

Despite being a source package manager, Nix can also be used to deploy binary software packages (i.e. software for which no source code and build scripts have been provided). Unfortunately, getting prebuilt binaries to run properly is quite tricky. Furthermore, apart from some references, there are no examples in the manual describing how to do this either.

Since I am receiving too many questions about this lately, I have decided to write a blog post about it covering two examples that should be relatively simple to repeat.

Why prebuilt binaries will typically not work

Prebuilt binaries deployed by Nix typically do not work out of the box. For example, if we want to deploy a simple binary package such as pngout (only containing a set of ELF execuables) we may initially think that copying the executable into the Nix store suffices:

and attempt to run the executable, we stumble upon the following error:

$ ./result/bin/pngout
bash: ./result/bin/pngout: No such file or directory

The above error is quite strange -- the corresponding file resides in exactly the specified location yet it appears that it cannot be found!

The actual problem is not that the executable is missing, but one of its dependencies. Every ELF executable that uses shared libraries consults the dynamic linker/loader (that typically resides in /lib/ld-linux.so.2 (on x86 Linux platforms) and /lib/ld-linux-x86-64.so.2 on (x86-64 Linux platforms)) to provide the shared libraries it needs. This path is hardwired into the ELF executable, as can be observed by running:

In NixOS, most parts of the system are stored in a special purpose directory called the Nix store (i.e. /nix/store) including the dynamic linker. As a consequence, the dynamic linker cannot be found because it resides elsewhere.

Another reason why most binaries will not work is because they must know where to find its required shared libraries. In most conventional Linux distributions these reside in global directories (e.g. /lib and /usr/lib). In NixOS, these folders do not exist. Instead, every package is stored in isolation in separate folders in the Nix store.

Why compilation from source works

In contrast to prebuilt ELF binaries, binaries produced by a source build in a Nix build environment work out of the box typically without problems (i.e. they often do not require any special modifications in the build procedure). So why is that?

The "secret" is that the linker (that gets invoked by the compiler) has been wrapped in the Nix build environment -- if we invoke ld, then we actually end up using a wrapper: ld-wrapper that does a number of additional things besides the tasks the linker normally carries out.

Whenever we supply a library to link to, the wrapper appends an -rpath parameter providing its location. Furthermore, it appends the path to the dynamic linker/loader (-dynamic-linker) so that the resulting executable can load the shared libraries on startup.

For example, when producing an executable, the compiler may invoke the following command that links a library to a piece of object code:

Patching existing ELF binaries

To summarize, the reason why ELF binaries produced in a Nix build environment work is because they refer to the correct path of the dynamic linker and have an RPATH value that refers to the paths of the shared libraries that it needs.

Fortunately, we can accomplish the same thing with prebuilt binaries by using the PatchELF tool. With PatchELF we can patch existing ELF binaries to have a different dynamic linker and RPATH.

Running the following instruction in a Nix expression allows us to change the dynamic linker of the pngout executable shown earlier:

According to the information listed above, two libraries are required (libm.so.6 and libc.so.6) which can be provided by the glibc package. We can change the executable's RPATH in the Nix expression as follows:

$ patchelf --set-rpath ${stdenv.glibc}/lib $out/bin/pngout

We can write a revised Nix expression for pngout (taking patching into account) that looks as follows:

A more complex example: Quake 4 demo

The pngout example shown earlier is quite simple as it is only a tarball with only one executable that must be installed and patched. Now that we are familiar with some basic concepts -- how should we a approach a more complex prebuilt package, such as a computer game like the Quake 4 demo?

When we download the Quake 4 demo installer for Linux, we actually get a Loki setup tools based installer that is a self-extracting shell script executing an installer program.

Unfortunately, we cannot use this installer program in NixOS for two reasons. First, the installer executes (prebuilt) executables that will not work. Second, to use the full potential of NixOS, it is better to deploy packages with Nix in isolation in the Nix store.

Fortunately, running the installer with the --help parameter reveals that it is also possible to extract its contents without running the installer:

$ bash ./quake4-linux-1.0-demo.x86.run --noexec --keep

After executing the above command-line instruction, we can find the extracted files in the ./quake4-linux-1.0-demo in the current working directory.

The next step is figuring out where the game files reside and which binaries need to be patched. A rough inspection of the extracted folder:

reveals to me that we have both files of installer (./setup.data) and the game intermixed with each other. Some files seem to be required to run the game, but the some others, such as the setup files (e.g. the ones residing in setup.data/) are unnecessary.

Running the following command helps me to figure out which ELF binaries we may have to patch:

According to the above information, the executable requires a couple of libraries that seem to be stored in the same package (in the same folder to be precise): libstdc++.so.5 and libgcc_s.so.1.

Furthermore, it also requires a number of libraries that are not in the same folder. These missing libraries must be provided by external packages. I know from experience that the remaining libraries: libpthread.so.0, libdl.so.2, libm.so.6, libc.so.6, are provided by the glibc package.

This executable has a number dependencies that are identical to the previous executable. Additionally, it requires: libSDL-1.2.so.0 that can be provided by SDL, libX11.so.6 by libX11 and libXext.so.6 by libXext

Besides the executables, the shared libraries bundled with the package may also have dependencies on shared libraries. We need to inspect and fix these as well.

Inspecting the dynamic section of libgcc_s.so.1 reveals the following:

We import the Nixpkgs collection so that we can provide the external dependencies that the package needs. Because the executables are 32-bit x86 binaries, we need to refer to packages built for the i686-linux architecture.

We download the Quake 4 demo installer from Id software's FTP server.

We automate the steps we have done earlier -- we extract the files from the installer, move them into Nix store, prune the obsolete setup files, and finally we patch the ELF executables and libraries with the paths to the dependencies that we have discovered in our investigation.

Apparently, the libgcc_so.1 library bundled with the game is conflicting with Mesa3D. According to this GitHub issue, replacing the conflicting version with the host system's GCC's version fixes it.

In our situation, we can accomplish this by appending the path to the host system's GCC library folder to the RPATH of the binaries referring to it and by removing the conflicting library from the package.

Moreover, we can address the annoying issue with the missing q4base/ folder by creating wrapper scripts that change the current working folder and invoke the executable.

The revised expression taking these aspects into account will be as follows:

(As a sidenote: besides creating a wrapper script, it is also possible to create a Freedesktop compliant .desktop entry file, so that it can be launched from the KDE/GNOME applications menu, but I leave this an open exercise to the reader!)

Conclusion

In this blog post, I have explained that prebuilt binaries do not work out of the box in NixOS. The main reason is that they cannot find their dependencies in their "usual locations", because these do not exist in NixOS. As a solution, it is possible to patch binaries with a tool called PatchELF to provide them the correct location to the dynamic linker and the paths to the libraries they need.

Moreover, I have shown two example packaging approaches (a simple and complex one) that should be relatively easy to repeat as an exercise.

Although source deployments typically work out of the box with few or no modifications, getting prebuilt binaries to work is often a journey that requires patching, wrapping, and experimentation. In this blog post I have described a few tricks that can be applied to make prebuilt packages work.

The approach described in this blog post is not the only solution to get prebuilt binaries to work in NixOS. An alternative approach is composing FHS-compatible chroot environments from Nix packages. This solution simulates an environment in which dependencies can be found in their common FHS locations. As a result, we do not require any modifications to a binary.

Although FHS chroot environments are conceptually nice, I would still prefer the patching approach described in this blog post unless there is no other way to make a package work properly -- it has less overhead, does not require any special privileges (e.g. super user rights), we can use the distribution mechanisms of Nix in its full extent, and we can also install a package as an unprivileged user.

Steam is a notable exception for using FHS compatible choot environments, because it is a deployment tool that conflicts with Nix's deployment properties.

As a final practical note: if you want to repeat the Quake 4 demo packing process, please check the following:

To enable hardware accelerated OpenGL for 32-bit applications in a 64-bit NixOS, add the following property to /etc/nixos/configuration.nix:

hardware.opengl.driSupport32Bit = true;

Id sofware's FTP server seems to be quite slow to download from. You can also obtain the demo from a different download site (e.g. Fileplanet) and run the following command to get it imported into the Nix store:

It's important to note that currently there is allegedly a bug in strip where it corrupts executables if patchelf is applied to them beforehand with a longer rpath than originally in the file. For now you can work around this by setting "dontStrip = true;" in your derivation.