Managing libraries with Nix

February 27, 2018

While learning Haskell and using its really smart library dependency management tools (cabal and stack), i realized that the C++ eco system has a problem: There are no handy established tools that let the developer declare which libraries (and versions) are required for a project which can then be automatically installed in a portable way. Nix however convinced me to be more versatile and powerful than Conan and handier than Docker, Vagrant, etc. (although it’s fair to say that i am mixing use cases here a little bit!) In this article, i am going to showcase this great tool a little bit.

This article is rather long (mostly because of many command line excerpts). Feel free to jump to the end, where we will compile and run the same project with 3 different compilers just by changing the command line a bit.

Use case example

So i have been developing a little library to see if i can implement a handy parser library in C++ that models how you build parsers in Haskell using the parsec library.

The project can be checked out on github.com/tfc/attoparsecpp. It is however not important to look into it. This article is not at all about parsers, Haskell, or my specific library. My little project shall just serve as an example project that has unit tests and benchmarks.

For libraries it is important that they build warning-free with:

different compilers

even different compiler versions

In addition to that, it is a nice-to-have to also compare the benchmark numbers among those!

That means: in order to build this project you need to install those. Some developers just install them the oldschool way or they pull them in as git submodules and then embedd them into the Makefile (or cmake pendants etc.). Some other developers would define Docker images (or Vagrant etc.). There is also the Conan package manager which enables the developer to just define which libraries are needed.

Installing Nix

Nix is a powerful package manager for Linux and other Unix systems that makes package management reliable and reproducible. It provides atomic upgrades and rollbacks, side-by-side installation of multiple versions of a package, multi-user package management and easy setup of build environments. …

Nix can be installed on Linux, Mac, and other Unixes. (I guess it can be installed in the Linux-Subsystem on Windows, but i am not sure as i am no Windows user). Just as a side note: There is even a Linux distribution called “NixOS”.

The installation of Nix is really simple. Please first study the content of https://nixos.org/nix/install and then run the following command in the bash (Or run the parts of the script you like. It is interesting how about 90% of the critisizm on Nix concentrate on this shell command.):

$ curl https://nixos.org/nix/install |sh

The installation script will download and extract a large tarball into the /nix folder on your system. In addition to that, it will activate a build daemon and create some user accounts in order to isolate things while building packages. After the installation, there will never be any need to use sudo in combination with nix calls again. It is generally possible to install nix on systems where even creating /nix is not allowed (see the installation guide for more details).

Installing project dependencies

After cloning the C++ project on a mac where only git and clang++, we will have trouble building the project without installing the libraries:

For the unit tests, we only need the catch library. That’s easy, as the Nix repository has that.

Another “catch” is that the make call ended up invoking c++, which is our Mac system compiler that is not really prepared to be used with Nix-installations of packages. This is part of some mechanics which the Nix docs cover much better.

Ok, the compiler can find the catch.hpp header, but it does not compile. The problem here is that i wrote the unit tests with catch2 and the package in the Nix repo is a little bit too old. I learned Nix the hard way by doing the following:

Writing our own Nix expressions

The Nix ecosystem provides a rich database of packages for everything which does not only include libraries, but also applications. In this case however, the library is too old and we need a more current one.

While the catch library just consists of a single header that is really easy to download, it is also a really simple example for building a Nix derivation that automatically obtains it from github and provides it for building. So let’s do that here. We will also install a newer google benchmark library (which is a bit more complicated as it is not header-only) this way later.

A Nix derivation is kind of a cooking recipe that tells where to get what and what to do with it in order to make it useful. In order to get catch version 2.1.2, we create a file catch.nix in the project folder:

There is a lot of voodoo going on here for anyone who does not know Nix. Nix is its own scripting language (a purely functional one), that is why it that script initially looks so complicated. The important parts are:

lines 8-11, fetchurl:

Where to get the catch header file?

For control reasons we also define what hash it needs to have.

lines 17-20, installPhase:

We define that catch.hpp needs to be installed into $out/include/catch/catch.hpp, wherever that is.

There was a lot going on after firing that command: Nix even installed curl and all its dependencies, because it needs a tool to download the header file. The last line tells us that there is now something in /nix/store/62g4h135grzi5xn5y7hyrxg1r8ac408g-catch-2.1.2. Let’s have a look into it:

There’s again some voodoo for Nix-novices, but the important part is that we call our package catch.nix in the context of a standard build environment (stdenv). With stdenv we don’t need to reference the compiler explicitly any longer.

Building the benchmark library involves compiling it with cmake, as it is more than just headers. Luckily, the Nix expression language came with its own library installed. It has handy little helpers like fetchFromGitHub that accepts some arguments needed to construct a download link from it and automatically unpack it!

The line nativeBuildInputs instructs Nix to install cmake for building this package. Everything else is automatically deduced. After adding this Nix derivation to our default.nix file, it will build google benchmark for us before we can run our own makefile:

The magic --arg stdenv "with (import <nixpkgs> {}); gccStdenv" line pushed a GCC build environment into the stdenv variable. The $CXX --version command in the --command part of the command line shows that it’s really GCC 6.4.0 now (instead of clang).

Using the same strategy, we can also run our benchmarks with all these compilers. We could even write a Nix derivation that actually does this and generates a nice GNUPlot chart from all benchmark runs.

Fallout

While playing around, we installed at least 3 different compilers and recompiled the google benchmark library for each of them. A nice thing about this is that this all needs to be done only once. The resulting packages can then be used again on the next invocation of a nix-shell environment. Even better: if another project happens to need the same compiler/libraries, then they are in place already! These things are shared system wide now:

So we can see that we have one version of clang, two versions of GCC, just one version of catch and three versions of googlebench. There is of course only one version of catch becasue it is just a header that does not need to be recompiled for different compilers.

But how does Nix know which googlebench installation belongs to which clang/GCC?

The long cryptic prefix of every package folder is the hash of its build configuration! The compiler choice is part of the build configuration, of course.

If another project has dependencies that overlap with ours in the sense that some dependency turns out to have the same configuraiton hash for the same package, then it will be shared. As soon as the configuration changes a little bit - another package is created.

Using --pure, we can check if our default.nixreally contains the complete list of dependencies. That feature is something most other dependency management tools don’t do for us.

This way it cannot happen that a project builds on one computer, but not on the other, just because someone forgot to install something else that is implicitly needed.

Summary

Nix just helped us with:

fetching, compiling, and installing dependencies including compilers and libraries

easily changing the compiler and version between builds

managing all those dependencies without interference

getting rid of it again

Maybe Conan would also have been able to do that (apart from the --pure feature). Nix however does not only work for C/C++ projects: It can be used for Rust, Haskell, Python, Ruby, etc. etc. - because it is a universal dependency manager.

Writing your own Nix derivations is only necessary if custom- or extremely new package versions are needed. It is also not hard to do. Existing packages can be rebuilt with different configurations, too.

Being completely amazed, i also installed NixOS on my laptop. What’s great there is that i am now able to configure the whole system with just one .nix configuration file. When that file changes, Nix automatically restarts only the affected services. If it does not work, then it is possible to roll the system back to a previous configuration. (Remember all the stuff in /nix/store/...? It’s still there until it’s garbage-collected!) The same system configuration could be used to clone the system elsewhere, etc.

This article is really just scratching the surface of Nix/NixOS’s possibilities.