Simplifying CI with Nix

So, I know I haven't written anything in a while, and that's because the last
semester was my heaviest yet. But now since that has passed, I no longer have
excuses to not write more articles. So go ahead, read on.

Before starting this article, if you do not know about Nix/NixOS, first go and
read up about it: NixOS. NixOS in my
opinion is the single most promising operating system ever written. Instead
of being FHS compliant, NixOS keeps all files under hashed directory names
under /nix/store so that files from different packages never have any
collision. This allows the package manager to install multiple versions of the
same packages, along with deterministic and atomic builds and upgrades.

Throwback

Now, I started using Nix a long time ago (~Sep 2015), often switching between
Arch and Nix depending on my usage. Mostly I had to switch to Arch every time I
couldn't figure out how to configure something in Nix (the methodologies, like
the ideoligies are very different). About an year ago I switched to Nix
full-time (i.e. no other operating system on dual boot). The reason being that
I have learnt enough about the workings of Nix to make it do my day to day
bidding without much effort.

Precursor

Every time I write a package for something, the installation procedure for
Nix is quite different from other operating systems. While most operating
systems would just ask you to compile and put the binary in /usr/bin, in
Nix you aren't allowed to change the global state of the OS. This means that
every package has to be accompanied by a config file that precisely lists all
dependencies and the build environment needed by the package. How the OS
provides that environment is none of our concern. If our package has a
dependency, I need to only write that in my package's config file, and even
without globally installing the package, my package would be able to build
against the dependencies as well as use them during runtime. It is like a
virtualenv except that it can be for any package, written in any language(s)
and that has dependencies of any language(s). It's how virtualenv would look
if it was 1000x smarter and more useful.

Since it simplifies development so much, I write my project's config file
before even starting the project. When new dependencies or requirements emerge,
it is a simple matter to modify a couple of lines in the config file. The best
thing about this approach is that my packages are always testable, even if I
reinstall the whole OS, I know that to run that package, I only need to run a
shell with that config file. No more iteratively installing dependencies based
on error messages.

Weren't we supposed to talk about CI?

Yes! Sorry.. I was building up to it. One of the major advantages of Nix is
that Nix is a package manager that can be installed on other operating systems
as well. Thus an existing Arch or Ubuntu or ... OS can actually benefit from a
lot of this NIXey goodness by installing the Nix package manager. And very
recently, struggling with getting the dependencies "just right" on Travis, I
decided to exactly this.

Wait.. Travis?

(Skip this if you know about Travis CI)

Travis CI is a hosted, distributed continuous integration service used to
build and test projects hosted at GitHub. Travis CI automatically detects
when a commit has been made and pushed to a GitHub repository that is using
Travis CI, and each time this happens, it will try to build the project and
run tests. This includes commits to all branches, not just to the master
branch.
The point of this is that you can often discover very quickly if your commit
broke something, and fix it before it becomes a problem.

And the problem was?

When writing a program, it is very easy to lose track of all its dependencies
that you installed to get the code working, or the libraries it links against
that are under a different name on your OS as compared to the OS running on the
CI. At some point, everybody finds themselves battling with the package manager
on the CI, iteratively running the tests again and again and fixing the next
error that is reported on failure of the build.

I ran into this problem when I was trying to set up the CI for my blog's code.
Yes, the website you are reading - that's what I am talking about. Since the
final compiled product is a static website, I used to compile it on my PC and
upload the compiled files for hosting. On my PC, the dependency management was
just fine, because - you guessed it - I had written a Nix config file for it.
Even so it was very hard to get it right on Travis, since it was running Ubuntu
and my blog uses a rather lot of dependencies (jekyll, as well as python NLP
libs for the search).

Solution!

Then I realised that I don't need to take this trouble. Nix always builds
everything deterministically. Which means that if a build config file works
fine on my laptop, it would work fine on Travis too. And since it is possible
to install Nix on Ubuntu, I decided to see if it'd be able to use my existing
config file to build my code using Nix instead.

The result was a small bunch of scripts that you can just curl directly from
any build that uses a default.nix (Nix config file) for its builds and the
Travis build would first install Nix and then complete the build using the
Nix config file.

Example

The above link would also be a good example of how to use the Nix CI from a
project that uses Travis CI. Hint: look into the .travis.yml file. Note that
you can directly curl the installation and configuration script from the above
repository directly into your CI job. The link below is a fine example of how
to do that.

Note that when sourcing from TravisCI, the script to run is install-nix.sh
but from GitLab CI, the script to run is install-root.sh. The latter is a
wrapper around the former but assumes that the user running the script is root
and the wrapper (install-root.sh) creates a non-root user with sudo
priviledges and runs the install-nix.sh script as the newly created user.
This is not required in Travis since the default user in Travis is already a
non-root user with sudo priviledges.

ABOUT THE AUTHOR

My name is Pallav Agarwal. I am a recent graduate of the
department of Computer Science and Engineering, Indian
Institute of Technology Kanpur, India. I love
experimenting with tech, and this blog is a way for me to
give a little helping hand to other's who are like me
(but don't know it yet).

I am ambitious, intelligent, competitve (sometimes too
much), loyal and brutally honest. People I respect the most
are teachers, which is partially why I myself like to teach
too. Apart from programming, I also like travelling,
adventure sports and trying new food items. If you like a
post, have a query, or just want to chit-chat, let me know
here