A Popper Template for Linux Kernel Research (part 1)

TLDR: We describe how to follow the Popper convention to manage
and automate experiments that modify the Linux Kernel. This first part
deals with building the kernel and provisioning a VM. In part 2, we
cover the experiment execution and validation of results using this
template.

A typical exploration involving Linux consists of code that implements
one or more new features in the kernel, and then run experiments in
order to test a hypothesis. In this case,
Popper is followed to manage and track the
changes done to the experiment, analysis and validation of results.

In this post we will show how to hack the kernel in a DevOps oriented
way by following Popper, with the goal of making it easier for others
to reproduce results.

The above diagram depicts the series of steps that we will go over in
some level of detail:

Create a repository for your experiment (0).

Fork the kernel repo and clone it into your machine (1, 2).

Build new kernel packages (3, 4, 5).

Provision a VM with the custom kernel (6, 7).

For 3 and 4, we will create scripts and, for illustration purposes, we
will write these scripts from scratch. These scripts are part of a
Popper template, so one can avoid writing them from scratch by quickly
importing it to an existing Git repository (see last last section).

Creating a Repo for the Experiment

We want to create a repository that will hold the scripts, tests,
analysis, results and possibly a manuscript describing all. In order
to do that, we first initialize a git repository:

We then create a repository at GitHub and link it to the local
repository that was just created. To do this, you need an account at
GitHub (see a guide
here).
Once this is done, the repository is linked by doing the following:

$ git remote add origin $your_repo_url

where $your_repo_url is the URL to the repository at GitHub.

Add the kernel source repo

We now will add a kernel source repository as a “sub-repository” to
the mypaper repo that we just created above. To do so, we first fork
the kernel source repository at https://github.com/torvalds/linux.
To fork the repository, you need an account at GitHub. For more info
on how to fork a repo, take a look at this
guide.

The --depth 1 flag passed to the submodule add subcommand
instructs git to make the submodule shallow so that the whole repo
isn’t pulled in. To learn more about submodules, see
here,
here or
here.

Build Kernel Packages

Now, we want to build the kernel in a reproducible manner. One way to
do this is to use Docker. We create a
experiments/kernel-experiment/docker folder and create an image with
all the dependencies of our build environment. Instead of pasting the
contents of this folder, you can take a look at them
here.
To learn more about Docker visit this
page.

This container automates all the steps involved in building a kernel.
To build, we do the following:

The script above is placed in a
experiments/kernel-experiment/build-kernel.sh script and becomes
part of the mypaper repo.

Provision a VM with new Packages

In order to test our kernel, we’ll use a VirtualBox VM. Instead of
interacting directly with the VirtualBox commands, we use the awesome
Vagrant tool. We first grab a vagrant box that runs Debian Jessie.
This is done by defining a Vagrantfile like the following:

The above script can be placed in a
experiments/kernel-experiment/vagrant/ folder. The output of the
building process described in the previous section generates packages
and puts them in this experiments/kernel-experiment/vagrant/debs
folder. In order to provision the VM with these packages, we do:

Again, we put the above script in the
experiments/kernel-experiment/vagrant/provision.sh script so this
becomes part of our mypaper repository. After the above executes, we
have a VM running with our customized kernel. To login to the VM, we
can do:

Importing the Popper Template

The scripts that we created in the previous sections are part of a
Popper template, so one can avoid writing them from scratch by quickly
importing the template to a repository. To do so, install the
Popper CLI and, assuming you have already create the mypaper repo
(first subsection above), we do:

$ popper init
$ popper experiment add linux-cgroups my-linux-exp

You can ignore for now why this template is named linux-cgroups.
(after reading part 2 this will make more sense). The contents of the
experiments/my-linux-exp would look like the following:

Next, we add the submodule for the Linux repository as we did in
above in a linux/ folder. After this, the
.build-kernel.sh command will build kernel packages in
vagrant/debs. The provisioning of the VM is done in run.sh.