Menu

Alpine Linux sucks for hosting Docker Containers

After setting up Alpine Linux to run docker containers and fixing a whole bunch of issues I encountered, I now came to the point where I tried to run a node.js app inside a Docker Container that runs on an Alpine Linux Host. But node immediately exited with a Segmentation Fault. After some experimenting, I went and just started the official node docker container in the latest version and started node inside. It still didn’t work.

First, some network config changes done by docker, then docker complains about missing support for nested cgroups, then some more network config changes before node crashes with a segmentation fault. And eventually some error messages because GRSEC prevents node from writing a core dump.

As with all Linux distributions, if I can’t get them to run properly after a few hours of trying, I throw them away and won’t touch them again for a few years. Now I have added Alpine Linux to that list. Fortunately, there are enough alternatives out there.

But since I’m not really interested of trying out many different Linux distributions, I’ll just go back to Ubuntu. Yes, it is still a ~600 MB download instead of an 80 MB one but at least it works most of the time.

11 thoughts on “Alpine Linux sucks for hosting Docker Containers”

FYI this is happening because Alpine doesn’t use GlibC but rather uses Musl so a lot of things need recompiling – switching to a glibc based Alpine will do the trick. Alpine does have a few quirks for new comers which can make it frustrating

I’m about to use Alpine Linux for a router, so I found this post looking for problems with Alpine before I dove in. I’m glad the author is talking specifically about Docker containers.

So this isn’t really a problem with Alpine more as it exposes some interesting things about Docker. Docker doesn’t run naively on Mac/Win and probably never will. Even with the newest Docker releases that use much better OS-based hypervisors (instead of the old VirtualBox based Docker Machine); they are still running under a hypervisor. I’d even argue that Docker doesn’t really work on Linux. It works on a really customized Linux kernel. For most people this doesn’t matter because installing Docker also installs all the correct kernel modules for it. If you’re running Gentoo/Funtoo, you typically have to go into your Kernel config and manually configure a lot of docker specific options (FYI: I run Gentoo and ran into this). Interestingly enough, Docker does seem to work naively on FreeBSD (although I haven’t tested it to see how/how well it works).

If you’re going to run Docker dev environments, I’d suggest a distro where it’s prepackaged with all the kernel dependencies. If you’re doing a larger deploy, I’d suggest looking into stuff like Kubernetes, DC/OS (Mesos) or other management layers for running large mappings of containers.

The problem you are having with node here has nothing to do with musl libc. It is the hardened kernel (PaX) that has a security feature that will prevent applications to allocate memory that is both writeable and executable at the same time (W^X). This improves security since it makes it much harder to inject code to be executed with a security vulnerability. However, this also breaks anything that uses JIT (just-in-time) compilers. Node, java, luajit are examples that will break. In Alpine we set attributes on the affected binaries that disables this security feature, which is why those works in Alpine. The node:latest image has not those attributes set so kernel will deny node to do its JIT thing.

There are multiple ways to work around this:
– you can use the linux-vanilla kernel package, which is a normal, non-hardened kernel
– you can disable the PaX feature systemwide with: echo 1 > /proc/sys/kernel/pax/softmode
– you can set the file xattr in your node container to disable mprotect:
FROM node:latest
RUN apt-get update && apt-get install attr && attr -s pax.flags -V m /usr/local/bin/node

I think none of those are optimal. I would like to find a solution where you can disable PaX features per container. But we are not there yet.

Alpine can be good for some things, but its documentation is lacking – things are missing, etc. I just tried to setup LXD/LXC and got a bunch of errors working through their official wiki trying to make a debian container. I apk-added all the lxc packages available and it still didn’t work. Then I noticed debootstrap was complaining about not having perl, so I installed that and it worked. We’ll see how it holds up.

Honestly, I got Alpine working pretty good as a hypervisor with qemu-kvm, libvirt, and ZFS, so that’s saying something. Don’t try and install kimchi, though … (shudder)

I hit on the same issue you had in this post and debugged this a bit more. The problem in my case had nothing to do with musl and everything to do with grsecurity and pax.
Node, specifically, will load data in memory pages to execute (JIT), this is prevent / stopped by pax.

For now my workaround (which is not ideal) is to set pax on the alpine container host in softmode, which will prevent segfaults in the containers. Another fix is to install paxctl in your container images and whitelist the ELF themselves (paxctl -cm /usr/bin/node). I found this too cumbersome and impractical in my case since my alpine docker hosts are used for development only.