Compile Node.js apps cross-platform — Part 1

These days it’s not uncommon to see several links popping up on Hacker News regarding Node.js compilation stacks. There are several solutions balanced between easy-to-use and portability. One of the most easy to use, which already supports cross-platform compilation (which means the ability to create binaries to and from different Operating Systems and Processor Architectures) is pkg. The downside is that it does abstract filesystem APIs and some useful (often inevitable) things, like dynamic import, do not work:

This import would not work on pkg compiler and the majority of node.js compilers out there.

To be able to compile node is a good thing, although several solutions out there still miss some pieces and they are not battle-tested in production.

Also to be able to compile a node.js application the developer must be familiar with low-level programming because when things will go wrong you will have to debug and put in place policies that apply to a binary, not to node.js itself (although before the compilation you can actually use whatever you like, respecting your compiler restrictions — some of them do not support native modules).

Here at Promotely, Node.js covers several services used in production, and the ability to be able to compile it would just mean an easier to maintain production system and isolated, self-encapsulating node installations. It means we are able to compile from source Node.js with whatever flag is more appropriate to our use case (with both linked or static libraries) and avoid to install and maintain system wide (or virtualized/containerized environments) Node.js executables. With a compiled node.js binary, node itself is your application, at the expense of dealing with lower levels and bigger application sizes. You can also more easily enforce policies to update binaries whenever a new version of node is available and give your teams the ability to deploy updates independently with proper testing (so you won’t need to update things on the system and deal with process restart, each team is responsible for it’s own environment).

Meet node-compiler

Terminal simulation of a simple node.js compilation

Node-compiler is a nice, self-contained compiler developed by Minqi Pan. We first looked into it because the installation of other tools is usually done via npm, making it difficult to put the final executable under version control (yes, we do put compilers in our repository in order to allow reproducible builds). It was also matching an internal experiment we were working on, meaning it takes a different approach. Rather than patching node.js APIs it creates a virtualized environment, allowing node.js to work as is by patching the system at a lower level, in this case we’re talking about it’s ability (through SquashFS) to virtualize the filesystem. It does this by mounting the filesystem in RAM, moving your sources in there with other needed stuff, and compile everything inside the node executable. This way when node starts up you are effectively starting your script with it.

The building process doesn’t differ much from what you’d do to compile a plain node.js from source, but you do this through a ruby script (at the time of this writing). In fact it builds node from source rather than starting from pre-built binaries, this makes the process slower (which is also good since you will likely find answers to compilation issues directly into node core community, which is way bigger than any other node.js compiler project).

Awesome, so what’s the issue with it?

At the moment it builds for your system. Period.Meaning it does not support cross-compilation.

Since we need a feature like that because we want to be able to compile for different production targets from our CI/CD environment we also want to be able to set the destination operating system and processor optimizations (architecture). We knew node.js is already capable of cross-compilation so we started tweaking node-compiler in order to get the same support.

It was pretty easy to add that kind of support inside the ruby library, but then we started digging with a reliable to compile to and from any platform to any platform (disclaimer: we’re talking about gnu-supported platforms).

This became much more slow because each test in order to get a working toolchain and then compile node.js takes a lot of time (~30 minutes each).

It seems at this point we’re on the good way to be able to support compilation to and from mac os x and linux platforms (older to newer versions) but we’re still working on it.

As soon as we’ll be able to get it working we will write down step by step our process and we’ll also try to make it straightforward for others as we thing it may be a useful thing for other companies and individuals too.

We really want to thank @pmq20 for it’s awesome idea, which opened doors to new ways to compile Node.js applications without having to modify existing code. We are working on extending it to support this kind of feature, and we’ll also be working on other issues the along the way to make it stable. We decided to stick with node-compiler because the freedom of our developers is the most important thing to us, we don’t want to tweak source code or pass it through a transpiler of some sort before to hit production, as things can easily break during automated pipelines.