Give Codeship a try

Want to learn more?

The Node.js community has embraced process monitoring tools such as PM2, Nodemon, and Forever, which is understandable. For example, in addition to process monitoring, PM2 also boasts features around logging and port-sharing or clustering.

However, I’m a firm believer in using the Linux init system for process monitoring. In this blog post, I’ll show you how to recreate process management, logging and clustering functionality using the Linux init system, systemd, and I’ll make the case for this being a superior approach.

Please note that I’ve no intention of casting aspersions on any of the tools I’ve mentioned. But I think gaining familiarity with Linux is important for Node.js developers; it’s important to use standard tools that are well-proven and widely understood by sysadmins everywhere.

A Note about PM2

I will be making reference to PM2 because it has become ubiquitous in the Node.js community, and therefore it will serve as most people’s frame of reference. PM2 makes it very easy to do:

Process management

Log management

Port-sharing magic for Node.js applications

PM2’s ease of use is certainly one of its strongest points; it hides some of the operational realities of running services on Linux from Node.js developers. In this blog post, I’m going to show you how to do each of these three things with systemd.

An Explanation of Linux Init Systems

Although PM2 and similar tools are ubiquitous in the Node.js world, that’s not necessarily the case in other communities. Running your application with the Linux init system will ensure that it’s familiar to any Linux sysadmin. Therefore, knowing more about Linux, the operating system on which the vast majority of Node.js applications run, is very important for Node.js developers.

First, let’s run through a brief primer on what Linux init systems are.

Each Linux distribution has a master process running as PID 1 (process ID 1) that is the ancestor of all processes that run on the system. Even if an application spawns a bunch of child processes and orphans them, the init system will still be their ancestor and will clean them up.

The init system is responsible for starting and stopping services on boot. Typically, sysadmins will write init scripts to start, stop, and restart each service (e.g., databases, web servers). Basically, the Linux init system is the ultimate process monitor.

systemd is more or less the standard Linux system in the latest release of most Linux distributions, so that’s the one I’m going to cover here. It should be relatively easy to translate these concepts into another init system, such as upstart.

Creating a Sample Node.js Application

To aid explanation, I’m going to use a simple, contrived Node.js application that talks to Redis. It has one HTTP endpoint that outputs “Hello, World!” and a counter taken from Redis. It can be found here:

This is failing because Redis isn’t running. Let’s explore dependencies in systemd!

Exploring systemd Dependencies

We can add the Wants= directive to the [Unit] section of a unit file to declare dependencies between services. There are other directives with different semantics (e.g., Requires=) but Wants= will cause the depended-upon service (in this case, Redis) to be started when our Node.js service is started.

Process Management

The first item of PM2 functionality we’re working toward is process management. This means restarting services when they crash and when the machine reboots. Do we have this functionality yet? Let’s find out.

So systemd is not restarting our service when it crashes, but never fear — systemd has a range of options for configuring this behavior. Adding the following to the [Service] section of our unit file will be fine for our purposes:

Restart=always
RestartSec=500ms
StartLimitInterval=0

This tells systemd to always restart the service after a 500ms delay. You can configure it to give up eventually, but this should be fine for our purposes. Now reload systemd’s config and restart the service and try killing the process:

It works! systemd is now restarting our service when it goes down. It will also start it up automatically if the machine reboots (that’s what it means to enable a service). Go ahead and reboot to prove it.

We’ve now recreated one of our three PM2 features: process management. Let’s move on to the next one.

Logging

This is the easiest of our three target features. systemd has a very powerful logging tool called journalctl. It’s a sysadmin’s Swiss Army knife of logging, and it can do anything you’ll ever need from a logging tool. No Node.js userland tool comes close.

Multiple Instances

We’ve covered two of our three features now. The last one is port sharing, or clustering as it is often called in the Node.js world. But before we can address that, we need to be able to run multiple instances of our service.

You may have noticed that our unit file has an @ symbol in the filename, and that we’ve been referring to our service as demo-api-redis@1. The 1 after the @ symbol is the instance name (it doesn’t have to be a number). We could run two more instances of our service using something like systemctl start demo-api-redis@{2,3}, but first we need them to bind to different ports or they’ll clash.

Our sample app takes an environment variable to set the port, so we can use the instance name to give each service a unique port. Add the following additional Environment= line to the [Service] section of the unit file:

Environment=LISTEN_PORT=900%i

This will mean that demo-api-redis@1 will get port 9001, demo-api-redis@2 will get port 9002, and demo-api-redis@3 will get port 9003, leaving 9000 for our load balancer.

Once you’ve edited the unit file, you need to reload the config, check that it’s correct, start two new instances, and restart the existing one:

I’m assuming a 4-core machine, so I’m running three instances, leaving one core for Redis (which is probably not necessary). Adjust this accordingly for your environment and application.

Now, on to the final part: load balancing.

Load Balancing

One could use NGINX or HAProxy to balance the traffic across the instances of our service. However, since I’m claiming that it’s super simple to replace PM2 functionality, I wanted to go with something lighter.

Balance is a tiny (few-hundred lines of C) TCP load balancer that’s fast and simple to use. For example:

Conclusion

We’ve successfully recreated the three main features of PM2 using basic Linux tools, in fact, mostly just systemd. But this is only a very basic implementation. There are a number of details I’ve overlooked for the sake of simplicity:

SSL termination.

Ports 9001-9003 are currently bound to the public IP, not the private (this is just laziness in my Node.js sample app).

The balance unit file has hardcoded ports 9001-9003; it should be relatively easy to dynamically configure balance and send it a signal to reload config.

I’d normally use containers so that the dependencies (e.g., Node.js version) is bundled inside the container and doesn’t need to be installed on the host.

Linux init systems such as systemd are the ultimate process monitor, and systemd in particular is so much more than that. It can do all that PM2 and similar tools can do, and then some. The tooling is far superior, it’s more mature, and it has a much larger userbase of seasoned sysadmins.

Learning to use systemd for running your Node.js applications (or any other applications for that matter) is much easier than you might think. Once you’ve spent a little time learning these concepts, I think you’ll agree that Linux is the best tool for the job. After all, you’ll need to configure the Linux init systemd to start PM2 on boot and restart it if it crashes. If you need the Linux init system to start your process monitor, why not just use it to run all your services?

Subscribe via Email

Over 60,000 people from companies like Netflix, Apple, Spotify and O'Reilly are reading our articles. Subscribe to receive a weekly newsletter with articles around Continuous Integration, Docker, and software development best practices.

We promise that we won't spam you. You can unsubscribe any time.

Join the Discussion

Leave us some comments on what you think about this topic or if you like to add something.

Thanks a million for this! Converting simple ‘upstart’ conf was a real struggle until I found this. Could I trouble you for an expanded explanation of the ‘@’ at the end of the service name? I didn’t use it for my simple services, but I didn’t even think it was a legal character to use!

Arnau Oncins

I think the ‘@’ refers to user. It’s used to execute the service for an specified user. So @1 it’s root (uid=1). You can specify an integer (uid) or an username.

I would add a couple of small things – – You can set systemd to run your process as a specific user and not as root. This will help improve the overall security of your system since you can give your app access to only what it needs. This might only be a side note since readers will need to know a bit about the unix permission system and how to create a new user in their system. Might be a good topic for a follow up article :-)

– You can get the port forwarding capabilities using nodejs clusters, you can use clusters with almost no change to your current node code and then you get almost the full features of PM2 without any external software

Chris Popov

Thanks for the article! I have used systemd and it worked. I have tried pm2 but it was to flaky and unstable to use in production. I am currently using supervisord which is another option and it just works fine.

Frey István

Hi! thx a lot.

When starting the service, i get an error: “Failed at step GROUP spawning /usr/bin/node: No such process” but /usr/bin/node is available. Any idea how to improve?

Timo

Nice example. I used your example with some Java Spring Boot applications and was really easy to follow