∴ikura

Hot-code burn marks

Updating an OTP system with new code without taking it out service is tough.
How tough? Well, Erlang tough. Meaning, there is a learning curve, but after
the hazing, man, does it ever feel good.

In the brochure, Erlang/OTP touts hot-code loading usually towards the end
of its ‘killer-features’ list. Imagine, it goes, running your servers, then
introducing a new feature, pushing a button, and bango, your feature is live!

Like all magic, there is rigor involved to pulling it off.
It can be a bit daunting to perform hot-code loading
(sometimes called hot-code swapping) for an entire OTP system.
Largely, it requires a steady hand & clear head to do it correctly.

Though there are plenty of resources on the web for performing this
‘as-seen-on-TV’ feat, I thought it would be helpful to
write a toy system that showcases the steps needed to make it all work,
while stressing the parts one should be wary of.

We will be using the slightly now hum-drum tool
rebar (not to be confused with rebar3),
relx, and a few
third-party Erlang libraries. So be sure to have
the above installed, along with a recent version
of OTP (I am using 17.5).

Fire is looming

We are going to put together an Erlang web-server that serves a single page. Every three seconds, some javascript
will call an API endpoint (‘/which-color.json’) that will toggle
the background color of our web-page.
When we install our updated system (i.e. with hot-code loading),
we will see a sudden change in the colors being cycled in our browser.

Some Erlang libraries will aid us along.
In addition to Cowboy, we will use the Erlang
port of the
Django templating language.
So, using rebar let’s go create our ‘fire is looming’ app with these
dependencies & get started.

mkdir fil
cd !$
rebar create-app appid=fil

This will yield the building-blocks of our OTP application;
three files should be created in a new ‘src’ directory.
We can go ahead and create a ‘rebar.config’
file in the project root directory with the following code:

Color state-machine

Let’s switch gears and create a color state-machine server.
This server will change from color to color at an interval of
our choosing. And, like most servers, it will respond to requests &
offer up what the current color is.

In the initial version of our system, the colors will cycle
through three greens: starting dark and getting slightly
lighter; then reset back.
Here is the code for our new ‘src/fil_color_fsm.erl’ file:

Our back-end is now in place, and we just need to do a few things
to present our one-page HTML file.

Django templating

The Django templating project for Erlang (erlydtl) is excellent & I use it often.
Although we could create a simple HTML file for this toy app,
I thought it beneficial to expose how these templates interplay
with hot-code loading. I have already
slipped in all the necessary erlydtl configurations above, so all we
need to do it create the template. Go ahead and make the directories
‘priv/dtl’ in the project root, then add ‘root.dtl’ in there
with the following:

For the back-end/system hackers out there, the JS above performs
a couple of things:

in a loop, call our API end-point to retrieve a color;

set the top/header element’s background to that color.

For the sake of this tutorial, this strategy will suffice. A
more elegant approach would to be use websockets.

Initial release

We now have most of what we need for our OTP release to be built &
served. We do need to configure our release with a couple of
things, so let’s bang that out right now. We want to create a
‘relx.config’ in our project root with the following:

And you will notice that we are calling a couple files in there
that don’t exist yet. We will need to create those in a new directory
called ‘private’ which sits in the project root. Our first file,
‘private/vm.args’ has the following:

-name fil@127.0.0.1
-setcookie fil123

And our second private file, ‘private/sys.config,’ has the following Erlang term:

[
{fil, [
{http_port, 8004}
]}
].

Using our installed build tools, we are ready to create our
release. We do that the following way:

rebar compile
relx tar

You will notice that a ‘_rel’ directory was created; it contains
a ‘tar’ of our OTP system we can now install. Let’s work out of a
‘/tmp/fil’ directory and copy ‘_rel/fil/fil-1986.a.tar.gz’ over
there now. We install our system as follows:

cd /tmp/fil
tar xvf fil-1986.a.tar.gz
./bin/fil start

If you open a web-browser to ‘127.0.0.1:8004’ — assuming
all went well — then you should see a wonky web-page
with a toggling green header.

Preparing version 1986.b

Our goal is to have our system update with zero down-time. In most
cases, simply stopping the running release and starting the newer
one would suffice, but some systems have tighter demands. So, let’s
make the changes needed to have this all work.

We can start with bumping the version in ‘src/fil.app.src’ which
now has the following:

Also, to showcase how erlydtl behaves with hot-code swapping,
we change a one-liner in ‘priv/dtl/root.dtl’ as well:

...
<section>
<h3>Hello #2</h3>
</section>
...

So, these are our code-changes, and now we need to switch gears &
prepare OTP for the delta(s).

First, familiarize yourself with
relups.
You will see that one has to explicitly tell the release handler
what’s changed. It’s a fairly deep subject, but for our toy app,
all we need is a few lines of Erlang terms placed into a config file.
Since erlydtl generates ‘.beam’ files from the ‘.dtl’
source, we need to tell the release handler that we have two changes
to account for — we put this all in a new file called
‘src/fil.appup.src’ which looks as follows:

Ideally, you should peruse the
relup/appup guides to grasp
what this all means, but for the most part it tells the OTP
release handler to prepare for the apt modifications, load, then
swap the new changes in.

We want to keep this new ‘appup’ file around in source as record,
but if we don’t copy it over to ‘ebin/fil.appup’ then nothing will
happen when we try to relup our new system.
In the project root, do this once:

cat src/fil.appup.src > ebin/fil.appup

N.B. The difference in file extension.

Finally, our ‘relx.config’ needs some tweaks, too. This version
bump leaves that file with the following:

Ready for hotness

We are done with our preparation & can now update our system. Working
from the project source directory, we need to compile the new code,
then generate a release with a slightly different twist:

rebar compile
relx relup tar

This, again, will give us our ‘tar’ — this time version ‘1986.b’ —
placed in ‘_rel/fil.’ An aside, if you blow away your ‘_rel’
directory, doing ‘relups’ from a fresh state will give you issues.
It’s best to keep a working directory of all your releases.

Everything is in place for the new version. If you have closed your
web-browser, be sure to take another gander at the old version now
by going to the ‘127.0.0.1:8004’ URL. Keeping that in your
peripheral, perform the following from the ‘/tmp/fil’ directory:

./bin/fil upgrade "1986.b/fil"

If all is well, your browser should be toggling the new red color
into its header. This indicates that our back-end has indeed
been swapped. A quick refresh of your browser should indicate our
template change, too: ‘Hello #1’ should now read ‘Hello #2.’ This
indicates that our ‘front-end’ was swapped successfully.

In conclusion

One will notice that it’s quite the dance in getting
hot-code loading right. It certainly comes at a cost, but thanks
to ‘relx,’ a good deal of the pain is hidden (believe it or not).
The rebar3 project is one to watch, too.
You can expect further automation of some of the steps above as it
matures.

All in all, without hot-code loading, ikura wouldn’t
have attempted to specialize in what it does. It would just be
too tough.
We need to update our code-base often.
Code updates while preserving our customer’s background jobs would
be excruciating without Erlang’s hot-code loading.