∴ikura

Prod-to-dev with Mnesia

Back in my dark days of Rails development,
it was essential to have a way to populate one’s local database,
say SQLite, with production data. And for
some reason, half a decade has gone by as an OTP engineer
before I needed to
perform this task in Erlang-land.

After looking into the matter, it was easy enough, but
the documentation was scattered. So, I thought I would
share my process for getting a production node’s db-data
onto my local development machine.

Ramping up

First off, I want to define a few things for the sake
of this guide. I will refer to our single production box
via the name ‘jimbo@prod’ while our development name will
be ‘jimbo@dev’ — these, of course, are the VM’s sname
values, or short names, and in your case, you’ll probably be
using fully qualified long names.

Also, we will assume our Unix user-name on production and
development is ‘jimbo’ for both.

Lastly, we will assume you have a preferred strategy for
attaching to production nodes — either via Erlang’s remote shell
strategy, or via erl_pipes.

Dumping production

The first order of business, is to make a copy of the
production database. We will want to output it somewhere,
so follow along and we will take care of that. On our
production box, let’s create some directories to aid
us along:

cd ~
mkdir tmp

Next, attach to your production system Erlang shell and
perform the following:

1> mnesia:backup("/home/jimbo/tmp/mnesia.prod.backup").

Done. You can now detach from the production Erlang shell.

Now, change into that dump directory and tar up the database
contents for quick transfer to your development machine. Perform
the following to do just that:

cd ~/tmp
tar cvfJ mnesia.prod.txz mnesia.prod.backup

And we can close our Unix session on production. We are
all finished with it.

Dev prep

Our OTP release could be anything at all. From a simple
server, to something quite complicated. But for the sake of
this guide, we need to make an assumption that our system
has an API resource whereby we can prime our Mnesia
database for fresh installs of a target system.

There is no standard way of doing this, but to create the
requisite database schema, I usually have some code that does
this all in a deterministic way, such as foo_lib:init(db).
which can be called once, manually.

Here is some example code on what this could look like. Here is
a snippet from a fictional module called ‘foo_lib.erl’ :

This code is just to give you a clue about how to set
up your database schema. Be opinionated about how you
like to do it, for sure. We will assume for the rest of
this guide, a fresh Mnesia database can be primed after
we start the OTP system, then call foo_lib:init(db).

With that lengthy aside out of the way, let’s get ready
to transfer our production data to our local box. First,
let’s create the familiar working space with mkdir ~/tmp
and copy over the compressed database to there now:

scp jimbo@prod-box.net:~/tmp/mnesia.prod.txz /home/jimbo/tmp/

Let’s assume all went well with scp, ssh access and such, and
we will proceed with un-tarring the dumped database:

cd ~/tmp
tar xvf mnesia.prod.txz

There. A good deal of the nitty-gritty is out of the way.

Introducing ‘convert.erl’

Now’s the time to write some Erlang. We are going to create a
new module that reads the production Mnesia database and converts
the naming of the production node, to that of our development one,
wherever it occurs.

Recall, that our production sname is ‘jimbo@prod,’ but locally,
it’s ‘jimbo@dev.’ That’s why we need to do this.

Here is the contents of the new ‘convert.erl’ source-file; this is
on our development box & placed in ‘/home/jimbo/tmp’ :

Now, all the while working in ‘/home/jimbo/tmp’ we compile the new
module using the humble erlc command. We do that as follows:

erlc convert.erl

Next, let’s put convert to work and get our database dump in a state
where our local machine can work with it. In that same directory, do as
follows:

erl
1> l(convert).
2> convert:new().
3> init:stop().

Done. We now have a copy of our database ready to be loaded into the
development version of our OTP project. If you ls -lhart the
directory, you can see the new file convert created.

Pulling it in

Go ahead and change directories to where your OTP project
lives.

It’s time to fire up our project locally, but first we make sure
any old Mnesia development data is blown away so not to interfere.
Assuming that the location of these old Mnesia database files are in
the project’s root directory, you can tar those up and place them
aside, or simply rm -fr ./Mnesia.* if you won’t be
missing any development data.

Now go ahead and start up your OTP release locally, and attach
to it in your favorite manner.

Recall that we now have a missing database schema, and need to
create one afresh. We will want to do that right away, followed
by a Mnesia routine that populates our local database with our
production data. We do both as follows:

And there you have it: your local Mnesia database now holds all
the goodies from production. If this is something you need to
do often, then the lion’s share of the above ought to be automated.
But there’s no shame in bookmarking this post & coming back here
when you need to do it all again.