Disaster recovery with MondoRescue

This is an edited version of a post that originally appeared on a blog called The Michigan Telephone Blog, which was written by a friend before he decided to stop blogging. It is reposted with his permission. Comments dated before the year 2013 were originally posted to his blog.

Image by mattbraga via Flickr

Many of us face the problem of having a server that we know we should backup frequently, but we don’t do it because it’s either too difficult to figure out how, or the backup solutions offered don’t actually restore the entire system if it crashes, so we figure, “why bother?” If your system crashes, the thing you really need is a way to restore the entire system from some recent point in time.

Well, here’s one possible solution for you, assuming your server runs some form of Linux, and it’s from the fine folks at Sunshine Networks in Brisbane, Australia. I refer you to their article:

Now, don’t let the title throw you – there’s nothing Elastix-specific in this article. The instructions should work with just about anything running under the CentOS operating system, and with minor tweaks to the installation process, under other versions of Linux. What this software is supposed to do is give you an ISO file that can be burned to CD’s or DVD’s, or stored on a network share on another machine. If the worst happens, you fix the hardware problems and then reinstall from the ISO file, and the way it’s supposed to work is that you get back to exactly where you were at the time of the last backup. Now, I haven’t personally ever had to attempt a restore, but apparently others have and consider this a great piece of software. Obviously, I’m not making any guarantees, but it’s got to be better than no backup at all, right?

EDIT: Since I originally wrote this article, I’ve actually had the opportunity to use MondoRescue to restore a failed system (in this particular case, one that runs on a virtual machine). To say it worked great is an understatement. You just boot from the .iso file and it installs EVERYTHING back as it was. The only issue I had was that it couldn’t communicate with the network because the name of the network adapter was apparently different on the original and new systems — once I reconfigured the network settings to select a valid adapter (eth0, for example) it appeared to work just as it had on the day of the backup. And the restore process was surprisingly fast (much faster than the original installation, in fact)! Of course I cannot guarantee it will work that well for you, but I was blown away by the speed of the restoration, and I’m not that easily impressed!

I must also note that the article on the original Sunshine Networks site seems to have disappeared, so I changed the link to point to an archived copy on the Wayback Machine. However, in case that fails at some point, here is how I installed MondoRescue. Their instructions gave three different ways to do it, and I used this one, which (with perhaps a change in the file used) should work on any Red Hat or Centos based system (this was noted as “Tested on Elastix 2.0 32-bit” — if you are running something else, don’t just follow these instructions because you may need a different file):

after mondo installed correctly, you should disable the RPMForge repository, just to be on the safe side :
nano /etc/yum.repos.d/rpmforge.repo
change “enabled = 1” to “enabled = 0”

(They used vi to edit the repository; I changed it to nano. Use whichever text editor you like).

However, the file shown here is probably NOT the right one for your system. So, first go to http://packages.sw.be/rpmforge-release/ and read the descriptions for each file, and be careful to select the right one for your system, and substitute that filename in the two lines where it is used above.

After installation, you can start the program by running /usr/sbin/mondoarchive, which will bring up a GUI (of sorts). The original article notes that:

your full iso will ( under default settings ) be created in the following directory :
/var/cache/mondo/mondorescue-1.iso
there is a small recovery CD here :
/var/cache/mindi/mondorescue.iso

END OF EDIT.

The article has you use the mondoarchive GUI to make the backups (well, they actually say mondorescue, but when I downloaded the software the program was called mondoarchive), and that’s fine to start with. But eventually, you’re going to want to automate the process so you can use it in a cron job to do unattended scheduled backups on a regular basis. I have this running on one machine and send copies of the backups to another, like this (cut and paste from this article to get the full lines without wrapping) :
#!/bin/bash
mondoarchive -OVi -d "/var/cache/mondo" -E "/asterisk_backup" -N -9 -G -s 4G
ssh myaccount@server2.net rm /home/myaccount/server1backup/mondo/mondorescue-1-old.iso
ssh myaccount@server2.net mv /home/myaccount/server1backup/mondo/mondorescue-1.iso /home/myaccount/server1backup/mondo/mondorescue-1-old.iso
scp /var/cache/mondo/mondorescue-1.iso myaccount@server2.net:~/server1backup/mondo
ssh myaccount@server2.net rm /home/myaccount/server1backup/mindi/mondorescue-old.iso
ssh myaccount@server2.net mv /home/myaccount/server1backup/mindi/mondorescue.iso /home/myaccount/server1backup/mindi/mondorescue-old.iso
scp /var/cache/mindi/mondorescue.iso myaccount@server2.net:~/server1backup/mindi

The first line calls the mondoarchive program to create the backup – the -E argument excludes any directories you don’t wish to back up (I have a directory of backups made using another method that I didn’t want backed up) and you can read about the other arguments in the documentation (also see the full HOWTO). The remaining lines connect to the external server and delete the oldest backups, rename the previous backup, and then copy the new backups over. To do it the way I’ve done it here, you must have ssh access to the other server and you must be able to connect without using a password, using public/private key authentication. You may also have to log into the remote server and create the directories (/home/myaccount/server1backup/mindi/ and /home/myaccount/server1backup/mindi/ in this example – obviously you can call the directories whatever you wish, it’s entirely up to you).

There is, of course, more than one way to remove the pelt from a deceased feline, and you’ll probably have your own method for moving the files to another server. In some situations it appears that MondoRescue could do it for you (look at the n option), but it doesn’t include a provision to remove the oldest file and rename the previous one (not that I could see, anyway), so that’s why I did it in a shell script.

The folks at Sunshine Networks have several other great how-tos – you might want to give them a look! And for more useful information on MondoRescue, particularly how to perform a restore, see Configure IT Quick: Use Mondo Rescue to back up Linux servers (but please realize that article was written in 2003, and the install has apparently been made less complicated since then, so don’t use their installation instructions).