I often say there are only two types of data: data that is backed up, and data that is waiting to be lost.

Even if you’re the most contentious hax0r who always backs up your data, if your backups are stored in the same physical location as the source data, then your data is still “waiting to be lost” in the event of a fire, flood, theft, or other disaster. By combining Amazon’s low-priced S3 (Simple Storage Solution) Cloud-based storage solution with some excellent some open source backup tools, you can now be more prepared than ever without spending a fortune.

This how-to demonstrates how I combined the following tools to automate my off-site backups:

Step 1: Set up your Amazon S3 Storage Bucket

I won’t walk through all the steps to do this, as Amazon makes it easy. Just sign up for their S3 service (you only pay for what you use), sign in, find the Security Credentials page and take note of your Access Key ID and your Secret Access Key. You’ll need them later. You should also set up an S3 Bucket to store your backups. Write down the name of your bucket for use in a later step.

Step 2: Download AutoMySQLBackup (optional)

If you don’t have any MySQL databases to back up, or you have your own preferred method of backup in your databases, you can skip this step. AutoMySQLBackup is a free utility that quickly and easily create dumps of your MySQL data – which we’ll back up to Amazon S3 in a later step.

Download AutoMySQLBackup from SourceForge and run the simple install.sh script to set it up. I followed this excellent blog post to help me get AutoMySQLBackup configured and working. I had to make a few minor changes because I’m using a more current version of AutoMySQLBackup and some of the variable names in the config file were different, but it’s pretty straightforward. Once you’ve got it backing up your databases, you’re ready to move on.

Step 3: Download and Install Duplicity

Duplicity is the program that does most of the heavy lifting in this situation. It manages the actual file backup (full or incremental), compression, encryption, and the file transfer to any number of off-site storage locations. Lots of documentation is available online, in case your needs differ from the ones explained here. As always, Google is your friend.

To install Duplicity if you’re running Fedora, RHEL, or CentOS, it’s as simple as doing:

yum install duplicity

For Ubuntu or Debian users, do:

apt-get duplicity

If you’re running some other flavor of Linux, refer to the Duplicity website for help installing.

Step 4: Create a GPG Key for Backups

Because you’re going to be transferring your precious data over the Internet, and storing it in an off-site location that shouldn’t be, but still technically could be, accessed by snooping Amazon employees or hackers, it’s best to encrypt your data before sending it… “to the CLOUD!” Seriously, those commercials are so annoying.

Even if you already have a GPG key, I recommend creating a separate one just for backups (which we’ll also store in a secure location later so you’re never stuck without the ability to decrypt your data later). Do:

gpg --gen-key

You can accept all the defaults, but make sure you use a passphrase when creating this key, since Duplicity will require it. After you’ve answered all the questions, the output should look something like this:

We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
.++++++++++++++++++++..+++++++++++++++++++++++++++++++++++++++++++++++++++++++.+++++
.+++++..+++++.+++++++++++++++++++++++++++++++++++++++++++++.....>.++++++++++................................+++++
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
+++++.++++++++++..++++++++++...++++++++++...+++++.+++++..+++++.+++++..+++++++++++++++.+++
++++++++++++..+++++++++++++++..++++++++++..+++++++++++++++++++++++++...+++++..+++++>+++
+++++++>.+++++>+++++......................+++++^^^
gpg: key 1F6C9247 marked as ultimately trusted
public and secret key created and signed.
gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0 valid: 2 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 2u
pub 2048R/1F6C9247 2011-11-05
Key fingerprint = FC81 D8E3 8090 EEE3 1D98 E000 045C D80E 1F6C 9247
uid Backup Key <[email protected]>
sub 2048R/12D6A5B0 2011-11-05

Take note of your key’s public GPG Key ID, which is listed on the line where it says “key xxxxxxxx marked as ultimately trusted” (in this example, it’s 1F6C9247). You can also find your GPG key’s public ID with:

You’ll see your key’s ID on the top row after the slash. Write it down (don’t worry, it’s not a security risk like a password) to refer to in the next step.

Step 5: Download and Configure dt-s3-backup Script

While trying to make all the aforementioned tools work together, a stumbled across a very cool script that already did it for me. This blog post explains the script, and the script itself is hosted on GitHub.

Download the script to your server (I put mine in /usr/local/bin) and then open it up in an editor. You’ll need to put the following in the appropriate locations inside the script:

AWS_ACCESS_KEY_ID: Your Amazon Access Key (duh!)

AWS_SECRET_ACCESS_KEY: Your Amazon Secret Access Key (double duh!)

GPG_KEY: Your GPG Key ID of the key you created in the previous step

ROOT: I changed this to just “/” so that I could back up anything on the system. You’ll pick the exact directories you want in a bit.

DEST: Since we’re backing up to Amazon S3, comment out the “file:” line, uncomment the “s3+http:” line, and put the name of the Amazon S3 bucket you created for backups in the first step. If your bucket name were “my.awesome.backups” then this line would be DEST=”s3+http://my.awesome.backups/”

Skip the INCLIST and EXCLIST options for now, and tinker with the STATIC_OPTIONS to your liking. These will simply be passed to Duplicity, so you can check the Duplicity docs for all the possilibities. I have mine set to STATIC_OPTIONS=”–full-if-older-than 4W” which means my backup (which I run daily) will do incremental backups unless it’s been 4 weeks, in which case it will do a full backup. I also kept the default CLEAN_UP_TYPE and CLEAN_UP_VARIABLE settings. Again, refer to the Duplicity docs for other options.

Finally, I also tinkered with the Logfile settings and Email Alert settings.

Step 6: Choose which directories to include and exclude

Use the INCLIST and EXCLIST sections of the dt-s3-backup.sh script to list which directories you want to include and exclude while doing your backups. Examples are shown in the script. Make sure that whatever directory you used to store your database backups with AutoMySQLBackup is included. If you want hidden directories excluded, be sure to include them. The following are my lists:

These settings work for me, but there’s no guarantee they will work for you. It’s your data, so you should completely understand what is and isn’t going to be backed up.

Step 7: Do a Test Run

To test things out, find the following line in the dt-s3-backup.sh script and uncomment it (remove the #):

#ECHO=$(which echo)

As explained in the comments, this will run the script in test mode, which will spits out the full Duplicity command and send it to the email address you set up in the Email Alert settings.

Save your edited version of the script and run it with:

dt-s3-backup.sh --backup

Because it’s in test mode, it should think for a bit and then email you some output, which includes the full command that will be passed to Duplicity. If everything looks good, comment the #ECHO line out again, and go for it:

dt-s3-backup.sh --backup

Depending on many factors (the amount of data you’re backing up, the speed of your system, the speed of your connection to Amazon S3, the phase of the moon), you’ll have to wait for a bit. My system takes about 5 minutes to run a full backup.

If something goes wrong, check all your edits, and check the links to the other blog posts I’ve included. I won’t be any help answering support questions in this thread, because I’m not the author of any of these applications.

Step 8: Check Your Files

Assuming your backup worked, you can ask Duplicity to list all the files in your backup with:

dt-s3-backup.sh --list-current-files | more

Keep in mind that these will count as a requests against your Amazon S3 allowance. You get a bunch of free ones, but managing your Amazon bill is completely your responsibility.

Other options for dt-s3-backup.sh are available in its README file. I recommend experimenting with them until you’re familiar with the ones you’ll need.

Step 9: Automate

Once everything is working as you want it, don’t forget to create cron jobs for AutoMySQLBackup and dt-s3-backup.sh, I dump my databases nightly, and I do an incremental backup with dt-s3-backup weekly. Use whatever settings work best for you.

Step 10: Provide Feedback

I always welcome your feedback, especially if you have suggestions for making the process in this article easier to do or understand. If you have a different backup method that works for you, please feel free to share it. Because I’m not the author of any of these utilities, however, I can’t provide support in using them. Check the links I’ve provided for support, or contact the application authors directly if you’re having trouble.

Good luck moving your data from “waiting to be lost” to “backed up.” I know I sleep better knowing I’m better prepared to deal with disaster!

Awesome setup for offsite backups. I have several MySQL DB’s that need to be backed up and a lot of the tools you mention here rock. I will be checking them out.

http://trucsdedev.com Thomas

What happen if you loose the gpg key ?

http://www.stevejenkins.com/ Steve Jenkins

Same thing that happens if you lose your GPG key in any other situation: yer screwed. That’s why it’s vital to keep an exported version of your key(s) in a secure location. I keep copies of mine on two separate USB keys in two separate safes. Bottom line: don’t lose your GPG key – you’ll lose the ability to decrypt anything you’ve encrypted with it.

John M

Just curious why you chose to use Duplicity in favor of something like s3cmd in this case. Also, newer versions of the AutoMySQLBackup have built in AES-256 openssl encryption available.

http://www.stevejenkins.com/ Steve Jenkins

Hey, John. Fair question. I use s3cmd on some other projects, and love it. However, I prefer Duplicity because it can do the full and/or incremental backups, and I find that this approach makes the include and exclude arguments easier for me to manage.

I’ll check into the newer versions of AutoMySQLBackup. Thanks for the heads up!