Backup OpenStack object store or S3 with rclone

Table of Contents

Introduction

This is a guide that sho ws you how to make backups of an object storage service like OpenStack swift or S3. Most object store services save data on multiple servers, but deleting a file also deletes it from all servers. Tools like rsync or scp are not compatible most of the time with these services, unless there is a proxy that translates the object store protocol to something like SFTP. rclone is an rsync-like, command line tool that syncs files and directories from cloud storage services like OpenStack swift, Amazon S3, Google cloud/drive, dropbox and more. By having a local backup of the contents of your cloud object store you can restore from accidental deletion or easily migrate between cloud providers. Syncing between cloud providers is also possible. It can also help to lower the RTO (recovery time objective) and backups are just always a good thing to have and test.

Installation

rclone is written in the Go programming language, so installation is quite easy, it's a single binary. They only provide Snap packages, no regular .deb or .rpm packages. I personally rather have just a repository or have it packages upstream, but snap works as well.

This guide uses an Ubuntu 16.04 server. By default snapd (the snap package manager) should be installed, but if that's not the case, install it:

apt-get install snapd

Use snap to install rclone:

snap install --classic rclone

The --classic argument is required because it disables the security confinement otherwise it won't be able to access some user files.

On the test machine I used the snap binary was not in the $PATH, I had to logout and log back in. rclone's binary is in /snap/bin/rclone.

If you are on another distro or want to do manual installation, you can do so:

Depending on the data you received from your cloud provider, some of the following options are required. The tenant name in my case is and I use the tenant_id in this field. When ever someone renames the tenant the config won't break:

If you have problems with a Swift backend, please see the last part of this guide. Most likely your credentials or other data like project ID or region will be wrong.

For this example I have also setup another swift backend at a different OpenStack provider (fuga by CYSO). You can setup any cloud provider you like, or just use SFTP (via SSH) to some location remote. rclone abstracts this away for you.

One important point with rclone is that by default it does not follow symlinks. This is because the software works on Windows as well and there is no support for symlinks there. If you do have symlinks then you must give the -L / --copy-links command line option.

Local backup of an Object Store

After you've set up the rclone remotes we can configure a backup to the local machine. This can be a server somewhere or you workstation. To keep the example simple, there is no automated cleanup in this guide, but you can easily set this up. The command syncs the backend to the local filesystem, based on the day, so if you schedule this cron once a day you have a full backup every day.

rclone sync swift1:loadtest /root/backup/$(date +%Y%m%d)/

There is no output. Listing the directory does show a full backup locally:

Running this as a cron script every day allows you to have a backup of the object store at a different location, plus versioned. rclone does not support incremental or differential backups, (see documentation),

Sync two object stores

Syncing two object stores with rclone is usefull when you need the contents to always be online, even if one service provider has a large outage. If your application supports it, the best thing is to let the application do dual uploads to multiple object stores. It could then also load from different object stores if one is down.

If dual upload is not available, you can use rclone to do a sync between object stores. rclone does have to download every file locally before uploading it to the other side, so the machine you use to sync object stores must have enough free disk and lots of bandwidth.

Using the above commands, you could also implement a backup of one object store to another. This example just syncs the stores, so that in case of a disruption you can change the configuration in your application and not have downtime or loss of data for a long period.

This example uses two swift object stores, since just changing configuration for swift is applicable in more cases. If you sync Amazon to swift you need to have swift and s3 compatibility in your software. (or any other two different protocols). Most swift object stores do offer S3 emulation, but compatibility differs between software versions so test that beforehand.

In this example I have setup another object store with Cyso (fuga.io) to do the syncing. CloudVPS object store is named swift1 and fuga is named swift2 in the rclone config. The data in container loadtest goes from CloudVPS to fuga. Files added, changed or removed at CloudVPS are added, changed and removed over at fuga as well, there is no versioning.

By having a second live version of your data, you are able to meet a lower RTO (recovery time objective). If one service provider has a major outage, you don't have to wait hours, or even days until it is fixed. You just restore the backup or change your configuration and are up and running again.

Do note that as with every backup, it's important to test this regularly. Do a failover once in a while or try to do a restore and see what works and what not. Document it so that your team can do it as well, saves you another call in the middle of the night.

Errors with Swift

I tried to setup a backend at another cloud provider (OpenStack over at CYSO, fuga.io). After setting up the configuration with the correct username, password, auth_url and such (since the nova and swift CLI worked), rclone kept giving a non-descriptive error:

Setting the loglevel to DEBUG or specifying verbose mode did not help. The documentation states the following:

Due to an oddity of the underlying swift library, it gives a "`" error rather than a more sensible error when the authentication fails for Swift.
So this most likely means your username / password is wrong. You can investigate further with the `--dump-bodies` flag.