Kevin van Zonneveld

Synchronize Files With rsync

3 minute read

Synchronizing files from one server to another is quite awesome. You can
use it for backups, for keeping web servers in sync, and much more. It's fast and it doesn't take up as much bandwidth as normal copying would. And the
best thing is, it can be done with only 1 command. Welcome to the wonderful
world of rsync.

Installing rsync

On most modern Linux distributions you will find rsync comes
preinstalled. If that's not the case, just install it with your package
manager. On Ubuntu this would look like:

$ aptitude -y install rsync

done!

Simple - One Command

Let's copy our local /home/kevin/source to /home/kevin/destination which
resides on the server: server.example.com:

-z compress, saves bandwidth but is harder on your CPU so use it for slow/expensive connections only

--progress shows you the progress of all the files that are being synced

--size-only compare files based on their size instead of hashes (less CPU, so faster)

Note that this sync excludes hidden files since it uses the bash *. If you want to include
hidden files, write the source like this: /home/kevin/source/ and remove the trailing slash
from the destination like so: /home/kevin/destination.

Well, that's it! But read on if you want to learn how to automate this.

Advanced - Automatic Syncing With SSH Keys

Alright so syncing files on Linux is pretty easy. But what if we want to
automate this? How can we avoid that rsync asks for a password every time?

There are different ways to go about this, but the one I mostly use is
installing SSH keys. By installing your SSH key on the destination
server, it will recognize you in the future and permit instant access. So this
way we can automate the synchronization with rsync.

Easy Script

I've written another article explaining on setting up SSH keys. It also
includes a script that can do all the work for you.

Did It Work?

Open a terminal and type:

$ ssh server.example.com

It should not ask you for any password. Great! this means we can also run
rsync directly without logging in! If you need more in depth information on
this, I wrote an article on logging in automatically with SSH keys.

That's it! New files are automatically updated @
server.example.com:/home/kevin/destination/ every hour. Files that are
deleted from /home/kevin/source/* are also deleted at the destination, thanks
to the --delete parameter.

--exclude-from=/root/sync_exclude specifies a /root/sync_exclude that contains exclude patterns (one per line). filenames matching these patterns will not be tranfered

--bwlimit=1024 This option specifies a maximum transfer rate of 1024 kilobytes per second.

Pitfalls

Of course you should really be carefull where and when to install SSH
keys, because if one machine is comprimised, it's very easy for a cracker to
hop to the next system without logging in. So choose wisely when to use this
technology. You might consider 'pulling' the files in from the backup machine if that one is less exposed. This way if your main machine gets hacked, they can't hop to your backup machine.

Keys are user user specific. So if you're going to run programs as root
that need to automatically login to systems, you must also install the key as
root.