File Synchronization with Unison

Keeping directories in sync on multiple machines can be difficult. Running Unison is one way to make the task easier.

Unison is a file-synchronization tool that runs on
Linux, UNIX and Microsoft Windows.
Those of you who've
used IBM Lotus Notes or Intellisync Mobile Suite probably
have an idea of what synchronization is good
for, as compared to one-way mirroring options such as rsync. You
might have mirrored a company document directory
to your laptop, for example, and then modified a
document or two. Other people might have modified
other documents in the same directory by the time you
get back. With rsync, you'd need to reconcile
the differences between the two directories manually or risk
overwriting someone's changes. Unison can sort out
what has changed where, propagate the changed files
and even merge different changes to the same file if
you tell it how.

Think of Unison as two-way rsync with a bit of revision control mixed
in. The most common use is keeping your local and remote home
directory, or some data directory you often use in different contexts, in
sync. It
uses the rsync algorithm to keep network traffic down and should be
tunneled through SSH over untrusted networks. No extra work is
needed—simply specify
ssh:// when adding a directory location. Quite a bit of
extra disk space often is needed for Unison, though, because the synchronizer needs to keep
track of what the files looked like on the last run.

Getting, Compiling and Installing Unison

Unison's home page is maintained at the University of Pennsylvania; the project
leader, Benjamin C. Pierce, is a professor in the Department of Computer
and Information Science. See the on-line Resources for the URL.

Unison isn't as widely deployed as rsync, so you might not be able
to find a precompiled package for your distribution. But the binaries
downloadable from the Unison home page should work for most people.

If you'd like to compile from source, you can. A few extra hoops must
be jumped through, however, because Unison is programmed in OCaml, not
the most common language. See Resources if there is no handy package
for your distribution.

Compiling and installing Unison is simple; type make
UISTYLE=xxx. The GTK user
interface needs additional OCaml bindings for GTK, so I use the text
interface in this article. Typing make UISTYLE=text or
make UISTYLE=gtk
should give you a Unison executable. Simply copy the executable to
somewhere in the path on both machines you want to synchronize.

In this article, I'm using the current stable version of Unison, 2.9.1, unless
otherwise noted. You need to use the latest betas if you're going
to synchronize files larger than 2GB.

The developer versions tend to work well. They are what the developers run
themselves on their own precious data. Sign up for the unison-hackers
mailing list if you feel a bit adventurous. Jerome Vouillon,
Benjamin C. Pierce and Trevor Jim tend to hang out there discussing
improvements. Commit logs also float by, so you can track what is
going on.

Configuring and Using Unison

Unison keeps its config and working files in a .unison directory in your
home directory or wherever you want to put it. Set the UNISON environment
variable to specify an alternate location.

The default configuration is stored in .unison/default.prf. Listing
1 shows a plain config file suitable for testing. Synchronizing two
directories is now as simple as:

Unison then asks the user about any differences between the
directories and offers reasonable defaults. It does take a bit of time to
get used to Unison's way of thinking, however. And, Unison is no substitute for
backups. Unison happily propagates back the deletion of all the files
in one replica, for example, which can be a rude awakening for programmers
used to CVS. For example:

rm dir1/* ; unison ssh://server/dir1 dir1

doesn't do what
you expect from a:

rm dir1/*; cvs update dir1

Deleting a file is
an action that is replicated on the other side upon synchronization. So,
this example command removes all files in dir1 on both sides.

Once you feel comfortable, consider adding auto =
true to the
Unison profile. This skips questions about any non-conflicting changes
but
gives you a chance to back out at the end.

The Unison manual is recommended reading. It is clear and well written and explains
what happens at most corner cases.

I think you all will like this product "Antida" that I just used. It is from Dimatas Technologies can can do data sync across win, Mac, Linux, Solaris. Also it does allow you to send and receive data from computer which does not have Antida installed. You can check the software at www.dimatas.com

Im running many ecomerce websites from the same source code, just few config files keep site variables and images ect , but im manually updating them all at the moment when i make a change, ive looked at subversion which seems to be over kill , and unison, but is unisonone to one relationship ?

what im looking for is a process whereby i alter one set files of codes and this updates many diffrent webservers, the code is currently ASP running on iis

There is a nicely convenient tool called usbsync which has specially been desgined for the purpose of keeping a usb flash drive in sync with multiple Unix/Linux hosts. At the time it's just beta but looks like it's going somewhere.

I found it quite useful as it allows me to perform my simple daily synchronization tasks without having to remember lengthy unison or rsync commands.

Unison has one serious limitation compared to rsync when you use it for backups (rather then
keeping 2 copies in sync): I would like to simply backup my files and not remove deleted files on the backup even if they had been deleted on the source. The reason is that occasionally I delete a file by mistake and if
I realize it after the unison backup happened then it will have been deleted on the backup.
This option is very easy to support and I am surprised that Unison does not offer it

Another missing feature (also wrt using it for backups) is that it does have the ability to use hard links on the
target; this feature would have allowed one to use it as a time machine.

Does anyone know of an open source tool to multi-directionally any number of servers in sync. Basically what I am looking for is something like unison but at the kernel level, so that as soon as a change happens, it propagates the change, and doesnt have to be manually run from the command line or in a crontab. Also something that propagates posix locks would be very helpful. I am looking to mirror samba shares in real time in 3 different cities. There are commercial products that do this (peerfs etc), but I am a big proponent of open source.

You could use incron and unison. incron watches for events, so when you e.g. save a file, or make a new one, you can run commands. So you could easily set that up and have it sync your files quite fast.

Unison is no longer under active development as a research project.(Our research efforts are now focused on a follow-on project called Harmony, described at http://www.cis.upenn.edu/~bcpierce/harmony.) At
this point, there is no one whose job it is to maintain Unison, fix
bugs, or answer questions.

Since so many have to live in mixed environments Unsion is capable of backing up that other OS's data to a linux box as easily as linux to linux. It's also a gui tool for the windows and non techy Linux desktop user, They can backup their personal data as often as they to the same repository you use for the main backup, and you don't have to listen to them complaining that the last backup was 10 hours before they downloaded that movie in the e-mail. (Hey you should have backed it up yourself!) I have found that no one uses the added feature until the first person has a meltdown. Suddenly it's a very popular feature since all they have to do is click.

There is now a visual sync tool for Linux that does something similar to ViceVersa except that it doesn't do automated syncs yet. Manual only. Have a look and see if may be useful. sourceforge.net/projects/synco/

It answers both comments, I think the biggest issue here is the Diff algorythm nd rsync is taking care ( if not the best out there anyway ) the rest can easily be done via script, both on windows and linus. And yes there is a not only a client but a GUI for windows, and on linux a one line cron script take care of it.

iFolder in its current open source state is unusable. When there is a server backend and you are using the full version it is excellent but there is a use for unison - its easy to setup and there are Windows clients for it also.

I've been using unison for about six months to sync my work dev server and work live server. I spent a fair amount of time looking for a solution to my problems, which were that I needed the sync the servers BOTH ways (the websites allows uploads, which I need synced to me and obviously I need to sync script updates from dev to live). When I finally found unison, I had it installed and setup within mintues. The fact that the text interface is idiot proof is great too, especially considering my dev servers HD crashed yesterday! Reinstalled OS, install unison, sync servers, job done :)

I've been using Unison for about 3 years now. It's saved my butt many times. I use it to synchronize about 7 different machines running various linux flavours, Windows 2000 and XP and even on a Linksys NSLU2.
Especially useful for synchronizing (over ssh) between home and work. or even anywhere and home.

I gave up on the graphical interface as I found it a bit heavy on resources. I thought I'd never use the text interface which I found horrible but one day I tried it again and now always use that. I have a perl script wrapped around the unison text client to simplify creating the profile files for different machines/OS on the fly using DISPLAY/SSH_CLIENT variables to figure out where the server is.

That's not to say I'm not on the lookout for a replacement.
Unison development is slow now (I'm sure it's not helped by the fact that it's written in CAML but that's just my opinion) probably as the team seem to have moved onto something bigger and better....

As a replacement I guess I want:
- multi-OS support as unison
- open source as unison
- a more active community around the tool,
more plugin tools
- better handling of file and directory moves/renames
it's a pain if you just rename a large file to have to manually rename it on the other machine OR put up with a long transfer.
Of course this may not be an easy problem to solve but having a
special move tool, e.g. unisonrename which logs the move
- ability to synchronize against different sources, e.g. imagine synchronizing against a tar archive or a CD/DVD-based archive (and maybe just logging or saving the differences) for example.

Interested to hear your views on what could be improved in unison-like tools, or what alternatives exist or are in development.

I have my documents on a usb hard drive which I use to sync between a linux (home) and a windows PC (work). To make the drive readable by both linux and windows, the drive is formatted in fat32.

When connecting the drive to the linux machine for synchronization, however, all the changed files will get the executable bit set under linux. To avoid this, I have to connect the usb drive to a windows pc at home and sync through the samba server to linux (I use vuBrief on Windows). This works, but it is one of the few reasons why I maintain a windows PC at home. One reason to much from my point of view.

A linux sync tool which could deal with the executable bit (set by the windows OS) would be very welcome!

It may be interesting to give Ext2 IFS a try ( http://www.fs-driver.org ) ! (see also FAQ entry about accessing Ext3 FS at http://www.fs-driver.org/faq.html#acc_ext3).
FYI, I'm using it for weeks now on a WinXP SP2 box with an external 160Go firewire HDD. This disk was an hawfull warehouse (145Go) of files of any types and any sizes up to 5Go, coming from many computers, from different times. I had to cleanup and organize it with no Linux box available... three full days of moving, deleting, and renaming later (with valuable help of dualXplorer BTW) I had my job done :)
The only trouble I encountered was some files or folders locking, all solved with a remount.

As Linux continues to play an ever increasing role in corporate data centers and institutions, ensuring the integrity and protection of these systems must be a priority. With 60% of the world's websites and an increasing share of organization's mission-critical workloads running on Linux, failing to stop malware and other advanced threats on Linux can increasingly impact an organization's reputation and bottom line.

Most companies incorporate backup procedures for critical data, which can be restored quickly if a loss occurs. However, fewer companies are prepared for catastrophic system failures, in which they lose all data, the entire operating system, applications, settings, patches and more, reducing their system(s) to “bare metal.” After all, before data can be restored to a system, there must be a system to restore it to.

In this one hour webinar, learn how to enhance your existing backup strategies for better disaster recovery preparedness using Storix System Backup Administrator (SBAdmin), a highly flexible bare-metal recovery solution for UNIX and Linux systems.