LVM and Removable IDE Drives Backup System

Backing up data is a problem we all face, and as the size and complexity of the systems for which we are responsible grow, so do our backup problems.

When the company I work for, a civil engineering and surveying firm,
decided to move all its AutoCad drawings onto a central fileserver,
we were presented with a backup situation orders of magnitude
larger than anything we had confronted before. We had at that time (now
considerably larger) about 120,000 files, totaling 200GB, that were in
active change and needed to be backed up at least daily.

My first thoughts were of some sort of tape backup system, but as I
began to research them, I was shocked at the prices I encountered. A tape
autoloader large enough to contain our filesystem ran about $12,000
and a 40Gig tape was $89. When I first convinced my boss to let me run
Linux on our servers, cheap was a big selling point. So, what are the
alternatives?

I had been using a removable 120GB IDE drive to transfer data back
and forth from home; it had cost me $101 for the drive and $17 for the
removable bay. I also had a retired 1GHz P4 box that had cost about $800--the
math was starting to look interesting. If I could find a way to
stick 120GB drives together, I would be home free. That's
when I discovered Linux LVM (Logical Volume Manager).

With LVM you can, if you like, treat the removable drives as large, cheap
tapes and use any of the traditional rotation schemes of incremental
and full backups, such as the Tower of Hanoi. We took a somewhat different course for our needs.
It has always seemed awkward to me that in order to
recover a file from tape you had to search for it sequentially, restore
the full backup, apply the incremental backups in order and, finally,
arrive at the version you wanted. I wanted a backup system that would let
me open and examine drawings to determine if they were the appropriate
version before restoration, and with LVM that's what I got. It takes a
few more drives to do it this way, but the 12,000 bucks I saved on
the autoloader buys a whole lot of drives.

I probably should take this opportunity to warn everybody that I'm still new enough to Linux
that I do things that doubtless make more experienced people shudder in
horror. All I can say in my own defense is I'm still learning, and many
of these things have worked so well in the real world I have found no
need to change them.

We started by setting up the P4 box with one IDE drive for the operating system,
a CD-ROM drive (to load the operating system) and two removable drive bays,
each containing a 120GB IDE drive. It has
since grown to four drives on a RAID controller as JBOD, but let's keep this
example simple. We loaded Red Hat 8.0 on the backup server--the
first Red Hat version to include LVM--and gave it an IP address on the network.
Configure your box in a way appropriate to your own network.

Because we were going to be moving these drives in and out of the box and off site,
we needed a way to keep track of which drive was which. I used stick-on
letters and labelled the part of the first drive bay that remained in the case
A and the part that came out A1. The part of the second drive
bay that remained in the case was B and the part that came out was B1. These
were our backups for one day. Subsequent days were called A2B2, A3B3 and so on.

Having done this, we were ready to configure our drives, in this case /dev/hdb and /dev/hdd,
for LVM. Start by logging in as root and type fdisk /dev/hdb.
fdisk returns a helpful warning that because we are using 120GB drives,
the number of cylinders is set higher than a certain number, but we can probably
safely ignore this. Trust me, we can.

Assuming we are starting with a new unformatted disk, we wish to create
a new partition. Type n and then press Enter. fdisk
now asks if we wish to create an extended or primary partition. We
want a primary, so type p. fdisk now wants to know the number
of the partition (1-4), so type 1.

Next, fdisk wants to know the beginning cylinder of our partition. The
default is 1; accept it by pressing Enter. fdisk now wants to know the
ending cylinder of our partition. The default is the last available
cylinder on the drive; again, accept this by pressing Enter.

Let's take a look at what we've created so far. To see the partition
table, type p and press Enter. We should see something like the following:

Up to this point we haven't actually modified the
partition table. These changes are being held in memory by fdisk until
we tell it to write them. At any time, we can exit fdisk without making
permanent changes by typing q.

So far everything looks good except for the partition type, which seems
to be 83 Linux. We need to change this to Linux LVM, which we do by
typing t and then 8e.
Now, when we read the partition table by typing p,
we should see:

This is what we want, so we now officially change the partition table by
typing w. At this point, fdisk becomes quite excited
and tells us The partition table has been changed and returns
some ongoing information about what it's doing before unceremoniously dumping us back
at the command prompt.

We're done with the hairy part, except that
we need to repeat the process for any other drives we are going to
include in our logical volume set.

At this point we are ready to begin creating the logical volume. Much
useful information can be found in the
lvm man pages. Typing man lvm puts you at the parent page, which provides
directions to the subheading man pages. Now that our disks are formatted as Linux LVM, we
need to create them as physical volumes for inclusion in a volume group.
We do this by typing pvcreate /dev/hdb1 /dev/hdd1.

We now have to join the physical volumes we created into a volume
group from which one or more logical volumes can be created. A
peculiarity of LVM is we must at this time set a physical extents
size for our volume group. That is, we have to define a
certain sized unit in terms of megabytes, called a physical extent.
Any logical volume we create from this volume
group can contain no more than 64k of these physical extents. If we
suspect our logical volume eventually may contain a terabyte of data, we
need to set our physical extents to 16MB. We do this while
creating our volume group by typing:

vgcreate -s 16M a1b1 /dev/hdb1 /dev/hdd1

We now have a volume group called a1b1 composed of the two physical
volumes, /dev/hdb1 and /dev/hdd1.
Let's create a logical volume that uses all the available room on this
volume group by typing lvcreate -L 226g a1b1.

The -L option specifies the size (226GB). For some reason, I've never
been able to get a drive advertised as 120GB to yield more than 113GB
at this point. In the lvcreate line above we actually accepted the
default logical volume name (lvol1), so our logical volume is now
/dev/a1b1/lvol1. Notice that the last character is the numeral one and
not a lower case l. Now we need to create a filesystem on our logical
volume. I prefer to use the ext3 filesystem for our drawing files because of their
large size. We can create this by typing mke2fs -j /dev/a1b1/lvol1.

It takes a little while for the filesystem to be created, but when
it's done the only thing left to do is to create a mountpoint. We do this
by typing mkdir /mnt/back. We mount our logical volume by typing:

mount /dev/a1b1/lvol1 /mnt/back

We can test it by typing df -h /mnt/back; the operating system should report that we have about 226GB
available on /dev/a1b1/lvol1.

If we need to add drives to this volume
group in the future, we can do so on the fly with vgextend and e2fsadm.
We now need to repeat the process for each volume group that we wish to
create. I use one for each
day of the week, and once a month I pull one out of rotation and save it
for a year. This doesn't have to be done all at once; I did one a day
until all were complete.

Our network is composed of mostly Windows workstations connecting to
Linux servers through Samba. So, we set up a Samba configuration to make
our backup server available in the same way. First, copy the default
Samba configuration file to a safe place. On Red Hat systems this is
/etc/samba/smb.conf. Now, we compose a new
simpler smb.conf. Using your favorite editor, type:

Save this as /etc/samba/smb.conf. Now we'll restart the Samba server by
typing /etc/rc.d/init.d/smb restart. On the first night, we copy
the entire filesystem to a1b1 with cp. You may have to unalias cp on your system to make sure the -i flag
is not set. This cp process may take 10-12 hours for 200GB. On the next night,
we copy the entire filesystem to a2b2, and so on, until we have a
complete copy for each day of the week. On subsequent nights, we use cp -au
and only copy files that have changed since the previous backup. This only takes an hour or
two and has the virtue that any files accidentally deleted from the main server are preserved on the
backup.

When changing drives each day, it is somewhat tedious to activate the
volume group and mount the logical volume, so we wrote a little Perl
script to do it for us. Using our favorite editor, we put this together:
like this:

and saved it as /home/me/mountback.pl. We make
this executable by typing chmod 700 /home/me/mountback.pl.

On the first line, our script creates a variable called $hold, runs the operating
system command vgscan to search for volume groups, pipes the output of
this command to grep, which searches for a line containing an a, followed by a digit from
0-9, followed by a b, followed by a digit from 0-9, and places it in $hold.
On the next line we create a variable called $hold2, run the substring command on the
line of text placed in $hold, grab the four characters after the first 39
and place them in $hold2.This should be the name of our volume group,
a1b1.

After running vgscan manually a few times it becomes obvious where the
name will appear in the line of text. On the third line we run vgchange
with the -a option (activate) and give it the y option for yes and the
name of the current volume group, which should be in $hold2. On the
fourth line we mount the logical volume at our previously created mount
point. On the last line we print to the screen the name of the volume we
have mounted. If we have done something wrong, we get some strange
messages here.
I have always intended to go back and add some error checking
to this script, but somehow or other I've never gotten around to it. It's
been working fine day after day, year after year.

So, to sum up our
procedure, to change out our backup drives we type shutdown -h now. When
the machine is finished shutting down, we remove the drives labeled
a1 and b1 and replace them with those labeled a2 and b2. Then, we start up the
machine and log on as root. Type /home/me/mountback.pl, and we should see
a2b2 mounted.

We're now ready for the current night's backup to happen. This is
best run as a cron job from the main server, which presumably has more
and faster processors, RAM and so on. I've been contentedly using a shell
script based on cp -au for some time and have recently begun experimenting with rsync.
I have every reason to believe tar would work fine as well. A simple script would be something like:

This mounts the logical volume from our backup server to a
mountpoint on the main server, copies all files that have changed since
the last backup and sends us an e-mail message giving the name of our
backup cron job, the time the backup started and the time it finished. The script
then unmounts the logical volume.

I hope this article is helpful to anyone who has found him or
herself in a similar backup situation and is considering removable IDE
drives as a possible solution. When I first started in this line of work,
I would study articles that showed how to do things I desperately needed to
learn, follow them carefully and get a good grasp on the process, only
to reach the end and have the author say something like, "Now all you
have to do is make a few changes to the usual configuration files, recompile the kernel and
you're done", which would render the whole thing useless to me. I have
tried not to let that happen here, but if anyone tries to implement this
solution and runs into
a snag, drop me an e-mail at mikef@farnerbarley.com, and I will try to
help you over the hump.

Mike Fogarty is the System Administrator for Farner, Barley and
Associates, a civil engineering and surveying company in central
Florida. He has a Linux System Administration Certificate from the
University of Illinois and is a member of SAGE, the System
Administrators Guild.

The reckless way that folks in this forum seem to disregard the seriousness of the EMP threat is akin to the reckless way that people for decades regarded the hurricane threat for New Orleans. May I remind everyone that our country published the Report of the Commission to Assess the Threat to the Unites Sates from EMP Attack, and that this report is about 200 pages? This matter is not for ostriches, it is for those are courageous enough to face reality.

The fact of the matter is that unless responsible data center managers take prudent steps, your technology is toast. It would only take a single nuke launched from the Gulf of Mexico or the Atlantic, exploded somewhere above the heartland. Short of a nuke, a Skud type missile launched from a commercial freighter would cause major Eastern seaboard destruction. Read the report from the Sage Policy Group which describes this threat. Short of malicious EMP attack, solar storms could destroy a major part of the grid including the large scale transformers. Our country no longer manufactures large scale transformers.

Let's please stop the jokes, or move this discussion to a more intelligent forum.

I have not yet used LVM although I intend to soon and have given backups a lot of thought. My LVM set up will be fairly large, consisting of several physical drives in the machine - lets say 5, and several sets of removable drives for backup, each of which would be 5 similar drives. I would back up from a snapshot to one of my backup drive sets. Is it necessary to mount all 5 backup drives at the same time to do the backup? What I would like is to have one removeable drive bay, and to be able to plug in each of the 5 backup drives one after the other from one backup set and copy the data there. Even better would be if the data on the backup drives was already useable so that it could be just updated.

If I had a non-LVM system, I could have 5 seperate partitions and do the backup very easily in this way with rsync, copying only the changed data, so doing an incremental backup while keeping a full backup of each of the 5 partitions available. Obviously this doesnt have any of the advantages of LVM, like combining the drives into one logical volume, or being able to add more drives easily.

If Ive made the decision to use LVM does this automatically lead to me having 5 monted drives and 5 removable drive bays, 10 drives in all or is there another way?

Good article to get some people thinking about backups. Just like most everything in the opensource (and real) world we put ourselves in, there is more than one "right" way to do any of the things we do. Some have mentioned logical ways to enhance what you have done (eg rsync to increase efficiency). I would put looking into a NBD (network block device) high on your research list. Redundant mirrored pairs are alos a good idea. Others have mentioned packages to others have spent much of their time creating for our collective good. I would like to mention rdiff-backup as one of those great options. It's availible at http://rdiff-backup.stanford.edu/.

I do have to agree with greatly with one of your statements: ..."I do things that doubtless make more experienced people shudder in horror."... Indeed, indeed (lol). First and formost, you have zero error checking or return code status checks for any part of the scripts (let alone the critical sections). Generally, you don't write directly to your own mail file (a log file is understandable)...that's kinda wierd.

Realy, in the end, it seems to work for you and that makes it great. Again, good article.

Once upon a time, Sun had a file system known as TFS, the "Transparent File System". TFS was designed to "overlay" a real file system, and store only changes made to the original files. So, say you have a read-only media. You could mount an empty hard disk over the read-only media using TFS. The files would allow write access(!), but saves would be written back to TFS (since the underlying media was read-only anyway).

In the classic context, the read-only system was NTFS. In a more modern context, the read-only system would be DVD or CD.

I sought far and wide for "TFS for Linux" to no avail for exactly this purpose: archival of low usage files to optical media while still enabling changes. Any info on such a technique would be appreciated.

for what period of time are the backups maintained?
what is the total number of backup ide drives required?
do you have an estimate of how many GB the 200GB would bzip2 into - are they ASCII/binary?
how long a time does a backup run require?

Once upon a time, Sun had a file system known as TFS, the "Transparent File System". TFS was designed to "overlay" a real file system, and store only changes made to the original files. So, say you have a read-only media. You could mount an empty hard disk over the read-only media using TFS. The files would allow write access(!), but saves would be written back to TFS (since the underlying media was read-only anyway).

In the classic context, the read-only system was NTFS. In a more modern context, the read-only system would be DVD or CD.

I sought far and wide for "TFS for Linux" to no avail for exactly this purpose: archival of low usage files to optical media while still enabling changes. Any info on such a technique would be appreciated.

Using LVM to get around the nastyness of tapes is a nice idea, have not seen that before.

Have been using 2 approaches over the last 2 years at a small business; fortunately the data sizes are not so big - we only need to keep about 10gig under control.

The first one is a thing I call TIABS (Transparent Incremental Archiver Backup System or TIABS Is Another Backup Scheme) which is an unholy collection of shell scripts.

The 10gig workspace is a samba share on the same box as TIABS. This space has about 150meg churn a day. The hardware is an old Soyo / Intel BX / Celeron box running Mandrake 8.1; this gives us zero problems (but we have a hardware twin just in case).

TIABS works using incrementals against a master image of the share, which I recreate manually every few months. Essentially overnight TIABS does a cp -au type thing to identify changes, populates a blank folder with these then goes back and makes _many_ links back to previous day's image for the unchanged files.

That means the archive area has N folders, one per day, each apparently with a "complete image" of the samba share workspace being backed-up. All browsable (read-only) across the share; that's nice as it allows (for example) our web designer to go look at her website from 6 months ago and run it live right out of the archive area. Everyone can see the archive using Explorer on their own PC; very little intervention from me (though I do check it's pulse etc regularly) and best of all - no evil tapes :))

The biggest problems came from dealing with MS filenames which allow all sorts of embedded $, multiple spaces etc. I could not find *nix tools which would not trip up somewhere (cp, cpio, tar, rsync - my mind just glazes over thinking about the problems). All the incremental detection finally had to be done by script (ouch). It got ugly.

The second approach is using a rotation of big disks. The workspace plus every partition on each PC gets imaged across the local network twice monthly (samba mounts + tar etc); these are just compressed images of complete partitions; very handy if a complete rebuild (=reversion to a working PC) is needed.

TIABS itself uses ext3 and sits on it's own disk perminantly in the server; the disk is an 60gig unit split into 4 partitions. This holds about 10 months worth of daily images.

I'll just mention that we are in deep coutryside with pathetic overhead powerlines - we get 2 or 3 outages a month. Thank goodness for ext3! Floods? Should I mention the river? and there's also.... :)

Another similar backup option which I'm about to try is to use external USB2 (or FireWire) hard disks. They're hot pluggable, not too expensive, and you can get small portable ones if you want to take them off-site. Data throughput may not be quite as high as IDE (especially with a 2.4 kernel - USB2 support appears to be much faster in 2.6), but that may not matter if you're happy to leave backups chugging away overnight. Presumably they could be combined using LVM or RAID in just the same way as described in the article?

I use a usb-ide enclosure for one of those cheap ide bays. In windows, i can simply "eject" the hard drive bay device, replace the hard drive, and then plug the drive back in without rebooting. My guess is that you could do the same thing in linux by re-loading the usb mass storage driver. It wouldn't cost very much to do, also it would save the trouble of rebooting

A nice article indeed. Rsync has already been mentioned, so I don't need to do that again.
There is one thing that scares me a little: what happens if you find out a file has been corrupted two weeks ago? You will have no backups before that do you? Would a setup with CVS or a more modern version control system not be a better solution then?

Another thing: RH 8 is quite old. In fact it has already been EOL-ed by redhat. It uses LVM 1, which uses a dedicated kernel module. In 2.6 kernels, this has been replaced by LVM 2 and a smaller more generic solution in the form of the device mapper module.

We have been using a similar approach for several years. But there are still a few remaining issues unresolved.

1. We give each user a removable IDE to backup their own data. But the removable IDE is not Hot Swappable. The BIOS has to find out that there is a disk exists in the IDE channel, and pass such information to the Linux boot procedure, which, in turn, would enable the Linux kernel to have access to the disk. This means that EVERYTIME a user has to reboot the workstation he/she is using. Over the years, we found such approach is not very user friendly. Sometimes there are other people using that workstation via ssh or other remote login process. Sometime other user left a heavy number crunching process to run on the workstation and it is not ethical to reboot the workstation. Furthermore, users sometime accidentally switch off the removable disk (the removable caddy has a power switch) before a complete shutdown of the machine and hence resulting in data corruption (on disks using ext2 system). As a result, our users do not want to use the removable disk as a means of backup.
2. At the institution level, we are providing a monthly full backup and daily incremental backup to ensure that a user can retrieve his/her data of any version over a period of a month. Four year ago we built a backup server using 8 80GB IDE disks (the largest at that time) in RAID 5 mode (a net capacity about 540 GB). An open source package (afbackup http://sourceforge.net/projects/afbackup) is configured to implement our backup policy to backup data from 2 NFS servers over the LAN. The backup process has been running trouble free for 4 years (great Linux system

1.5 GB should be possible with a 12-port 3ware SATA-controller (3Ware Escalade 8506-12MI) and some large disks.

Reading the story, my first concern is the JBOD-like structuring of the drives, if 1 (one) drive fails the backup is toast.

I have implementet something similar on a 250Gb (net) scale for a customer. I used a h/w (p-)ata raid controller and running the drives in RAID5 (ok, this does cost approx 50% more gross-space) and a very fault-tolerant file system (reiserfs) as e2fs in my opionion is too fragile for backup purposes.

I second the rsync comment above, I implemented it with hourly versioning so that there is always data for each hour the last day, then each day the last week and each week for the last two months and each month one year back.

This huge number of backup takes up approx. 130% of the data, but that is highly dependant on your data's update frequency.

After all for most companies their data is their business today, so the good old "better safe than sorry" applies more than ever.

The system has been running automated without human intervention for more than one year now and the client is happy.

I recently implemented Bacula for our backup needs. We don't have such a high volume to backup (only 15 GB), but I'm sure Bacula can manage volumes of 200 GB.
The added advantage is that a catalog is maintained in an SQL database, so finding a version backed up on a particular date is very easy.It can back up to files or to tape. We chose to backup to removable USB2.0 storage (to be carried off site) and to DVD+R (smaller set of files (compressed) for historical backup).
Bacula can backup files from Linux, BSD, Irix and Windows clients over the network.
In your case I would create a RAID5 array with 7 or 8 200 GB disks for the daily backups and for holding the other ones before writing them to external media. Daily backups don't need more maintenance than checking the logs to see whether everything went fine. On a weekly basis the backup made on Friday can be copied to removable storage. (If you want, incrementals or differentials against these weekly backups can also be made parallel to the daily full backups).
That leaves the historical backup for which you could use removable storage or fixed IDE disks, as you do now.
Bacula takes care of maintaining the database.
One thing that is not possible, is to access your files directly without restoring them first. This means you can't accidentally modify them either though, while you are busy, trying to find the correct file.

Of course, if you are happy with your current solution, don't change a winning horse. I'm just trying to describe an alternative. One with which I'm very pleased and that I think the world should know about.

A very useful tool is rdiff-backup which will do much the same
as rsync only with increments (ie. it will allow you to maintain
a mirror of the last backup and then increments for the last 7 days
or whatever).

A nice touch is to NFS mount it read only on workstations so that
users can restore their own backups when they accidentally
delete a file etc.

I have a similar solution here, but we took another approach: 5 120GB drives in a RAID5 array (software Linux kernel driver) on a dedicated machine used as a 'data dump'. We also have a tape library (a big one) but due to the tapes' failure rate the thing is next to useless - sometimes takes days of retries to read a tape written a year of two ago (we store meteorological simulations data there). And the tape library absolutely sucks at file management - files aren't erased when deleted from the library virtual filesystem, library admin needs to trigger a 'compaction' process to actually erace the data. So think what happens at any approach to manage differential backups.
OTOH the RAID array, cheap and not very fast is a good medium to do differential backups and store data that may beome useless (and should be deleted) after a short period of time.

An EMP attack would ruin all that backupped data on a harddrive or tape. So that's why I'd buy DVDs for backing up data. It's optical and should be more resistent to EMP attacks. Which I think will happen someday.

I think if we ever experienced an EMP attack it would most likely be followed by some other sort of attack. Like by guys with guns and bombs and s*!t! I don't know about anyone else but, if that happens, the VERY LAST thing on my mind will be the safty on my safety of my backups.

Have you an EMP-safe people-backup-and-restore system? Without it, your point is ... pointless: who would be there to :
a) still suffer from not being able to restore
b) try to do it anyhow, if the media are there

BTW: those "EMP attacks" are often mentioned together with nuclear war. Will your DVD be heat-resistant? Or do you burn DVD's in heat-resistant glass? (asbestos?)

EMP weapons have been tested since the '50, and no unclassified weapon has come out of it. If you do not live in a political hot contry, your fear of EMP attack should be equal to fear of asteroid impact or other natural disasters; not very likely.

EMP weapons have been hyped by many on the net, but never has any failed or succesful attack been reported anywhere. Computers by design hare quite good at withstanding an EMP attack: the are enclosed in a faraday cage with all wires coming in fused at some point. Computers usually need to keep 1-3 GHz frequencies inside, so they are pretty resistant. An succesfull EMP attack might fry keyboards and other periferals, and it may even kill the mobo, it will not erase the hd, that is a little faradays cage (the hd casing) sitting in a bigger faraday cage.

You need a device using high explosives or massive capacitor banks to do any damage against computers; the much hyped magnetron dish on a truck will generate a narrow spectrum that will not penetrate computercases at all.

I'm attempting to setup a small network for my research lab. I'm interested in distributed simulation. Currently I have three computers, including a laptop. The operating systems are Win2000 and WinXP (laptop). I want to add a linux server and want to run win2000 in at least one of the computer. Is this possible? How can I get a quick start?

200GB / 4.7GB = 42.55 rounded up to 43. So, they'd need 43 disks!!!!!! I'm not even going to comment on how utterly stupid that idea is. Oh wait, I just did.Guess it's because that's such a stupid idea.