Recovery of RAID and LVM2 Volumes

To recover, the first thing to do is to move the drive to another
machine. You can do this pretty easily by putting the drive in a USB2
hard drive enclosure. It then will show up as a SCSI hard disk device,
for example, /dev/sda, when you plug it in to your recovery computer. This
reduces the risk of damaging the recovery machine while
attempting to install the hardware from the original computer.

The challenge then is to get the RAID setup recognized and to gain
access to the logical volumes within. You can use sfdisk -l
/dev/sda
to check that the partitions on the old drive are still there.

To get the RAID setup recognized, use mdadm to scan the devices for
their raid volume UUID signatures, as shown in Listing 3.

This format is very close to the format of the /etc/mdadm.conf file that
the mdadm tool uses. You need to redirect the output of mdadm to a file,
join the device lines onto the ARRAY lines and put in a nonexistent
second device to get a RAID1 configuration. Viewing the the md array in
degraded mode will allow data recovery:

Edit /etc/mdadm.conf so that the devices statements are on the
same lines as the ARRAY statements, as they are in Listing 4. Add the
“missing” device to the devices entry for each array member to fill out
the raid1 complement of two devices per array. Don't forget to renumber
the md entries if the recovery computer already has md devices and ARRAY
statements in /etc/mdadm.conf.

If md devices show up in /proc/mdstat, all is well, and you can move on
to getting the LVM volumes mounted again.

Recovering and Renaming the LVM2 Volume

The next hurdle is that the system now will have two sets of lvm2 disks
with VolGroup00 in them. Typically, the vgchange -a
-y command would
allow LVM2 to recognize a new volume group. That won't work if devices
containing identical volume group names are present, though. Issuing
vgchange -a y will report that VolGroup00 is inconsistent, and the
VolGroup00 on the RAID device will be invisible. To fix this, you need
to rename the volume group that you are about to mount on the system by
hand-editing its lvm configuration file.

If you made a backup of the files in /etc on raidbox, you can edit a
copy of the file /etc/lvm/backup/VolGroup00, so that it reads VolGroup01
or RestoreVG or whatever you want it to be named on the system you are
going to restore under, making sure to edit the file itself to rename
the volume group in the file.

If you don't have a backup, you can re-create the equivalent of an LVM2
backup file by examining the LVM2 header on the disk and editing out the
binary stuff. LVM2 typically keeps copies of the metadata configuration
at the beginning of the disk, in the first 255 sectors following the
partition table in sector 1 of the disk. See /etc/lvm/lvm.conf and man
lvm.conf for more details. Because each disk sector is typically 512 bytes,
reading this area will yield a 128KB file. LVM2 may have stored several
different text representations of the LVM2 configuration stored on the
partition itself in the first 128KB. Extract these to an ordinary file
as follows, then edit the file:

You will see some binary gibberish, but look for the bits of plain
text. LVM treats this metadata area as a ring buffer, so there may be
multiple configuration entries on the disk. On my disk, the first entry
had only the details for the physical volume and volume group, and the
next entry had the logical volume information. Look for the block of
text with the most recent timestamp, and edit out everything except
the block of plain text that contains LVM declarations. This has the
volume group declarations that include logical volumes information. Fix
up physical device declarations if needed. If in doubt, look at the
existing /etc/lvm/backup/VolGroup00 file to see what is there. On disk,
the text entries are not as nicely formatted and are in a different order
than in the normal backup file, but they will do. Save the trimmed
configuration as VolGroup01. This file should then look like Listing 6.

I reinstalled my linux box using ubunbu 10.04 alternate install as I setup LVM & raid.
Before the upgrade I copied data to a degraded RAID1 500GB with ext4 drive all working fine. Had even run fsck before the upgrade. Data was there.
I should have unplugged this drive, before the reinstall.
I manually partition drives, but did not change the drive with my data on it. I did not add this raid/drive to any LVM groups.
But when I rebooted I when looked at degraded RAID1 (yes it did mount) there is no partition! I installed and ran gparted (GUI) to find the drive marked as lvm flag!

Is there anyway to rebuild the partition on the degraded RAID1 drive
I have been using clonezilla CD using to image the disk to another one the same size, to try various things.
I have tried testdisk no luck, it can see the partition but not the files.
I tried photorec but all got I was files with ramdom file names and no directories.
What does LVM write to a volume? Does it destroy the whole FAT?

This is very interesting. However I have seen that fdisk or sfdisk was used, I would strongly suggest to only use gnu parted. such as "parted -l" so it will produce the same output than "fdisk -l" or "sfdisk -l" but parted does also fully support Intel EFI/GPT partition tables.

If I may I can also suggest some reading that I wrote about similar issue.
It's an howto on how to Create a Raid5 under Linux RHEL5.4 using md, lvm and ext4 filesystem. (look at http://panoramicsolution.com/blog/?p=92 )

In an enterprise production environment, perhaps there are situations where LVM over Raid XX is desireable, even essential. But a lot of the posts here seem to reference home users trying to recover their personal photos, and stuff like that.

The redundancy of Raid is only a benefit if you can figure out how to get your data when a drive fails. The flexibility of LVM has to be weighed against the increased complexity of recovering from a failure.

I guess my bottomline is, don't make things more complicated than they need to be.

And when you're building your new system, before you put any vital data on it, break it and see if you can recover. Takes notes about how you did it, and save those notes somewhere OTHER than on the system you're working on! You'll thank yourself later. Perhaps you've already found some useful links to help on the subject. But in 5 years, when a drive fails, will those links still be active? Don't count on it. Make a recovery plan, and keep it somewhere safe.

If you break it and you can't fix it, then you either need to learn more or simplify your system configuration.

And, of course, back up your data. And back it up onto more than one device (you can alternate daily/weekly, whatever is comfortable for your situation). If the lightning strike picks the moment you're doing backups, and it hoses the "if" AND the "of", what are you going to do then?

If possible, for critical situations, in addition to backups for the data, have backups for the HARDWARE as well. These fancy NAS RAID devices are great, until they fail. If all you have is the NAS box, and some consumer type PCs, exactly what are you going to stick those 3, 4, or 5 drives into? Rest assured that by the time your NAS box fails, you won't be able to buy an exact replica and just stick the drives into it -- that model will be in the pages of history. Then you will get to have a "learning experience".

When I get to the next line, I noticed that there is nothing inside the mdadm.conf file EXCEPT the output of the mdadm command. So now my mdadm.conf file contains ONLY the following:
ARRAY /dev/md1 level=raid1 num-devices=4 UUID=76a49610:8be3458e:6a4c59a2:58407b7e devices=/dev/sdb1,missing

Wehn I try the next two commands, I get:
mdadm -A -s
mdadm: no devices found for /dev/md1

My main (4 year old) desktop computer popped it's motherboard and I'd just received the machine I bought to replace it.

There wasn't a spare power connector in a location where I could put my old drive in but I figured no great loss - the new drive was larger and I'd just rely on the AMANDA backups I'd been running to an outboard USB drive.

Loaded up CentOS, attached the USB drive I'd been using for backups, manually restored the Amanda configuration, started extracting the data I needed and - poof. After a few hours of messing about determined that my backup drive picked that time to die and it wasn't just a matter of it spinning down when it wasn't supposed to (Seagate FreeAgent ICAC).

So it was out to the local store to pick up an external SATA dock to put the old drive in and I was presented with the problem of getting access to my data. Thank you, you just saved me untold hours of sussing all that out via the man pages!

So I've been trying to recover my LVM2 over RAID 5 off and on for about a year now, yeah tell me about it! It seems when I decided to upgrade my Ubuntu distro, I didn't think to backup, of course, I completely lost my LVM2/RAID 5. I've tried many processes and software to recover but nothing has proved worthy. I landed on this great tutorial in hopes of recovery only to be stopped dead in my tracks, as it seems I no longer have an md RAID device (dev/md0). I followed this tut diligently and stopped at Listing 3. for obvious reasons. To sum it up, I'm hoping I can receive some direction as to how I may be able to recover my data with my particular situation.

The following is the output of the commands leading up to my dilemma as well as a few more that may help. A huge thanks in advance!

Combine the logical volumes into RAID sets, making /dev/mdn's out of 2 or more LVM logical volumes.

So, is it LVM2 over RAID5? Or maybe RAID5 over LVM2 (with maybe some LVM2 on top...)

HTH.

- Ab.

p.s. LVM is great and all, but I really like the idea of being able to pull a physical disk drive out of the wreckage and having it full of complete files and stuff. So, for me, it's RAID1 and no LVM. I guess it's 'cause I'm old school and never really trust my backups (As if!) to be current at the moment of truth. Or something.

I did a dd copy of the start of the physical volume as suggested and found that some identification fields after the timestamp were needed to make a workable config file for vgcfgrestore, i.e. I needed the descriptive lines from:
MyVolName{
id = "xxxx..."
seqno = ...

down to
# Generated by ...

followed by
contents = ...
version = ...

(I also included description, creation_host and creation_time - these fields probably aren't required).

My lacie ethernet disk (with disks in raid 5 configuration) doesn't boot anymore. I need to recover some data, so I connected the three hard disks that form the array to my PC. booting from a System Rescue live CD and following the guidelines of this article I was able to rebuild the array, but I had to stop at point 6, when trying to recover LVM volumes, as I get the following string from the first sectors of the disk and I don't know how to proceed:

Today a customer brought here a LaCie Ethernet Disk 2G that didn't boot, just like yours. I connecter the disks to a spare computer and booted with a Knoppix CD, and after some hacking I finally managed to get to the data. Not sure if it actually uses LVM or what, but I couldn't recover a valid LVM configuration so I had to do without.

The XML-like data at the beginning of the volume is actually very useful because it tells you where the data partition is located within the md device. The first sector is 22272, i.e. 11403264 bytes from the beginning. To reach it, create a loop device:

# losetup -o 11403264 -r /dev/loop1 /dev/md1

(I assume you already assembled the RAID device)

Then just mount the loop device:

# mount -o ro -t xfs /dev/loop1 /mnt/

(note that eveything is mounted read-only, just in case)

That's it, mount a USB disk or a network share and get that stuff to a safe place!

I'm having similar trouble with my Lacie 2Big Network Drive which was configured as Raid1.
It would be wonderful if the loop device solution were to work for me.
However, my understanding of linux is very limited and, apparently, just copy pasting the above into my terminal window and editing (the obvious) volume name is not enough.

The system-disk in my file-server just crashed the other day and as I was installing it all over again tonight I had no problem figuring out how to mount my old LVM's that wasn't in RAID-configuration.... but then my precious backup-volume for documents/photos I was using RAID and couldn't figure out how to get it back online until I googled and found this article.

While juggling drives and trying to fix an annoying boot problem, I managed to overwrite the MBR of one of the drives. I had unwisely chosen to use the entire device as an LVM PV (instead of a partition spanning the whole drive), so that whacked the PV metadata. Many thanks to Richard for writing the original article, and in particular to Toby Fruth, whose reply led me through the steps to recover my PV and all the LVs on it. I was fortunate in not having to reconstruct the VG config file from raw sectors; LVM made backup copies of the VG configs every time I made a change, so I had a recent backup copy at hand.

I ran into this issue while trying to recover my original drive for my home server, which in turn of google searches I found this thread. Than k you so much for the excellent explanation of this issue!

I took a differn't approach however once I understood what was in conflict. I popped in a new drive, that I wanted to recover the data too, and just reinstalled my OS but did NOT use LVM this time. Just a good old fashioned swap/boot/and root partition scheme and then re-ran linux rescue which mounted the old filesystem easily, and was very easy for me to mount the secondary disk. Copied all files over, and put back my configs. All said and done, just a couple hours and my system is back up to normal after a drive failure. AWESOME!

After you installed the failed raid disk into the recovery box (or hooked it up via usb), couldn't you have booted the recovery box with a Live CD and simply mounted only the drive partitions you needed?

In otherwords, just don't mount the drive in the recovery box that had the equivalent vol group. That way there would have been no conflict right?

If i understand the problem correctly, the problem is NOT that the raid drive does NOT HAVE AN LVM CONFIG (or that it was damaged), it's just that it's the SAME as the recovery boxes LVM config (e.g. has the same volgroup name) which prevents it from being seen (i think?)

Another way of asking the question is this. If the recovery box did NOT have any LVM partitions or LVM config native to it.. could i simply plug the raid drive in and the recovery box would automagically find the raid LVM partitions or would I still have to something else to make it work? If I have to do something else to make it work, i'd totally appreciate it if you could explain what i would need to do (either a subset of the above article steps or just a streamlined set of guidelines).

That would fully help me understand this topic completely because i imagine at some point, if i have a system just like this, i'm going to need to recover it some day. And it would be pretty easy for me to NOT use LVM on the target recovery box.

> ... couldn't you have booted the recovery box with a Live CD and simply mounted

only the drive partitions you needed?

That was what I was originally hoping to do, but that did not work automatically. RAID arrays on USB-connected drives are not available to the system when it does its first scan for RAID arrays. Also, if the recovery box has a volume group with the same name, it will not recognize the newly-attached volume group.

I have used USB RAID arrays in production, and you have to take some extra steps to activate them late in the boot process. I typically use a script similar to this to do the job:

#!/bin/sh
#
# Mount a USB raid array
#
# Call from /etc/rc.d/rc.local

DEVICE=/dev/ExampleVolGroup/ExampleVol00
MOUNTPOINT=/mnt/ExampleVol00

# Activate the array. This assumes that /etc/mdadm.conf has an entry for it already
/sbin/mdadm -A -s
# Look for LVM2 volume groups on all connected partitions, including the array
/sbin/vgscan --mknodes
# Activate all LVM partitions, including that on the array
/sbin/vgchange -a y
# Make sure to fsck the device so it stays healthy long-term
fsck -T -a $DEVICE
mount $DEVICE $MOUNTPOINT

> In otherwords, just don't mount the drive in the recovery box that had the equivalent vol group. That way there would have been no conflict right?

That's mostly right. You'd still need to scan for the RAID arrays with 'mdadm --examine --scan $MYDEVICENAME' , then activate them after creating /etc/mdadm.conf.

If you had other md software RAID devices on the system, you might have to fix up the device numbering on the md devices.

> If the recovery box did NOT have any LVM partitions or LVM config native to it.. could i simply plug the raid drive in and the recovery box would automagically find the raid LVM partitions or would I still have to something else to make it work?

On a recovery box without any software RAID or LVM configuration, if you plugged the RAID drive directly into the IDE or SATA connector, it might automagically find the RAID array and LVM volume. I have not done that particular experiment, you might try it and let me know how it goes.

If the drive was attached to the recovery box using a USB enclosure, the RAID and LVM configurations probably won't be autodetected during the early boot stages, and you'll almost certainly have to do a scan / activate procedure on both the RAID and LVM layers.

You might have to scan for RAID partitions, build an /etc/mdadm.conf file, and then scan for volume groups and activate them in either case.

The most difficult part of the recovery outlined in the article was pulling the LVM configuration out of the on-disk ring buffer. You can avoid that by making sure you have a backup of the LVM configuration for that machine stored elsewhere.

I emailed Mr. Bullington-McGuire, for I had created a self-inflicted dilemma. I had run the following command:

pvremove /dev/sdb2 -f

Why? Because I thought I needed to remove LVM data from a drive in order to mount it under a new install, which had been on a different drive. I could have done it this way:

mount /dev/VolGroup00/LogVol00 /mnt

assuming that another LogVol00 was not already mounted and that a /dev/VolGroup00/LogVol00 did not already exist. Of course, they originally did exist under the new install on the new drive, so I did another new install, using different LVM names on the new drive.

So, I managed to recover from the pvremove by doing a pvcreate, using a restore file created with the instructions in this article.

just making a backup and poof the power goes :( on reboot i can't get to my lvm and my 50gig backup is awol!!
your ickle guide saved my life as the missus sims2 data was on there and its more then my life is worth to lose that

Just another saved me comment! Thanks! I thought I was hosed, but this article pointed me in the direction I needed to go to recover my essential data. Yes, yes, I do backups, monthly and archive media every 6 months, but now, I have learned: always RAID1 or RAID5, no LVM, test UPS control regularly, and invest in large external eSATA/USB/Firewire drives to do nightly incrementals and keep them unmounted when not in use.

Trending Topics

Webinar: 8 Signs You’re Beyond Cron

Scheduling Crontabs With an Enterprise Scheduler
11am CDT, April 29th

Join Linux Journal and Pat Cameron, Director of Automation Technology at HelpSystems, as they discuss the eight primary advantages of moving beyond cron job scheduling. In this webinar, you’ll learn about integrating cron with an enterprise scheduler.