Restoring Data from Hardware based clones/snapshots

This procedure is based on a taking hardware level SAN snapshots and clones on a NetApp filer of a Red Hat Enterprise Linux 5.1 system and using these to restore data to a running production system. This should work with other setups, but your mileage may vary. Please add/update if you find you need to adapt this procedure for your storage device.

Scenario: You have an issue on a production system where you need to restore data from last week. The server is running a live production service, with data stored on a LUN exported over fibre channel (you could also apply this to iSCSI and other transport protocols), this LUN has hardware snapshots taken 3 times daily on the SAN. You want to copy these files back to the production LUN one at a time, the clone you create from the snapshot has the same Physical Volume (PV) and Volume Group (VG) UUID's.

For the purpose of this example:

/dev/mapper/mpath0 - Boot LUN (This system is also booting from SAN disk)

2. Check servers LVM filter and make sure it can only see the application LUN's and boot LUN's, you can do this through the /etc/lvm/lvm.conf filter e.g.:

filter = [ "a|/dev/mapper/mpath0|","a|/dev/mapper/mpath1|","r/.*/" ]

3. Re-scan SCSI bus to find the new LUN: (or whatever you need to do for your transport method)

Issue:

echo "- - -" > /sys/class/scsi_host/host[0-3]/scan

Monitor tail -f /var/log/messages and multipath -d -l and all paths to the new LUN should be visible

4. At this point you want to make sure no-one else is running LVM commands on the system!

5. Run

vgimportclone.sh -n vgAPP-clone /dev/mapper/mpath2

Which will change the volume group name to vgAPP-clone and change all the associated uuids. If you are using an older (pre 2.02.28) version of LVM and vgimportclone.sh is not installed on your system you can either pull it from CVS (it is known to work back to 2.02.21) or follow the "Changing uuids on a clone by hand" procedure below to change the uuids by hand (vgimportclone.sh is an implementation of this procedure with error checking)

6. You should now be able to mount your /dev/vgAPP-clone volume groups, file systems and recover data as required

Changing uuids on a clone by hand

1. Create a temporary LVM system directory, copy lvm.conf to this and set LVM_SYSTEM_DIR to point to the directory:

mkdir /lvmtemp

cp /etc/lvm/lvm.conf /lvmtemp

export LVM_SYSTEM_DIR=/lvmtemp/

NB: The purpose of this temp LVM system directory is so that if your box is fenced/rebooted mid-way through this process, your production services should not be affected by any changes you have made

2. Change filter in your temporary /lvmtemp/lvm.conf file to only see the cloned LUN and your boot LUN:

filter = [ "a|/dev/mapper/mpath0|","a|/dev/mapper/mpath2|","r/.*/" ]

3. Run pvscan: Output should be similar to the following and only your Boot and Cloned application LUN should be visible:

PV /dev/mapper/mpath2 VG vgAPP lvm2 [6.00 GB / 0 free]

PV /dev/mapper/mpath0p2 VG vgRoot lvm2 [32.00 GB / 0 free]

4. Change the UUID's of the cloned PV & VG and rename to avoid duplicates and avoid clashes with the application LUN:

pvchange --uuid /dev/mapper/mpath2 --config 'global{activation=0}'

vgchange --uuid vgAPP --config 'global{activation=0}'

vgrename vgAPP vgAPP-clone --config 'global{activation=0}'

NB: The key here is the switch --config 'global{activation=0}' this deactivates device-mapper interaction and thus the commands only change the LVM metadata and not the devices on the system (otherwise the system will think you are changing the VG used for the production application)