Saturday, March 29, 2008

Implementing Veritas Flashsnap for database backups

Snapshot Technology is a fast and efficient method to take backups of large scale databases. Unlike RMAN which is Oracle based, Snapshots uses mirroring technologies managed by the system/storage. There are many variants of snapshots available and the most commonly used method is one in which an entire copy of the database is made. This permit off-host backups and can be incrementally refreshed on a daily basis.

Veritas Flashsnap is a Point in Time Data Mirroring solution using Veritas technologies. Veritas Flashsnap is tightly integrated into Veritas Volume Manager (VxVM) and Veritas Filesystem (VxFS). It requires a separate license to be purchased and enabled (at least for versions 4.x). It is important to note that Flashsnap is VxVM Volume based rather than LUN based (as Shadow Image or Timefinder).

This article is an attempt to create a reference configuration for implementing Flashsnap to backup an Oracle database using Storage Foundation for Oracle 4.0.

Flashsnap has many options available as to the kind of Point in Time Mirror to be deployed. This article focuses on the Full sized Instant Snapshot option by which a mirror is taken of the entire database on dedicated LUNs which can then be imported onto a Media Server for backup to tape.

Since it is VxVM volume based, the LUNs used for the mirror can be on different storage array (e.g. a low cost array using SATA drives) than your primary storage. If you were to use Shadow Image etc, then you would need to dedicate storage on the primary array as used by the host.

The rate of synchronization between source (database) volumes and target (dedicated backup) volumes can be controlled with fine granularity.

Unlike Array based solutions, VxVM allows you to split and deport the target volumes as a different disk group from the parent. No need for complicated scripts to change the disk group name when you wish to import the backup LUNs on the source system to do a recovery.

In most big shops, the primary storage will be an Enterprise Array shared between many hosts. When using Array Based technologies, during the synchronization of large number of LUNs, a noticeable impact will be observed on the Primary Array thus affecting every attached host on the array. However when doing Flashsnap, the impact would be only to the host running the flashsnap operation rather than the primary array.

The source and target volumes need not be of the same configuration and layout. Unlike Array based technologies, one can have different LUN configurations and also different sizing.

One can deploy a Flashsnap solution in less than 30 minutes. It is easy to implement and works well if configured and managed correctly (like all good solutions done right the first time).

System Configuration

The system configuration and specifications as was used in this article is documented below.

The first task would be to check the License for Flashsnap. Both Flashsnap and FastResync need to be enabled. The Flashsnap license also allows the vxdg split/join command to be performed allowing volumes to be split onto a different disk group.

In order to ensure that Veritas Volume Manager is adequately tuned to meet enterprise demands, the below variables were set.

/etc/system changes from default:

set maxphys=8388608set vxio:vol_maxio=16384

set vxio:vol_maxioctl=131072

set vxio:vol_maxspecialio=16384

set vxio:vol_default_iodelay=10

set vxio:voliomem_chunk_size=131072

set vxio:voliomem_maxpool_sz=134217728

Tthe volpagemod_max_memsz variable is set to 1024m in a startup script.

/usr/sbin/vxtune volpagemod_max_memsz 1024m

The vxiod parameter is set to 32 by editing the vxvx-startup2 init script.

Emulex Configuration changes

lun-queue-depth=30;tgt-queue-depth=256;num-iocbs=2048;num-bufs=1024;

Storage Connectivity

The SAN was rated at 2Gbit/sec and the connectivity is as shown below.

Primary Storage - lpfc0 and lpfc1 were on IO Board 0 and lpfc2 and lpfc3 were on IO Board 1.

Backup Storage is connected in an exactly similar manner except that dedicated HBA’s were numbered lpfc4 to lpfc7.

Flashsnap Configuration

Henceforth the volumes to be backed up are called Primary Volumes and the Mirror Copies of the volumes are called Backup Volumes. Same convention for Primary and Backup LUNs.The basic configuration steps are

Ensure that the host sees the backup LUNs as presented by the Storage.

Configure the backup LUNs and add them to the same diskgroup as the Primary Volumes.

Create Backup Volumes using the backup LUNs as of the same size as the Primary Volumes.

Prepare the Primary and Backup Volumes for Flashsnap

Perform the initial sync.

Split the Backup Volumes onto a different diskgroup and import on the media server for backup.

Hence going forward, do a incremental refresh as required. This would entail importing the Backup Diskgroup and joining it to the Primary Diskgroup to perform the refresh.

Step1 and Step2 –

It is assumed that Step 1 (Host sees backup LUNs) and Step 2 (Backup LUNs are initialized and added to same diskgroup as Primary Volumes) is completed successfully. For the sake of clarity, the Backup LUNs are initialized and added to the Primary Diskgroup using ‘SN’ as the prefix for the disk name.

While one can create the backup volume of any layout, it is ideal if the same layout were maintained. It is important to note that the Backup Volumes are created using the Backup LUNs and using the –U fsgen option. The source volumes would also ideally be created with the –U fsgen option.

The below volume SNtest05 is going to be Backup Volume for Primary Volume test05. All Primary volumes which need to be backed using Flashsnap need to have Backup Volumes similarly created.

Step 4 is to prepare the Primary and Backup Volumes for Flashsnap. This basically involves creating DCO logs for the volumes. Primary volumes need to use a disk belonging to the Primary Disks for DCO logs and similarly Backup Volumes need to use a disk belonging to the Backup Disks for the DCO logs. In order to meet performance requirements, it is ideal if disks were dedicated for the sole purpose of DCO logs only. These disks would be reserved and not used in any volume creation.In the example given below, testdg06 is the disk going to be used for DCO logs for all the Primary volumes and SNtestdg87 is used for the Backup Volumes.

Using a regionsize of 32k allows for faster resynchronizations though the initial sync may take more time.

In order to verify that Fast Resync and Instant Snapshot has been enabled successfully for the volume, the below tests can be done.

bash-2.03# vxprint -g testdg -F%fastresync test05on

bash-2003# vxprint -g testdg -F%instant test05on

Step 5:

Step 5 is to run the initial sync which will associate the Backup Volumes with the Primary Volumes and do a mirror copy of the Primary Volume onto the Backup Volume. It is performed using the vxsnap command.In this example, the iosize parameter has been set to 12M which is aggressive. The default is 1M. Running at 12M will impose a significant overhead on the system during the time the sync operation runs – especially if synchronizing a large number of volumes at the same time.

The above command will create a snap record and associate the Primary with the Backup Volume. It will automatically run in the background. Monitoring on the progress can be done using vxtask list or vxsnap syncwait.

bash-2.03# vxsnap -g testdg syncwait SNtest05

The syncwait command will return to the prompt only when the snap process has completed.A vxprint output of the Primary Volume or the Backup Volume will now show the sp record.

Step 6 is now to split the Primary diskgroup and separate the Backup Volumes into a different diskgroup. Doing so allows the Backup Diskgroup (Volumes) to be deported and hence imported on the Backup/Media Server.

bash-2.03# vxdg split testdg SNtestdg SNtest05

This command will move the volume SNtest05 to a new diskgroup SNtestdg. There are multiple ways to do a split – either using the –o expand option or specifying all the volumes in the command

bash-2.03# vxdg split testdg SNtestdg SNtest05 SNvol2 SNvol3 etc

or

bash-2.03# vxdg –o expand split testdg SNtestdg SNtestdg71

bash-2.03# vxdg deport SNtestdgThe –o expand option will then split the Primary Diskgroup and move all volumes using the into the SNtestdg diskgroup – in this case the disk is SNtestdg71.

Once this is complete, then a deport of SNtestdg can be performed and SNtestdg can be imported on the Media Server for backup to Tape.

Step 7

For performing an incremental refresh of Backup Volumes from Primary Volumes, the Backup Diskgroup SNtestdg needs to be imported on the Primary Host, joined to testdg and the refresh performed.

In order to take a proper and valid backup of an Oracle database, the below is the sequence of operations to be performed. In order for this to be successful, the archive logs must be placed on a dedicated mount point.

Import and join the Backup Diskgroup to the Primary Diskgroup.

Alter all tablespaces except the temporary tablespace into begin backup.

Run a refresh of all the volumes including archive logs and wait till it completes.

Alter all tablespaces except the temporary tablespace into end backup.

Sync the archive logs (only) mount point again and wait till it completes.

Split the Backup Volumes into the Backup Diskgroup and deport the Backup Diskgroup.

Import on Media server and run backup to tape.

The above sequence can be done everyday as part of the backup cycle.

Performance Impact to the host during the initial sync and further refreshes

The entire refresh of 3.2TB took around 6 hrs thus showing an average write throughput of 150MB/sec with a peak of around 250MB/sec. The system was showing an aggregate throughput of 300-500MB/sec (read+write). However there was a considerable load on the system with a run queue of around 20 for the period of the sync. Further refreshes for a database of 1.2TB (used size) with a 50% read-write ratio (150GB of redo/day) takes around 30 minutes on an average.