Protecting Your Tables Against Application Errors

“Do applications need to backup data in Windows Azure Storage if Windows Azure Storage already stores multiple replicas of the data?” For business continuity, it can be important to protect the data against errors in the application, which may erroneously modify the data.

If there are problems at the application layer, the errors will get committed on the replicas that Windows Azure Storage maintains. So, to go back to the correct data, you will need to maintain a backup. Many application developers today have implemented their own backup strategy. The purpose of this post is to cover backup strategies for Tables.

To backup tables, we would have to iterate through the list of tables and scan each table to copy the entities into blobs or a different destination table. Entity group transactions could then be used here to speed up the process of restoring entities from these blobs. Note, the example in this post is a full backup of the tables, and not a differential backup.

Table Backup

We will go over a simple full backup solution here. The strategy will be to take as input a list of tables and for each table a list of keys to use to partition the table scan. The list of keys will be converted into ranges such that separately backing up each of these ranges will provide a backup of the entire table. Breaking the backing up of the table into ranges this way allows the table to be backed up in parallel. The TableKeysInfo class will encapsulate the logic of splitting the keys into ranges as shown below:

BackupTables is the entry method into the backup process. The client can provide the TableKeysInfo for each table. For example if partition keys are GUIDs then here is how we can invoke the BackupTables method:

The BackupTables now iterates through each table and for each range in the table, it invokes BackupTableRange which is responsible for saving the result set for the assigned range into a blob. For simplicity, we the example does this sequentially, but for faster backup you would want to parallelize the below to do the regions and tables in parallel.

/// <summary>
/// Backup each table to blobs. Each table will be stored under a container with same name as the table./// </summary>
/// <param name=”tableClient”></param>
/// <param name=”blobClient”></param>
/// <param name=”tablesToBackup”>
/// The tablesToBackup will contain the table name to a list of keys that will be used to partition the query/// </param>public static void BackupTables(CloudTableClient tableClient,CloudBlobClient blobClient, List<TableKeysInfo> tablesToBackup)
{if (tableClient == null)
{throw new ArgumentNullException(“tableClient”);
}

try{// we will use this id as the folder name. The blobs will be stored under:
// <lower cased table name>/<backupid>/string backupId = DateTime.UtcNow.ToString(“yy-MM-dd-HH-mm-ss”);

// list each range in each table and backup up each rangeforeach (TableKeysInfo tableKeysInfo in tablesToBackup)
{CloudBlobContainer container = blobClient.GetContainerReference(tableKeysInfo.TableName.ToLower());
container.CreateIfNotExist();

The BackupTableRange builds a query that will scan the assigned key range and then invoke BackupToContainer as shown below. We use the BackupEntity class to read the result. The BackupEntity stores an internal XElement called EntryElement that stores the raw OData XML for the entity that is received in the query response. To get hold of this raw data, we use the ReadingEntity event on the context as shown in the code. The ResolveType delegate is used to provide the type name that WCF Data Service client should use (See this forum post for details).

The BackupToContainer creates a Block Blob to save the result set. Each block in the blob contains a collection of batches. Each batch contains a collection of entities that can be part of a single Entity Group Transaction i.e. batch command. This means all of the entities in the same batch must have the same PartitionKey value. The entity is stored as an entry element which is the raw format that the OData protocol uses to send the entity over the wire. The xml in a blob will look like the following with each Block portion being in a single block.

Since each block is a well formed xml, it allows us to read one block at a time and execute the group of transactions in them during the restore process.

To build this structure, we will need some classes to be defined that we will go over first.

State class – maintains a global state that is used while iterating through the entities in the query. It keeps track of the size that has been serialized into a memory stream. It also controls the entity iteration over the query.

Batch class – is a group of entities that can be part of a single batch transaction. It reads an entity from the query and as long as the number of entities is less than 100, the entities have the same partition key, and we do not exceed the size limit, we keep returning entities to batch up.

Block class – is a single block element that stores the group of batch commands. A block has a size limit to maintain. A block can be at most 4MB but we limit it to less than that because when the batch request is sent, more xml tags for the request are added as required by the OData protocol and there is a per entity overhead to consider.

Blob class – is a list of blocks that will be stored in a single blob. We set an arbitrary limit of 20 blocks per backup blob to allow easy parallelization of the backup restore at the blob level. This can easily be changed to a larger value, but must be less than 50,000 blocks, since that is the limit per Block Blob set by the storage system. The name of the blob will be <Backup Id>/<guid>_<Min range>_<Max range>and will be placed in a container which has the same name as the table but lower cased.

Backup Id – is the unique id formed from the timestamp when the backup started. The format used is: “yy-MM-dd-HH-mm-ss”

Min range – the min key used for the query. “null” if the min key is unbounded in the query range.

Max range – the max key used for the query. “null” if the max key is unbounded in the query range.

Given the above classes, serializing into a blob is simple. As long as there is an entity we have not processed in the iterator, we will group entities into batches and batches into blocks. Each block class will be written to an Azure Blob Block and we will reset the MemoryStream since the next entity will be written to a new batch that belongs to a new block. When we hit the limit for blocks in a blob, we will invoke PutBlockList with all the block ids written and create a new blob for future blocks. The BackupToBlock stores the data we have written into memory stream to a block only if we have seen at least one entity.

static void BackupToContainer(CloudBlobContainer containerToSave, CloudTableQuery<BackupEntity> query,string backupId, PartitionKeyRange range)
{// A block can be at most 4 MB in Azure Storage. Though we will be using much less
// we will allocate 4MB for the edge case where an entity may be 1MBMemoryStream stream = new MemoryStream(4 * 1024 * 1024);State state = new State(query, stream);

while (!state.HasCompleted)
{// Store the resultset to a blob in the container. We will use a naming scheme but the scheme does not
// have any conseuqences on the strategy itselfstring backupFileName = string.Format(“{0}/{1}_{2}_{3}.xml”,
backupId,Guid.NewGuid(),
range.Min == null ? “null” : range.Min.GetHashCode().ToString(),
range.Max == null ? “null” : range.Max.GetHashCode().ToString());CloudBlockBlob backupBlob = containerToSave.GetBlockBlobReference(backupFileName);

// if we have written > 0 entities, let us store to a block. Else we can reject this blockif (entityCount > 0)
{
backupBlob.PutBlock(block.BlockId, stream, null, requestOptions);return block.BlockId;
}

return null;
}

/// <summary>
/// The class that maintains the global state for the iteration/// </summary>internal class State{protected MemoryStream stream;IEnumerator<BackupEntity> queryIterator;

/// <summary>
/// This entity is the one we may have retrieved but it does not belong to the batch/// So we store it here so that it can be returned on the next iteration/// </summary>internal BackupEntity LookAheadEntity { private get; set; }

/// <summary>
/// Represents a collection of entities in a single batch/// </summary>internal class Batch{static int MaxEntityCount = 100;// Save at most 3.5MB in a batch so that we have enough room for
// the xml tags that WCF Data Services adds in the OData protocolstatic int MaxBatchSize = (int)(3.5 * 1024 * 1024);

State state;

internal Batch(State state)
{this.state = state;
}

/// <summary>
/// Yield entities until we hit a condition that should terminate a batch./// The conditions to terminate on are:/// 1. 100 entities in a batch/// 2. 3.5MB of data/// 2. 3.8MB of block size/// 3. We see a new partition key/// </summary>internal IEnumerable<BackupEntity> Entities
{get{BackupEntity entity;long currentSize = this.state.CurrentBlockSize;

/// <summary>
/// Represents all batches in a block/// </summary>internal class Block{// Though a block can be of 4MB we will stop before to allow bufferstatic int MaxBlockSize = (int)(3.8 * 1024 * 1024);

This allows us to store a single table into multiple blobs and allows restore to be parallelized across these blobs.

Table Restore

To restore tables from blobs, we can get the list of blobs in the container “table name” and then for each blob, we retrieve the list of blocks. For each block in the blob, we load the xml and retrieve the entry elements and create a BackupEntity instance for each entry we add to the context. Once we have added all entities in a batch element, we can call SaveChanges with Batch option to execute the transaction. The restore process assumes that it is a new table with no existing entities – so any “conflict” errors during the adding process is ignored.

The RestoreTo method gets the block list and for each blob, it then downloads the data using range gets for each block. It then retrieves the list of batch elements for each block, and for each batch element it invokes ExecuteBatch method.

// get all blocks and for each block read it separately as it is an xml doc by itselfIEnumerable<ListBlockItem> blocks = (IEnumerable<ListBlockItem>)blob.DownloadBlockList(BlockListingFilter.Committed, options);

The ExecuteBatch method retrieves all entry elements in the batch and creates a BackupEntity instance and adds it the context. It sets the WritingEntity event to control the xml sent over the wire. WritingEntity is called after WCF Data Services forms the xml element, but before it is serialized over the wire. We use the WritingEntity event to replace the property element of what WCF Data Services has written with the property element that we have retrieved from the entry element saved in the blob.

The following are some improvements that can be applied to the above code are:

Parallel processing of backup and restore for faster backup and restore.

For backups, we can parallelize the query execution over the different ranges as well as over each table in the storage account. When we process the ranges in parallel, it can be good to randomly select the query ranges to better spread out the load of the backup over your tables in production.

For restore we can parallelize by processing either different blobs or blocks in parallel and if we have multiple tables to restore, we can also parallelize across the different backup blob containers (representing different tables).

Rather than taking the list of keys as input from the user, we could remember the ranges we see from each run of the backup, and use them as input to guide the next backup to be performed. In doing this, each time we perform the backup, we would remember the key ranges we see, and use that information to form better backup ranges for the next backup to be performed in parallel. This can then be continuously improved based on how the dataset changes. In fact, with the above approach, one could list through the blobs from the prior backup and get the key ranges from the blob names. Then use this information to form the desired size of ranges when performing the next parallel backup.

You may want to store a version or other metadata in the xml and with the blob so that it can be used during restore.

You may also wish to store the blocks in a compressed format.

The code provided in this post is meant to be used as building blocks and not a full-fledged backup and restore tool. The idea was to go over some challenges and the options available to solve them. Please test the code if you intend to use it in your application.

Libraries that support backup

In this section we wanted to list known backup tools that we are aware of. We should point out that we have not verified the functionality claimed by these utilities and their listing does not imply an endorsement by Microsoft. Since these applications have not been verified, it is possible that they could exhibit undesirable behavior.

Also, please email us (using the link on the left of this site) if you have any ideas on the above code or if we have missed any library that provides data backup for storage accounts and we will add them to this list.

Thanks for this article. I am fairly new to Azure, and there is something I would like to know. Could you give tips on how to integrate this backup solution (and also the earler solution for blobs) with an existing Asp.net/Silverlight application that is running within Azure? For instance, what role should the backup code run under, and what is the recommended way to trigger a backup within the application? We need to automate this as much as possible.

@Mikel, I would run this in a worker role that gets the schedule for backup from a Azure Table. You can now modify the "schedule" row to change the frequency/time of backup. The table itself can be modified either via a ASP.NET/Silverlight app or using some app which allows you to modify rows in Azure Table. The worker role can get this schedule on startup and from there on, either poll every hour for a change or just sleep until it is time to perform the backup job (if applying the change to schedule is not very important). If you update the schedule via an app, you could trigger the change by updating the config for the worker role using service management APIs.