Backup to the Future

This is the 30th article in the Spotlight on IT series. If you'd be interested in writing an article on the subject of backup, security, storage, virtualization or MSPs for the series, PM Kathryn to get started.

I’ve spent the last decade evangelizing the transition of backup from tape to disk. Early on, I was thrilled when free food would entice 15 people to suffer through my “ATA disk transforms backup” presentation. Thus, it’s a privilege to be the first vendor on this much larger forum (when you’re in Silicon Valley, stop by EMC for free food).

Step 1 – Don’t Take a Step, Yet
People ask, “What is the right choice for protecting virtual environments?” It depends (it’s an engineer’s default answer; it almost slipped out at my wedding, instead of “I do”). There’s no single right answer. But most of the wrong answers stem from one problem – focusing solely for the short term. Therefore, I’m not going to expound on optimizing guest or VCB backups. First, I’ve endured more VM backup “best practices” documents than ads for “unusual ways to reduce your belly fat.” Second, regardless of the implementation, traditional methods always hit a wall – either CPU or I/O.

Without a three year direction, you will become another cautionary tale. Any short-term solution should move toward the long-term goal. Since deduplication will be core to both the short and long-term strategy, focus on picking a flexible, scalable approach. It’s hard to hide large, bad capital purchases; your data center needs only so many doorstops.

Step 2 – The Future Starts Today
If traditional backup approaches will fail in your virtual environment within three years, what’s the future? Versioned replication.

What is a versioned replica? It is a mirrored copy that retains multiple independent versions of the data. It solves the virtualization protection challenges (i.e. CPU and I/O bottleneck) by backing up only new data. It solves traditional incremental backup challenges (i.e., “We have to run a series of 93 incremental restores? I’ll go pack up my cube.”) because replicas are always full backups. You solve the replica challenges (i.e., only one version of the data), by retaining multiple point-in-time versions with space efficient technologies (e.g., dedupe, snapshots).

Step 3 – OK, the Future Started a While Ago
Recently, as I pontificated about a glorious future of versioned replication, the customer groused, “The future ain’t what it used to be.” As a versioned replication user since 2003, he had reason to gripe.

His storage-based version replication (e.g., NetApp SnapVault, EMC RecoverPoint) is fast, but complex. He deploys it because backups complete within minutes, not days. Unfortunately, he pays a large operational cost for protecting storage containers (e.g., LUNs, volumes) instead of VMs. Each storage vMotion migrates the VM’s active data, while the backup copies stay behind in the original storage container. For each backup, he manually catalogs the storage container for each VM. Each restore is a “heart attack waiting to happen,” as he hopes that his hand-crafted database identifies the appropriate storage container for that date’s backup. “Why am I now building my own backup application?!” High-stress grunt work.

On the other hand, his backup-application versioned replication (e.g., EMC Avamar, Symantec PureDisk) is flexible, but not fast enough. Yes, it’s much faster to back up the modified 10TB, instead of the whole 1PB. Unfortunately, identifying changed data can still require hours of CPU and I/O intensive activity. Whereas storage-based solutions leverage optimized internal mechanisms to identify changed data, backup solutions rely on brute force. Therefore, he still spends a lot of time troubleshooting and tuning backup schedules. High-stress grunt work.
Fortunately, a new VMware invention changed his outlook on the future.

Step 4 – VMware Changed Block Tracking – The Missing Link to the Future
One of my favorite customers lamented, “Why can’t I have instant backups and reliable recoveries, with minimal effort?” There have always been trade-offs. The prior customer loved the performance of storage-based versioned replication, but hated the complexity. He loved the simplicity of backup-application versioned replication, but needed better performance
VMware’s Changed Block Tracking (CBT) is the final piece to the puzzle. VMware tracks the changes as they occur, so it can instantly tell the backup application (e.g., EMC Avamar, Symantec NetBackup) what to protect. By eliminating the brute force search for changes, backup-application versioned replication now delivers both performance and flexibility. My happiest customers have adopted this approach. They delete their enormous backup schedule spreadsheets and scrap their homegrown backup applications. High-stress grunt work – eliminated.

As one said, “My backups just work. Even better, so do my restores.”

Don’t Predict the Future, Build It
Almost all of my customers have deployed some type of versioned replication, and they’re testing CBT-based versioned replication as the next step toward “protection that just works.”

Their advice on getting started? Set your direction. Then lay the groundwork:

Deduplicated backup storage is the best starting point. Versioned replication depends on it; traditional backups benefit from it. Pick the right anchor for your solution.

VMware CBT depends on VMware configuration options. Enable it in your templates now, so it’s there when you’re ready.

Virtualization uniquely stresses the performance, scalability, and management of backup architectures. Technical and market trends lead to versioned replication. Consider your direction before making a choice that you (or your successors) regret. Over the last five years, deduplicated disk swept through the industry. Over the next five years, versioned replication will leverage the deduplication infrastructure to revolutionize data protection. Virtualization will be the catalyst.

Thanks for reading (the free food bribe still stands). Now it’s your turn. How do you protect your VMs today and what’s your plan? Do you use versioned replication? What’s your next step?

We haven't hit any snags with our regular backups via Backup Exec... yet. I expect that as we add more and more VMs and retire more physicals here that may well become an issue. Maybe by then the "tax" to get onboard will be more affordable, too. Nice article - we used Avamar when I was with IBM and it worked pretty well, provided the backup team staff knew what they were doing :-)

I was especially interested in the Future Starts Today - Versioned replication

like .vcc files with attitude :o) although I have never had real responsibility for data backup / recovery, it has always been a subject I have had an interest in, mainly because, even when this all kicked of big time [which in my eyes is NT4 onwards - as systems grew, and spread so that anyone with even a mid range budget could implement a major data store, and to me th ereal issue / problem building , was always where do we put it all, and how do we get it back

We found that many of our customers had been struggling with backup and restore times for quite a while. Their hopes were that going to Vmware would get the time to backup, and more importantly, to restore in to a workable window. Unfortunately this has not been the case. In fact, overall network throughput from the ESX layer has gone down. Our customers address this by automatically adjusting the network settings withing the VMware to adapt to changing conditions and take advantage of the bandwidth available. We have been able to increase throughput by up to 5000% with our app.

Thank you for getting this discussion going and letting everyone know they are not alone in their struggles.