Global Dedupe Boosts Backup Speed Many Times Over

April 11, 2014Chandar Venkataraman

Cloud backup customers often complain about the speed of data transfer or more accurately the lack thereof. Part of the reason is a good and necessary function: backup software must compare current files with the most recent backup version in order to determine what has changed. The backup software then backs up only the changed data to the cloud.

The problem is that an average—and enormous—80% of data are actually copies. On the corporate network many copies are produced by inefficient backup and replication processes. But in the mobile world the issue is the proliferation of devices. With multiple mobile devices per user, and frequent file sharing between users, huge volumes of duplicate files clutter remote data storage. Older backup systems read newly copied files as unique files. Backup and restore transport speeds slow to a crawl.

Global Dedupe is a Necessity

A fundamental answer to the problem is global deduplication. Device-level dedupe will only dedupe files on a single device, which does nothing for the many copies existing on multiple devices. In contrast, global dedupe recognizes file copies located on different devices, saves a single unique copy to the backup server, and only performs change detection on that single instance.

If the global dedupe knows to check common file types for tell-tale duplication patterns, so much the better. Client-side deduplication and caching also save internal bandwidth by never moving copies to the server at all. The result of all of this efficient global dedupe is dramatically improved data transport speeds.

Speaking of vastly improved performance, multi-threading is another crucial feature for accelerating both backups and restores. Multi-threaded parallel transfers enable multiple files to travel simultaneously across the pipes, efficiently using bandwidth for much greater performance. Restore operations, which can be considerably slower than backup, strongly benefit from multi-threaded transfers.