NAS vs. Object: Beating Bit Rot – Examining Data Durability

IT considers data durable if it survives anything that might happen to it because of how the user stores it. Typical data protection systems (e.g. backup, snapshots, replication) protect data from component, media, site, and human failure – or at least they’re supposed to. Object storage, whether from a cloud provider or onsite in the data center, can provide many of those features as well. My colleague Joseph Ortiz will compare the data protection techniques of NAS vs. Object storage in a future post.

What about Bit Rot?

What typical data protection systems do not protect against is bit rot. Bit rot is a layman term applied to the concept of magnetic degradation over time. A secret (that hasn’t been kept very well) about magnetic storage is that it all degrades over time. Ones become zeros and vice versa. With any magnetic storage, it is not a matter of if the data will be corrupted – it is when.

The degree to which a medium will allow bits to be coerced over time into changing polarity is called its coercivity value, and the formula to determine coercivity is KuV/kT, where V is the volume of the magnetic bit and K is the ambient temperature of that bit in Kelvin. Unfortunately for those wishing to store data for long periods of time, disk has poor values for both. (It has very small magnetic bits and disks are hot all the time.) When those values are plugged into the coercivity formula, the values that emerge suggest that data left on disk will begin to significantly degrade after five years. This means that if you are using disk as your long term storage medium, you must monitor for and repair magnetic degradation. This must be done at a level higher than the device level, because the underlying hardware layer will not detect this magnetic degradation.

Watch On Demand

Object storage software is perfect for monitoring and repairing the effects of magnetic degradation, because such products use a unique identifier for each object that is the result of some type of algorithm (e.g. SHA-256) being run against the contents of the object. By re-running the algorithm against an object and comparing it against its unique identifier, the software makes sure that no bit rot corrupts the file. Caringo’s Swarm does this using proactive period health checks, which allows them to identify and correct any bit rot even before an object is requested. Other vendors check for and repair corruption when they read the file.

Another way the software ensures durability is to make sure the object can survive simultaneous node or site failures. This is done by replicating the object to multiple locations. An object that does all of these things would be considered durable even over long periods of time.

StorageSwiss Take

A storage product that is truly durable will be able to overcome any kind of damage that could happen to a file, which goes beyond a component or site failure. It includes protection against bit rot over time – which requires some higher level of intelligence which is provided by object storage vendors. It can also provide protection against accidental or malicious damage to the data as well. When evaluating storage vendors, make sure to look into both sides of the equation.

Sponsored by Caringo

About Caringo

Caringo was founded in 2005 to change the economics of storage by designing software from the ground up to solve the issues associated with data protection, management, organization and search at massive scale. Caringo’s flagship product, Swarm, eliminates the need to migrate data into disparate solutions for long-term preservation, delivery and analysis—radically reducing total cost of ownership. Today, Caringo software is the foundation for simple, bulletproof, limitless storage solutions for the Department of Defense, the Brazilian Federal Court System, City of Austin, Telefónica, British Telecom, Ask.com, Johns Hopkins University and hundreds more worldwide. Visit www.caringo.com to learn more.

Share this:

Like this:

Related

W. Curtis Preston (aka Mr. Backup) is an expert in backup & recovery systems; a space he has been working in since 1993. He has written three books on the subject, Backup & Recovery, Using SANs and NAS, and Unix Backup & Recovery. Mr. Preston is a writer and has spoken at hundreds of seminars and conferences around the world. Preston’s mission is to arm today’s IT managers with truly unbiased information about today’s storage industry and its products.

Well, it seems if you are planning to store data online for a long period of time, then object storage will provide the highest level of durability with its support for object replication or erasure coding and the ability to perform periodic “health checks” or “repair on read” procedures. We have decades of experience using rotating magnetic disks. We do not have decades of experience with solid state disks. Initially, people were warned that SSDs would “wear out” pretty quickly and need to be replaced. Now people don’t talk so much about that possibility. So, what should we be worried about when it comes to the durability of data stored on SSDs? It seems reasonable to assume that SSDs will be deployed for object storage sooner rather than later. Will this improve data durability or not? Are SSDs subject to the same type of “bit rot” as HDDs?

Curtis, thanks for the link to what George had to say about flash durability. Now that SSDs have hit 15TB capacity, it is not unreasonable to think about using SSDs for warm data in an object store to reduce latency, and tiering that warm data to HDDs in the object store when the data becomes cold. Cold data could eventually be deleted or migrated to an archive tier for long term storage on large HDDs using SMR/HAMR or tape or optical if it needs to be retained for compliance or governance reasons. The future of data storage looks like flash and object, and it seems likely that some flash will soon be deployed in object storage clusters for performance, space and energy-conservation considerations. I would not be terribly surprised to see the use HDDs terminally decline by 2025.