Oracle Blog

inside the sausage factory

The need for triple-parity RAID

When I first wrote about triple-parity RAID in ZFS and the Sun Storage 7000 series, I alluded a looming requirement for triple-parity RAID due to a growing disparity between disk capacity and throughput. I've written an article in ACM Queue examining this phenomenon in detail, and making the case for triple-parity RAID. Dominic Kay helped me sift through hard drive data for the past ten years to build a model for how long it takes to fully populate a drive. I've reproduced a graph here from the paper than displays the timing data for a few common drive types — the trends are obviously quite clear.

The time to populate a drive is directly relevant for RAID rebuild. As disks in RAID systems take longer to reconstruct, the reliability of the total system decreases due to increased periods running in a degraded state. Today that can be four hours or longer; that could easily grow to days or weeks. RAID-6 grew out of a need for a system more reliable than what RAID-5 could offer. We are approaching a time when RAID-6 is no more reliable than RAID-5 once was. At that point, we will again need to refresh the reliability of RAID, and RAID-7, triple-parity RAID, will become the new standard.

Triple-Parity RAID and BeyondADAM LEVENTHAL, SUN MICROSYSTEMS
As hard-drive capacities continue to outpace their throughput, the time has come for a new level of RAID.
How much longer will current RAID techniques persevere? The RAID levels were codified in the late 1980s; double-parity RAID, known as RAID-6, is the current standard for high-availability, space-efficient storage. The incredible growth of hard-drive capacities, however, could impose serious limitations on the reliability even of RAID-6 systems. Recent trends in hard drives show that triple-parity RAID must soon become pervasive. In 2005, Scientific American reported on Kryder's law, which predicts that hard-drive density will double annually. While the rate of doubling has not quite maintained that pace, it has been close.

Problematically for RAID, hard-disk throughput has failed to match that exponential rate of growth. Today repairing a high-density disk drive in a RAID group can easily take more than four hours, and the problem is getting significantly more pronounced as hard-drive capacities continue to outpace their throughput. As the time required for rebuilding a disk increases, so does the likelihood of data loss. The ability of hard-drive vendors to maintain reliability while pushing to higher capacities has already been called into question in this magazine. Perhaps even more ominously, in a few years, reconstruction will take so long as to effectively strip away a level of redundancy. What follows is an examination of RAID, the rate of capacity growth in the hard-drive industry, and the need for triple-parity RAID as a response to diminishing reliability.

Just a note to warn away from the term "RAID-7", not sure what its current status is today, but it was certainly trademarked as a proprietary term, originally by SCC, to describe a particular implementation of a hardware based RAID operation they were pushing … so RAIDzN or similar I guess.

@Chicago I wonder too, but with current trends, my guess is that it will be common in the next 5-10 years. We'll see if hard drive manufacturers change direction.

@Craig Fortunately, the RAID-7 trademark was canceled September 11, 2004. RAID-8 and RAID-9 were also trademarked at one point, but have since been canceled or abandoned along with a bunch of other trivial RAID derivatives; only a couple of dubious trademarks remain. In any case... not really the point.

Mostly I shudder at the idea of ever wider RAIDs with ever greater alignment requirements and fragility with wider parity. i have seen morons "designing" 23+1 RAIDs and on Thumpers other morons doing 46+2.

Parity RAID as a RAID developer told me mostly only make sense as "salesman's RAID", to help persuade suckers that they can have it all: lots of storage and very little capacity wasted on redundancy. I have been persuaded by a smart guy from BNL that there is one case where parity RAID makes sense (essentially RO caches containing data bulk archived on slower mass storage like tape) though.

The obvious alternatives are RAID10, which scales up pretty well, and full redundant R-S chunking, which is used in distributed and highly redundant systems where

When technology serves you certain skewed performance envelopes, there is no way to use fancy snake oil like RAID with N+3 stripes to work around that; disk drive trends mean they are becoming mostly archive storage, and fortunately we have SSDs now for the cases where storage with the traditional relative performance envelope of disk drives is still required.

BTW, in repopulating a drive there is a subtle problem: the capacity of current drives is at the limit or beyond that of the undetected error rate of the checksumming systems used.

That is, duplicating a drive has a significant chance of incurring one or more undetected copying errors. Fortunately ZFS and other systems have their own layer of checksumming, but this adds to the other problems about "silent corruption" described for example here:

@Blissex The arguments you cited should hardly be revelations to anyone. As with many things, you have to make a choice with storage: cheap, fast, or reliable, pick two. For example, mirroring will have higher IOPS than parity RAID schemes, but cost more for each unit capacity. People who elect to have very wide stripes are hardly morons -- they've merely evaluated the tradeoff between storage efficiency (cost), and performance. Further, if your problem is throughput rather than IOPS, parity RAID-Z schemes are actually better than mirroring with the same total number of disks.

In addition, mirroring doesn't provide the reliability it once did (read the paper) which is why we see customers deploying three-way mirroring for critical data in, say, OLTP environments.

In short, there's no grand RAID conspiracy. In fact, with the Sun storage 7000 we make very clear the choices and tradeoffs between various storage configurations. Triple-parity RAID is far from marketing hokum -- it addresses a real problem and fills a niche distinct from that of mirroring.