My RAIDZ Storage Pool is Degraded

And no, this isn’t a comment on my server’s self-esteem. My RAIDZ pool is reporting that it’s got problems. It figures it happens on Friday the 13th. Weird, huh?

Actually I first saw the warnings a couple weeks ago when a scrub returned several errors in the zpool. The data was still good though because of checksumming in ZFS. I was getting a little anxious because I keep my iTunes library on the server along with raw footage from home movies. Important stuff to our family that I wouldn’t want to tell my wife that I had lost.

Most of the errors were on C7D0 (one of the four SATA drives in my setup) so I spent a week or two debating about whether I wanted to just upgrade the four 250GB drives to 1TB or 1.5TB drives. In the end, I decided on hoarding cash and I ordered a 250GB drive for $40. I’d still like to upgrade the array to 1.5TB drives when they hit $99 each, but I can wait to see if this new stimulus package will include free drives for homeowners along with the kickbacks to Wall St executives and help for stupid people who bought homes they can’t afford. I can only dream, right?

When the new drive arrived, I thought I would see if I could hot swap it. Smart, right? In hindsight it makes perfect sense that I would try testing out the resilency of ZFS and RAIDZ on a live array holding precious memories of my children’s births and early years when it was already starting to fail. Living life on the edge is what I like to do. Of course, when you try this on your own home file server, you might want to be a little more careful about tracing the SATA cables to make sure you actually pull the right one.

Panic. This was going to be a Friday the 13th for the ages.

There were a couple minutes there when I was literally sweating bullets. I was lucky though that the array wasn’t being used and so no data was lost. I brought c8d0 back up and cleared the zpool and all looked good.

Once I get everything up and running, I’ll write up the steps I used to make a backup, recreate the arrays, and copy my data back.