November 10, 2010

The Next Storage System

Imagine my horror. My Iomega NAS, the box that had been sitting purring under my desk since February just had a catastrophic failure. 1.5TB of my stuff gone. Pfft. I should have known better. Every year I have a disk fail. Every year without failure. So I said this time, I'm going to go RAID. I should have gone mirror. And from now on, I will go mirror. Redundant mirror with redundant cloud backup. I say never again.

Over the past 8 months, I had been gradually moving data from a small array of external USB drives to what is now a single point of failure, my Iomega StorCenter ix4-200d. It's a four drive NAS with each drive clocking in about 900GB. I thought I was being smart by using the RAID, thinking that if something went wrong it would be simple to recover. It turns out to be simple only if you can afford 4,000 bucks. Yeah thousand. It is literally a box of rocks. I was foolish to think JBOD was inferior.

I'm so through with Iomega. I thought I was through with Zip and then they got me with Jaz. I thought I was through with Jaz and this year they got me again. You want to know the most reliable drives I have. A couple of LaCie external USBs by Porsche Design. They have never failed me once in, what 6 years? I don't trust them either. So until I can get my HDFS working, I'm going to hack my own together. Here's the plan.

All of my old failed hard drives had names. And I've been able to recover most of the stuff from them. That's how I had things to move over to the NAS in the first place. I'm going to take the risk that the Seagate portable USB drives, the Free Agent Go, is going to be my best bet for this interim period. So for the approximately 2TB that I need for all my stuff, I'm going to get a couple more of those, in addition to the three I have, plus an older 1TB Seagate I know to be flaky and the pair of LaCies, and implement what I'm calling the Steelhead Project.

It's going to work like this. I'm going to block and tackle sets of data into named high level folders of several naming conventions. No high level folder will have more than 20GB of unzipped data. These will be distributed redundantly over multiple drives. The Iomega will be reused but as a backup, the working pieces will be two 1TB drives. The auxiliaries will be the Free Agents, and generally small drives will be on the boxes themselves.

In addition to re-igniting my Backblaze account, which I had running up through December of last year, I will periodically (and manually) zip up selected portions of the high level folders and upload them to Google Docs. If they are singly larger than 1GB then I'll partition them via 7zip.

On the whole, I had about 450GB backed up to Backblaze. Their recovery system has worked out nicely, and it's definitely worth 50 bucks a year, even though my last backup was almost a year old. The stuff that really matters actually shouldn't be on spinning disks anyway. Which reminds me...I will also, using the high level folders, zip stuff back to DVD.

The toughest part of this is partitioning the data into the high level sets. That's where some programming will come in. I want the working sets to be human navigable, but the backup sets to be in smaller chunks. Then I can use Advanced File Organizer to tell me quickly what are in the volumes.

Comments

The Next Storage System

Imagine my horror. My Iomega NAS, the box that had been sitting purring under my desk since February just had a catastrophic failure. 1.5TB of my stuff gone. Pfft. I should have known better. Every year I have a disk fail. Every year without failure. So I said this time, I'm going to go RAID. I should have gone mirror. And from now on, I will go mirror. Redundant mirror with redundant cloud backup. I say never again.

Over the past 8 months, I had been gradually moving data from a small array of external USB drives to what is now a single point of failure, my Iomega StorCenter ix4-200d. It's a four drive NAS with each drive clocking in about 900GB. I thought I was being smart by using the RAID, thinking that if something went wrong it would be simple to recover. It turns out to be simple only if you can afford 4,000 bucks. Yeah thousand. It is literally a box of rocks. I was foolish to think JBOD was inferior.

I'm so through with Iomega. I thought I was through with Zip and then they got me with Jaz. I thought I was through with Jaz and this year they got me again. You want to know the most reliable drives I have. A couple of LaCie external USBs by Porsche Design. They have never failed me once in, what 6 years? I don't trust them either. So until I can get my HDFS working, I'm going to hack my own together. Here's the plan.

All of my old failed hard drives had names. And I've been able to recover most of the stuff from them. That's how I had things to move over to the NAS in the first place. I'm going to take the risk that the Seagate portable USB drives, the Free Agent Go, is going to be my best bet for this interim period. So for the approximately 2TB that I need for all my stuff, I'm going to get a couple more of those, in addition to the three I have, plus an older 1TB Seagate I know to be flaky and the pair of LaCies, and implement what I'm calling the Steelhead Project.

It's going to work like this. I'm going to block and tackle sets of data into named high level folders of several naming conventions. No high level folder will have more than 20GB of unzipped data. These will be distributed redundantly over multiple drives. The Iomega will be reused but as a backup, the working pieces will be two 1TB drives. The auxiliaries will be the Free Agents, and generally small drives will be on the boxes themselves.

In addition to re-igniting my Backblaze account, which I had running up through December of last year, I will periodically (and manually) zip up selected portions of the high level folders and upload them to Google Docs. If they are singly larger than 1GB then I'll partition them via 7zip.

On the whole, I had about 450GB backed up to Backblaze. Their recovery system has worked out nicely, and it's definitely worth 50 bucks a year, even though my last backup was almost a year old. The stuff that really matters actually shouldn't be on spinning disks anyway. Which reminds me...I will also, using the high level folders, zip stuff back to DVD.

The toughest part of this is partitioning the data into the high level sets. That's where some programming will come in. I want the working sets to be human navigable, but the backup sets to be in smaller chunks. Then I can use Advanced File Organizer to tell me quickly what are in the volumes.