RAID 10 with mdadm

If I had to pick one fault of Linux, it would be that for almost everything, the Linux user is inundated with hundreds of possible solutions. This is both a blessing and a curse – for the veterans, it means that we can pick the tool that most matches how we prefer to operate; for the uninitiated, it means that we’re so overwhelmed with options it’s hard to know where to begin.

One exception is software RAID – there’s really only one option: mdadm. I can already hear the LVM advocates screaming at me; no, I don’t have any problem with LVM, and in fact I do use it as well – I just see it as filling a different role than mdadm. I won’t go into the nuances here – just trust me when I say that I use and love both.

There are quite a few how-tos, walkthroughs, and tutorials out there on using mdadm. None that I found, however, came quite near enough to what I was trying to do on my newest computer system. And even when I did get it figured out, the how-tos I read failed to mention what turned out to be a very critical piece of information, the lack of which almost lead to me destroying my newly-created array.

So without further ado, I will walk you through how I created a storage partition on a RAID 10 array using 4 hard drives (my system boots off of a single, smaller hard drive).

The first thing you want to do is make sure you have a plan of attack: What drives/partitions are you going to use? What RAID level? Where is the finished product going to be mounted?

One method that I’ve seen used frequently is to create a single array that’s used for everything, including the system. There’s nothing wrong with that approach, but here’s why I decided on having a separate physical drive for my system to boot from: simplicity. If you want to use a software RAID array for your boot partition as well, there are plenty of resources telling you how you’ll need to install your system and configure your boot loader.

For my setup, I chose a lone 80 GB drive to house my system. For my array, I selected four 750 GB drives. All 5 are SATA. After I installed Ubuntu 9.04 on my 80 GB drive and booted into it, it was time to plan my RAID array.

As you can probably tell, my system is installed on sde. While I would have been happier with it being labeled sda, it doesn’t really matter. sda through sdd then are the drives that I want to combine into a RAID.

mdadm operates on partitions, not raw devices, so the next step is to create partitions on my drives. Since I want to use each entire drive, I’ll just create a single partition on each one. Using fdisk, I choose the fd (Linux raid auto) partition type and create partitions using the entire disk on each one. When I’m done, each drive looks like this:

Now that my partitions are in place, it’s time to pull out mdadm. I won’t re-hash everything that’s in the man pages here, and instead just demonstrate what I did. I’ve already established that I want a RAID 10 array, and setting that up with mdadm is quite simple:

A word of caution: mdadm --create will return immediately, and for all intents and purposes will look like it’s done and ready. It’s not – it takes time for the array to be synchronized. It’s probably usable before then, but to be on the safe side wait until it’s done. My array took about 3 hours (give or take – I was neither watching it closely nor holding a stopwatch!). Wait until your /proc/mdstat looks something like this:

Everything I read told me that Debian-based systems (of which Ubuntu is, of course, one) were set up to automatically detect and activate your mdadm-create arrays on boot, and that you don’t need to do anything beyond what I’ve already described. Now, maybe I did something wrong (and if so, please leave a comment correcting me!), but this wasn’t the case for me, leaving me without an assembled array (while somehow making sdb busy so I couldn’t manually assemble the array except in a degraded state!) after a reboot. So I had to edit my /etc/mdadm/mdadm.conf file like so:

It certainly looks like my array should have been detected and started when I rebooted. I commented-out the default DEVICES line and added an explicit one, then added an explicit declaration for my array; now it’s properly assembled when my system reboots, which means the fstab entry doesn’t provoke a boot-stopping error anymore, and life is all-around happy!

Update 9 April 2011: In preparation for a server rebuild, I’ve been experimenting with mdadm quite a bit more, and I’ve found a better solution to adding the necessary entries to the mdadm.conf file. Actually, two new solutions:

Configure your RAID array during the Ubuntu installation. Your mdadm.conf file will be properly updated with no further action necessary on your part, and you can even have those nice handy fstab entries to boot!

Run the command mdadm --examine --scan --config=mdadm.conf >> /etc/mdadm/mdadm.conf in your terminal. Then, open up mdadm.conf in your favorite editor to put the added line(s) into a more reasonable location.

On my new server, I’ll be following solution (1), but on my existing system described in this post, I have taken solution (2); my entire file now looks like this:

Notice that I’m again using the default DEVICE line, and notice the new ARRAY line that’s been added. This seems to work a lot better — since making this change, I no longer experience the occasional (and strange) “device is busy” errors during boot (always complaining about /dev/sdb for some reason), making the boot-up process just that much smoother!