The entire system has hung 3 times in the past 2 weeks, requiring a hard reset. For the time being, I'm going to assume the system hang is unrelated to my md issue, although I can't completely discount that possibility. Each time we've rebooted, md7 has required a rebuild, but I can't figure out how to tell from the logs which disk triggered the rebuild. I thought iostat might be able to help me while the RAID was still rebuilding:

...but it looks to me like md7 is using sdd to rebuild all the other disks in that RAID. I thought maybe this was simply because sdd is an SSD and all the other disks are marked write-mostly, but in that case, it should only rebuild the one disk that was out of sync (unless all the spinning disks just happened to be out of sync, which seems unlikely to me).

Another theory that I have is that all the spinning disks are always out of sync upon reboot simply because the SSD's writes are so fast that it has time to finish writing a block while the others are still writing, then the system just happens to lock up before the other disks finish writing that block?

So, how do I tell which disk(s) triggered the resync? Is the fact that I have an n-way mirror with mixed SSD and spinning disks possibly responsible for the fact that all the spinning disks are always rebuilt after one of these freezes, or does the md driver guarantee that a block isn't considered written on one disk until it's successfully written on all disks?

This happened to me last month, smartd or /var/log/messages should show something, no?
–
luckytaxiNov 15 '12 at 17:55

The machine froze up in all 3 instances and we had to do a hard reset, so there is no information logged. /var/log/messages shows that md7 was resynced upon the subsequent boot, but as far as I can tell, there is no indication which disk(s) needed to be resynced.
–
robNov 15 '12 at 20:11

2 Answers
2

As Michael points out above, the hangs and consequent unclean shutdown are the reason you are seeing your RAID rebuild. The kernel md driver rebuilds unclean arrays in order to ensure they are truly in sync, since a hang, or crash or powerloss won't guarantee which writes actually got flushed out to disk.

Now, as to why sdd is getting used, the first thing to understand is that in an unclean shutdown, the actual array, as opposed to an individual member device, is marked dirty. In the manpage I linked above, the following is said about RAID-1:

If the md driver finds an array to be dirty at startup, it proceeds to correct any possibly inconsistency. For RAID1, this involves copying the contents of the first drive onto all other drives.

In your example, the md7 array has partitions on drives sdc, sdd, sde & sdf, but if you look at your mdstat output:

md7 : active raid1 sdd1[0] sde53 sdf54 sdc11

note how the first partition, marked with a [0], is on sdd, namely, sdd1. That's the reason sdd is being used -- it's the first drive in md7.

I understand that (at least linux) raid works something like a filesystem for these purposes - if the system crashes while it's in use, it will need to be checked on reboot. So the cause of your system's crashes may not be any disks in the array.

I already specified in the question that I am assuming for now that the system hang is unrelated (because a properly-behaving system should kick the bad disk off and e-mail me a DegradedArray event). What I want to do is find out (by querying a device or looking at a log) which devices are being resynced, and why. I can infer from iostat that all three spinning disks are being resynced, but I don't know how to confirm this.
–
robNov 15 '12 at 20:23