I am beginning to think that I have a hardware problem, otherwise I can't imagine why the mdadm restore should be so slow.

I have done a benchmark on /dev/sdc using Ubuntu's disk utility GUI app, and the results looked normal so I know that sdc has the capability to write faster than this. I also had the same problem on a similar WD drive that I RMAd because of bad sectors. I suppose it's possible they sent me a replacement with bad sectors too, although there are no SMART values showing them yet.

Any ideas? Thanks.

As requested, output of top sorted by cpu usage (notice there is ~0 cpu usage). iowait is also zero which seems strange:

dmesg printing out hardware errors? You have a high load average for no IOWait and no CPU time.
–
StrangeWillOct 20 '11 at 20:37

2

Tonight I will try unplugging other components to see if maybe the drive isn't getting enough power.
–
chrishiestandOct 20 '11 at 20:49

2

I think I figured out the problem. When I opened the case I found that the SATA power cable for two drives was running between fins of the video card's heat sink. This also explains why the problem didn't show up at first. When the heat sink heated up, resistance in the power cable increased, decreasing voltage. Strangely enough the system disk (also receiving power from said cable) didn't seem to have any problems. I re-routed the SATA power cable and now I'm getting a more reasonable albeit fluctuating rate between 5-20MB/s. No more dmesg errors so far.
–
chrishiestandOct 21 '11 at 7:54

1

@chrishiestand please add you solution as an answer to the question, and then accept it as your answer after 48 hours :)
–
pauskaOct 5 '12 at 10:15

A kernel bug is another good suggestion. I have considered upgrading to 11.10 to see if that fixes it. If I come up with nothing else I will try it. I had previously had a hardware raid controller that worked fine on this system. The problems began when I upgraded to 2TB drives so I ditched the old hardware raid for software raid. Under hardware raid, one of my drives would also fall out of the raid array and I couldn't figure out why. But then I had similar problems on software raid.
–
chrishiestandOct 20 '11 at 21:18

I've also had problems with partition block devices not being created by the kernel. For example, sometimes /dev/sdc1 would not be created at boot. I just assumed I had made a mistake partitioning, but perhaps it's because of this underlying problem.
–
chrishiestandOct 20 '11 at 21:22

The problem ended up being caused by hardware. Eventually I unscrewed all the HDDs from their chassis and moved them to new locations in the tower- and I tried unplugging peripherals that I knew I would not be needing. This fixed the problem. The raid array quickly rebuilt and my raid array has been stable for months.

Unfortunately I do not know what specifically caused the problem. I am guessing that something put noise into a data link, but I really have no clue. My motherboard is an Asus P5WDG2 WS Pro, in case anyone else has a similar problem with this motherboard.

Thanks to everyone who tried to help - this ended up being an edge case.