If I don’t start blogging some of the stuff I’m doing, I’ll never remember how I did it in the first place! So, time to try to come out from the darkness and see if I can keep discipline to post more.

Problem: One of our servers has a Linux md RAID1 device (md1), which would keep falling out of sync, especially after a reboot. Upon nearing the end of they re-sync, it would pause, fail, and start over. Entries from /var/log/messages

Then, the RAID1 re-sync would start all over again … only to fail a few hours later … and again.

Since there appear to be some bad spots on sda at the end of the disk, the solution seemed to be to shrink the partition a bit to avoid the bad spots.

Our md RAID1 (md1) consists of 2 x 694GB partitions on sda3 and sdb3. On top of md1 lives an LVM PV, VG, and lots of LVs. The PV is not 100% allocated, so we have some wiggle room to shrink everything.

Here’s the procedure we followed:

Shrink the LVM PV

“pvdisplay” showed the physical disk had 693.98 GB, so we just trimmed it down to an even 693GB.

Hey, what a coincidence … the Array Size is the same as the PV was! 🙂 Now, we need to calculate the new size. The new size of the PV is 1453325952 sectors, which is 726662976 blocks (even *you* can divide by 2!!).

Now, technically, we should shrink the partition, too. Here’s where I ran into a bit of trouble (ok, I’m too lazy to do the math). “fdisk” on linux doesn’t seem to want to let you specify a size in blocks or sectors, so you have to keep shrinking the ending cylinder number until you get in the range of the new block size. I’ll leave this as an exercise for the reader, and feel free to post a comment with the actual procedure. 🙂

After these steps were completed, we were able to perform the raid1 re-sync successfully.