Random musings on life, death and technology

Menu

Disasterous server upgrade

You’re probably aware that I run a small server at home which acts as my print, file, media, backup and mail server, as well as providing a series of websites and external services out onto the internet. It also manages my internet connection, and shares it (safely) amongst all the other computers in my home. I built it back in mid 2007 or so, when I was first switching from Windows to Linux, and chose to install Ubuntu Server 7.04 on it.

When the next Long Term Supported release came out (8.04) I eventually upgraded to it (via 7.10) and broke support for my PCI SATA card, which played havoc with my RAID array. I eventually worked around that by changing my RAID array to not use all the hard drives, and continued to run 8.04 until about 9 months ago, when I eventually upgraded to 8.10, which resolved the PCI SATA card problem. However, I never got around to rebuilding the system to use the extra hard drive, and continued running it with only two of the three drives active. It ran perfectly like that until I started experiencing problems printing PDF’s from my laptop, which is running the latest Ubuntu / CUPS / PDF viewer.

So I decided to upgrade the server to the latest 10.04 Ubuntu Server, which has been out for about 4 months now.

The upgrade to 9.04 went really smoothly, with no apparent problems. So I decided to push on to 9.10, which also went smoothly. And on the basis that those upgrades went so well, I decided to bring my server completely up to date, and upgrade to 10.04. And pushed my luck just a little too far.

Under 10.04 my server would randomly hang within at most 10 minutes. An inspection of the logs also showed that the problem with the PCI SATA card not working properly had returned, which may or may not have been the root cause of the hangs. Either way, I needed to back out the changes and return to 9.10. Which turned out to be harder than it should have been, because back when I’d originally partitioned the server, I’d not been very experienced, and so hadn’t adopted a very sensible partitioning scheme and hadn’t separated my data from the operating system. Getting back to 9.10 from 10.04 was going to be a ground up reinstall.

Since a reinstall was on the cards, I decided I may as well optimise the RAID array again. And adopt a more sensible partitioning scheme that separates the system from the data (in case I ever need to do a reinstall again!) And finally, because I wanted to spread my data across three different parts of the filesystem hierachy (/srv, /home and /var/www) it seemed to make sense to allow myself some flexibility in the future by finally adopting LVM.

To backup the system, I acquired a new 2.5″ 500GB drive (how technology has moved on in only 3 years!) for £55. Although apparently slightly extravagant, I am in the process of getting a new laptop at work, and will be able to use the 500GB drive in that to provide me with a lot more storage than I could otherwise have. I then backed up the servers entire filesystem to the new drive, before wiping all three server drives.

I then installed Ubuntu Server 9.10 onto the server, partitioning the drive into a 3-way RAID 1 set for /boot, and a 3-way RAID 5 set for LVM to then subdivide as required for my final logical partitioning. I then installed all of /boot, /, swap, /home, /srv and /var/www onto separate logical LVM partitions, and installed Ubuntu server onto the resulting system. All went perfectly, until the final reboot, when the system failed to restart, unable to mount the LVM partitions & filesystems. All my attempts to fix the problem failed, and in the end, after a day of fighting with it, I gave up and repartitioned the drives as a series of RAID 5 sets for the individual partitions that I needed. The drawback is that I cannot easily resize partitions in the future – a frustration to be true, but not a insurmountable problem.

In the end it’s taken over a week for me to get the server back up, and all the different services restored to full operation again. It’s not been a good experience. And my plans for “upgrading” in the future now all revolve around full OS reinstalls, preserving my data in their separate partitions.