Ranting and Raving

Citrix XenServer 6.2 — Applying Public HotFixes

My XenServer 6.2 systems were pretty far out of date with regard to updates and hotfixes. So, yesterday I decided to apply the hot fixes available from XenServer.org.

We just use the software available from the XenServer.org site. We anticipated that we’d be able to apply these hot fixes using the XenCenter GUI but it seems that this isn’t an option if one doesn’t have a support license from Citrix.

I use and license some other products from Citrix but I don’t currently pay for support for XenServer. I might going forward as this was kind of a bear. But, we may also look at some other options, too. And, even with the ability to upgrade via the XenCenter GUI it still seems kind of a throw back in the current days where one can just apt-get update and apt-get dist-upgrade to keep a Linux (Debianbased/derived anyway) fully patched.

It turns out that one can still apply these hot fixes but it requires getting the packages to the server and using some command line tools. In my case I have a pool — several XenServer hosts grouped together — so I wanted to update all the servers in my pool.

I downloaded the following service packs and hotfixes from XenServer.org

XS62ESP1013 and XS62ESP1013 were also listed on the site at the time but we’re advisories not actual packages.

I unzipped all of the above hotfixes and copied the contents of each into a single folder called HotFixes.

Before I get into the details I ran into a few of obstacles before I even started…

I couldn’t upgrade XenCenter on my Windows 8.1 workstations as the installer complained about not having dotNet3.5 installed. I used the Windows control panel to add this feature but
still no joy. I’m not sure if this is a problem on the XenCenter side or the Windows side. Turns out I already had the current version: 6.2 (build 1377) so bascially a lot of clicking around for no reason.

I used WinSCP to upload my HotFixes to the /root directory on the server but my first attempt to apply the first service pack failed because there was not enough room on / partition to complete the install. So, I moved my HotFixes folder to a share on one of my regular Windows servers and then just used the mount command to reach the package files. The commands issued on the XenServer we along the lines of:

I learned that High Availability needed to be turned off for these steps to work. From within XenCenter click on the pool you are trying to upgrade. Click on the HA tab and disable High Availability.

I now had High Availability OFF and access to all the HotFixes without using any space on my local / partition on the XenServer pool master.

So, now the fun begins…

Get a list of all the UUIDs for each of the servers in your pool

xe host-list

As an example two of the UUIDs in my pool look like:

6c5fc1d8-d0be-46b9-abb8-912701ca41ef and e44fc0df-d147-4a2e-be62-6a1d51620f00

Copy and paste yours into a text doc as you will need the UUID for each host in the next few steps.

Upload each patch to the pool. Very important!: After uploading each patch make a note (copy and paste to a txt document) the UUID issued directly after you issue the upload command. You will need these UUIDs later as well as the UUIDs you collected about the servers in your pool above.

Apply the patches to EACH server in your pool. So for the following commands I’ll use the UUIDs for two of my servers above and the UUIDs that were issued to me by the patch-upload command. These are just examples. Use the actual UUIDs from the output of your own xe host-list and xe patch-upload commands.

On my pool master I issued the following commands to upgrade that specific server:

The above needs to be repeated changing the host-uuid for each member of your pool.

I shutdown each of the servers.

I powered on the pool master first. Once it was fully booted I powered on the other members of the pool.

Everything came back up. XenCenter could see that the patches had been applied. But all was not well!!!

I was able to start each of my Windows virtual machines. But, I was not able to start ANY of my Ubuntu/Linux based virtual machines. Uggh!

I got the following error:

Error: The bootloader for this VM returned an error -- did the VM installation succeed? Unable to find the partition containing the kernel

Each of my Linux VMs are Ubuntu 14.04 server instances. These use grub2 which seems to be an issue for XenServer 6.2. In the past to get around this I made an edit to the /usr/lib/python2.4/site-packages/grub/GrubConf.py file as outlined here. But, this edit threw syntax errors when I tried the start the Linux VM.

When I made the edit outlined above I just got a new and different error complaining about the script syntax.

I’m not a Python programmer and my little fix wasn’t getting the job done. So, I took a shot in the dark and changed line 428 of /usr/lib/python2.4/site-packages/grub/GrubConf.py
from the original:if arg.strip() == "${saved_entry}":

to:if arg.strip() == "${next_entry}":

That did the trick and all my Linux VMs started with no problem. I of course had to edit /usr/lib/python2.4/site-packages/grub/GrubConf.py on each of my XenServers.

Generally, I have very few problems with XenServer 6.2 (the grub2 thing is the biggest annoyance). That being said when I look into alternatives I may just look at running Xen on Ubuntu. Like I wrote earlier being able to stay up to date just issuing apt-get update; apt-get dist-upgrade commands really spoiles you against procedures like those above. If anyone out there has thoughts/ideas/suggestions or opinions they’d like to share on virtualized environments/options my ears are open.

All of the above was quite a chore. I do hope these notes help someone else out.