Last year, I announced that we were working on a XenServer Administrators Handbook, and I'm very pleased to announce that it's been published. Not only have we been published, but based on the Amazon reviews to date we've done a pretty decent job. In part, I suspect that has a ton to do with the book being focused on what information you, XenServer administrators, need to be successful when running a XenServer environment regardless of scale or workload.

The handbook is formatted following a simple premise; first you need to plan your deployment and second you need to run it. With that in mind, we start with exactly what a XenServer is, define how it works and what expectations it has on infrastructure. After all, it's critical to understand how a product like XenServer interfaces with the real world, and how its virtual objects relate to each other. We even cover some of the misunderstandings those new to XenServer might have.

While it might be tempting to go deep on some of this stuff, Jesse and I both recognized that virtualization SREs have a job to do and that's to run virtual infrastructure. As interesting as it might be to dig into how the product is implemented, that's not the role of an administrators handbook. That's why the second half of the book provides some real world scenarios, and how to go about solving them.

We had an almost limitless list of scenarios to choose from, and what you see in the book represents real world situations which most SREs will face at some point. The goal of this format being to have a handbook which can be actively used, not something which is read once and placed on some shelf (virtual or physical). During the technical review phase, we sent copies out to actual XenServer admins, all of whom stated that we'd presented some piece of information they hadn't previously known. I for one consider that to be a fantastic compliment.

Lastly, I want to finish off by saying that like all good works, this is very much a "we" effort. Jesse did a top notch job as co-author and brings the experience of someone who's job it is to help solve customer problems. Our technical reviewers added tremendously to the polish you'll find in the book. The O'Reilly Media team was a pleasure to work with, pushing when we needed to be pushed but understanding that day jobs and family take precedence.

So whether you're looking at XenServer out of personal interest, have been tasked with designing a XenServer installation to support Citrix workloads, clouds, or for general purpose virtualization, or have a XenServer environment to call your own, there is something in here for you. On behalf of Jesse, we hope that everyone who gets a copy finds it valuable. The XenServer Administrator's handbook is available from book sellers everywhere including:

Overview

We were interested in getting XenServer 6.5 to boot via UEFI. Leaving servers in Legacy/BIOS boot was not an option in our target environment. We still have to do the initial install with the server in Legacy BIOS mode; however, I managed to compile Xen as an EFI bootable binary using the source and patches distributed by Citrix. With that I am able to change the servers boot mode back to UEFI and boot XenServer. Here are the steps I used to compile it.

Steps

Prepare a DDK

Prepare a build environment

Build some prerequisites

Unpack the SRPM

Compile Xen

DDK Preparation

Development will be done inside a 6.5 DDK. This is a CentOS 5.4-based Linux that has the same kernel as Dom0 and some of the required development tools.

Import the VM template per Citrix DDK developer documentation.

After importing, set the following VM options:

2 vCPUs

Increase memory to 2048MB

Resize disk image to 10GB

Add a network interface for SSH

Start the VM, set a root password, and then finalize resizing the disk by running:

Building the Prerequisites

This page: http://xenbits.xen.org/docs/4.3-testing/misc/efi.html says that it is required to use gcc 4.5 or better and that that binutils must be compiled with --enable-targets=x86_64-pep. I could not satisfy this with packages in the repositories, so I compiled and installed some requirements for gcc:

Compile Xen

The source code and required scripts are now all under /root/rpmbuild/, so just run:

# cd ~/rpmbuild
# QA_RPATHS=$[ 0x0020 ] rpmbuild -bc SPECS/xen.spec

The -bc flag causes the process to follow the spec file and patch the source, but then stop just before running the make commands. The make commands would fail to compile with warnings about uninitialized variables being treated as errors. Fix this by changing line 45 of ~/rpmbuild/BUILD/xen-4.4.1/xen/Rules.mk to read:

and the compile will finish successfully. I probably could have just made a quick patch to add the -Wno-error flag and allowed the rpmbuild to run the full spec file, but I didn't actually need to compile xen-tools etc, those are already compiled and installed on the XenServer installation. The only file needed is ~/rpmbuild/BUILD/xen-4.4.1/xen/xen.efi. With that in hand I created a xen.cfg file like this:

where the root UUID is the boot disk created during the XenServer install and the RSDP number came from running:

# dmesg | grep RSDP

I ran that in an EFI booted live Linux environment. I found that somevendors' UEFI implementations were able to provide the RSDP during bootand some were not, so without specifying it in the xen.cfg I had troublewith things like usb peripherals.

Boot XenServer

With the xen.efi and xen.cfg I was able to boot XenServer in UEFI boot mode using refind. We have done extensive testing on several different servers and found no problems. I was also able to repeat the process with the source code provided by the service packs up to and including Service Pack 1. I haven't tried any further than that yet.

Editors Note

For those of you wishing to retain Citrix commercial support status, the above procedure will convert the XenServer 6.5 host into an "unsupported configuration".

XenServer is a great choice of hypervisor for OpenStack based clouds, but there is no native integration between it and RedHat's RDO packages. This means that setting up an integrated environment using XenServer and RDO is more difficult than it should be. This blog post aims to resolve that, giving a method where CentOS can be set up easily to use XenServer as the hypervisor.

Environment

Hypervisor: XenServer: 6.5

Guest: CentOS 7.0

OpenStack: Liberty

Network: Neutron, ML2 plugin, OVS, VLAN

Install XenServer

The XenServer integration with OpenStack has some optimizations which means that only EXT3 storage is supported. Make sure when installing your XenServer you select Optimized for XenDesktop when prompted. Use XenCenter to check that the SR type is EXT3 as fixing it after creating the VMs will require deleting the VMs and starting again.

Install OpenStack VM

With XenServer, the Nova Compute service must run in a virtual machine on the hypervisor that they will be controlling. As we're using CentOS 7.0 for this environment, create a VM using the CentOS 7.0 template in XenCenter. If you want to copy and paste the scripts from the rest of the blog, use the name "CentOS_RDO" for this VM. Install the CentOS 7.0 VM but shut it down before installing RDO.

Create network for OpenStack VM

In single box environment, we need three networks, "Integration network", "External network", "VM network". If you have appropriate networks for the above (e.g. a network that gives you external access) then rename the existing network to have the appropriate name-label. Note that a helper script rdo_xenserver_helper.sh is provided for some of the later steps in this blog rely on these specific name labels, so if you choose not to use them then please also update the helper script.

Configure OpenStackVM/Hypervisor communications

Use HIMN tool (plugin for XenCenter) to add internal management network to OpenStack VMs. This effectively performs the following operations, which could also be performed manually in dom0 or use rdo_xenserver_helper.sh.

source rdo_xenserver_helper.sh create_himn

Note: If using the commands manually, they should be run when the OpenStack VM is shut down.

Set up DHCP on the HIMN network for OpenStack VM, allowing OpenStack VM to access its own hypervisor on the static address 169.254.0.1. Run helper script in domU.

Install XenAPI Python XML RPC lightweight bindings.

yum install -y python-pip pip install xenapi

Configure Neutron

Edit /etc/neutron/rootwrap.conf to support uing XenServer remotely.

[xenapi] # XenAPI configuration is only required by the L2 agent if it is to # target a XenServer/XCP compute host's dom0. xenapi_connection_url=http://169.254.0.1 xenapi_connection_username=root xenapi_connection_password=

Restart Nova and Neutron Services

for svc in api cert conductor compute scheduler; do service openstack-nova-$svc restart; done
service neutron-openvswitch-agent restart

Launch another neutron-openvswitch-agent to talk with dom0

XenServer has a seperation of dom0 and domU and all instances' VIFs are actually managed by dom0. Their corresponding OVS ports are created in dom0. Thus, we should manually start the other ovs agent which is in charge of these ports and is talking to dom0, refer xenserver_neutron picture.

So, you either just setup iSCSI or are having performance issues with your current iSCSI device. Here are some pointers to ensure "networking" is not the limiting factor:

1. Are my packets even making it to the iSCSI target? Always check in XenCenter that your NICS responsible for storage are pointing to the correct target IPS. If they are, ensure you can ping these targets from within XenServer's command line:

ping x.x.x.x

If you cannot ping the target, that may be the issue.

Use the 'route' command to show if XenServer has a device and target to hit on the iSCSI target's subnet. If route shows nothing related to your iSCSI target IPs or takes a long time to show the target's IP/Route information, revisit your network configuration: working from the iSCSI device config, switch ports, and all the way up to the storage interface defined for your XenServer(s).

Odds are the packets are trying to route out via another interface or there is a cable mismatch/VLAN tag mismatch. Or, at worse, the network cable is bad!

2. Is your network really setup for Jumbo Frames? If you can ping our iSCSI targets, but Re having performance issues with Jumbo Frames (9000 or 4500 Mtu size, based on vendor) ensure your storage interface on XenServer is configured to leverage this Mtu size.

One can also execute a ping command to see if there is fragmentation or support enabled for the larger MTUs:

ping x.x.x.x -M do -s 8972

This tells XenServer to ping, without fragmenting frames, your iSCSI target with an Mtu of 9000 (the rest comes from the ping and other overhead, so use 8972).

If this return fragments or other errors, check the cabling from XenServer along with the switch settings AND iSCSI setup. Sometimes these attributes can be powered after firmware updates to the iSCSI enabled, managed storage devicd

3. Always make sure your network firmware and drivers are up to date!

And these are but three simple ways to isolate issues with iSCSI connectivity/performance. The rest, well, more to come...

Overview

The following is a reminder of specific steps to take before electing a new pool master - especially in High Availability-enabled deployments. Albeit, there are circumstances where this will happen automatically due to High Availability (by design) or in an emergency situation, but never-the-less, the following steps should be taken when electing a new pool master where High Availability is enabled.

Disable High Availability

Before electing a new master one must disable High Availability. The reason is quite simple:

If a new host is designated as master with HA enabled, the subsequent processes and transition time can lead to HA see that a pool member is down. It is doing what it is supposed to do from the "mathematical" sense, but from "reality" it is actually confused.

The end result is that HA could either recover with some time or fence as it attempts to apply fault tolerance in contradiction to the desire to "simply elect a new master".

It is also worth noting that upon recovery - if any Guests which had a mounted ISO are rebooted on another host - that "VDI not found" errors can appear although this is not the case. The ISO image that is mounted is seen as a VDI and if that resource is not available on another host, the Guest VM will fail to resume: presenting the generic VDI error.

Steps to Take

HA must be disabled and for safe practice, I always recommend ejecting all mounted ISO images. The latter can be accomplished by executing the following from the pool master:

xe vm-cd-eject --multiple

As for HA it can be disabled in two ways: via the command-line or from XenCenter.

From the command line of the current pool master, execute:

xe pool-ha-disable

xe pool-sync

If desired - just for safe guarding one's work - those commands can be executed on every other pool member.

As for XenCenter one can select the Pool/Pool Master icon in question and from the "HA" tab, select the option to disable HA for the pool.

Workload Balancing

For versions of XenServer utilizing Workload Balancing it is not necessary to halt this.

Now that HA is disabled, switch Pool Masters and when all servers are in an active state: re-enable HA from XenCenter or from the command line:

xe pool-recover-slaves

xe pool-ha-enable

I hope this is helpful and as always: questions and comments are welcomed!

Testimonial

"Our job is to accommodate all the faculties’ needs as much as possible so we needed to find a solution that could support a large number of applications as well as save storage space and staff resources. This is where Citrix stepped in."

Jose ChanHead of IT DepartmentMacau Polytechnic Institute

"We expect that total costs for server infrastructure will be reduced by more than 35% because of XenServer."