Linux Virtualization with Xen

Xen is the new virtualization kid on the block. It's gaining visibility and importance at a speed only projects such as Linux and Apache have seen before. Xen has been around for a couple of years: it was originally part of the Xenoserver platform, which aimed to build a public infrastructure for wide-area distributed computing. Ian Pratt, the principal investigator of the Xenoserver project at the University of Cambridge Computer Laboratory, still leads the development team.

Xen ended up being much more than a part of this project. Now many Linux distributions and some hardware vendors are picking it up. As with many important open source projects these days, it even has a company--Xensource--backing commercial versions and providing support for corporate customers. Xensource also employs several industry veterans. In short, Xen(source) has everything a good open source platform need to becomes an extremely important player in the industry.

At X-Tend, one of our main problems was that we didn't have enough machines to test all new distributions and applications. Basically there was no financially realistic way to provide our users with a quick test environment. My guess is that half of the planet has similar problems.

Ages ago, we used UserModeLinux, but most new users found it too complex to use. Later, we bootstrapped Qemu instances from our central imaging server. That worked. If we wanted to do some tests on an isolated environment, we quickly started Qemu with the distribution we needed. The only annoyance was that we actually wanted an environment that was constantly online and, in the event of a power outage (this is a test environment, not production), we didn't have to spend too much time getting it back. It had to be scriptable and automatable, and preferably would not require X.

With the arrival of Xen last year, all of that changed. This article describes how we tackled our problem and how we actually now have a stable and performant environment to test everything we want. It's so stable, we now use Xen for production environments!

What's Xen?

Xen is a virtual machine monitor for x86 that supports the execution of multiple guest operating systems with unprecedented levels of performance and resource isolation. Xen is open source software, released under the terms of the GNU General Public License.

Xen has become one of the most popular virtualization platforms during the last six months. Although it's not such a young project, it is now gaining acceptance in the corporate world as a valuable alternative to VMWare.

Adding Xen to your machine changes it from an ordinary x86 machine to a totally new platform. It's not an x86 anymore. It's a Xen machine. All the operating systems that you want to run on your machine won't work anymore if they know only about x86; they need to know about Xen. Of course, the Xen and x86 architecture are really similar, so for the end user and the applications that run on a platform ported to Xen, there is almost no difference.

When Xen is activated, it will also need to boot its first virtual machine, called Domain0. Domain0 has more privileges than the other virtual machines and typically is used only for managing the other (less privileged) virtual machines. Domain0 is also responsible for managing the hardware. Porting a platform to Xen changes almost nothing to the drivers, which means that most drivers supported in traditional Linux kernels also work in Xen.

Within Domain0, the xend daemon handles the management of the virtual machines. Control it via the xm command-line utility.

From there, you can create other virtual machines, or domains.

Xen and Different Distributions

We've been running Xen on different platforms ranging from an "antique" Suse 8.2 with a 2.4 series kernel, a Debian box, and Fedora Core 4 with a fresh 2.6 kernel. Unlike some other projects, Xen currently doesn't care whether you use 2.4 or 2.6, so people who are comfortable with a 2.4 kernel can still benefit from the Xen features. However, future releases probably won't have 2.4 support. People claim that installing Xen is difficult, but it's not, certainly if you compare it with other similar tools. GetXen.org is the place to start; it contains a tarball with most of the required binaries and tools, a demo CD, and pointers to the source code. Some distributions such as Fedora include prebuilt packages. As of this writing, the official stable Xen release is 2.0.7, but most people are already working with the 3.0 betas. 3.0 might be out by the time you actually read this.

It's really easy to start. Here's how we deployed a Debian virtual machine on a Fedora Core 4 install. We opted for a minimal FC4 install. After the installation, we updated, upgraded, and installed Xen with a couple of small commands:

Your version numbers may vary. If that's there, then it's time to reboot into that new entry. Voilá, you now have your first virtual machine up and running. Yes, at first sight the regular Linux version you have just booted into isn't running on a regular x86 anymore but is running on a Xen.

If you already started xend at boot time, run xm list to see output similar to:

Building a Virtual Host

Your next step is to create another virtual machine. The easiest way to do this is either to download an existing chroot image of the distribution you like or to build one yourself. Xen can use file-backed virtual block devices (dd if=/dev/zero of=vmdisk bs=1k seek 2048k count=1), physical devices (the actual /dev/hda9), LVM volumes (phy:VolumeGroup/root_volume), or an NFS root for your virtual machines. I prefer to use logical volumes on my machines, as they are really flexible to work with. With an existing disk /dev/sda5 available, I created logical volumes to use in my virtual machine:

It's almost ready. Now you need the configuration file for this virtual machine. Most of Xen's config files live in /etc/xen. You need a separate config file for each virtual machine you want to deploy on your host. They look like:

The config file is rather straightforward, and the Xen packages include examples. Now start your virtual machine with the command xm create config file. Add a -c to that command to see the machine booting. You should get a login prompt within seconds. That's how fast a physical machine should also boot, but I'll keep on dreaming for a couple of years.

If you create a symlink to the /etc/xen/auto directory, your virtual machines will start at boot time, if you enable the xendomains script at boot time.

Building a Different Virtual Host

This is Fedora on Fedora, but I promised to give you a Debian on a Fedora. This is where you need debootstrap. It comes in an RPM, but if you want a correct installation you need to find an up-to-date config script for Sarge, which you can easily find on any Debian box you have around. From there, again it's a matter of creating a new LVM entry, mounting it, and using debootstrap to populate the Debian instance:

Basic Xen virtual machine management is simple. Use xm shutdown and xm destroy, respectively, to do a clean shutdown or an immediate domain kill. For console access:

$ xm console $id
# and
$ xencons localhost $port

are similar and give you a better console than just a telnet localhost $port. (The port is 9600 + the $id of the particular virtual machine.)

Networking Virtual Hosts

Of course, you probably want to connect your virtual machines to the network. You first need to understand the bridging tool brctl. Xen provides one or more virtual network interfaces to your guest hosts, but in your Domain0 you will also see some changes to your default network config.

For each interface you define in a virtual machine, Xen will create a vifx.y interface, where x is the domain ID and y is the number of the interface in your virtual machine. For example, vif1.1 refers to eth1 in the domain with ID 1.

There are different ways of getting networking active, but I will show only one. Suppose that your network is 192.168.11.0 and your physical machine usually is at 192.168.11.2. You want to add your first virtual machine on 192.168.11.3. Log in to your virtual machine and configure the eth0 in the virtual machine to have the appropriate IP address, just as you would do if it were a physical machine.

Now you want to have physical network interface and the interface of your virtual machine (vifx.y) in a bridge. If you haven't already created xen-br0, use brctl addbr xen-br0 to do so, and then add both interfaces to it. Where vif1.0 is the first Ethernet device in your first domain, the commands are:

Your eth0 and vif1.0 don't need an IP address; you can put your 192.168.11.2 on the xen-br0 interface and route all the traffic for the 192.168.11.0 network to that xen-br0 interface. With this done and IP forwarding enabled, you should be able to reach the network. You should now be able to go from your virtual machine to other machines on the network and vice versa.

Looking Ahead

With Intel's announcement that it will ship its VT CPUs soon, operating systems without freely available source will soon run in a Xen environment. The 3.0 release happened in early December 2005, and every major Linux distribution is planning on using Xen somehow. Red Hat wants it in the kernel, and Suse already has added options in Yast for it.

The virtualization and consolidation market is fast changing, and Xen is playing an important role there!

Kris Buytaert
is a Linux and open source consultant operating in the Benelux. He currently maintains the openMosix HOWTO.