The Future of Xen

I’m currently in Xen hell. My Thinkpad (which I won’t replace any time soon) has a Pentium-M CPU without PAE support. I think that Debian might re-introduce Xen support for CPUs without PAE in Lenny, but at the moment I have the choice of running without Xen or running an ancient kernel on my laptop. Due to this I’ve removed Xen from my laptop (I’m doing most of my development which needs Xen on servers anyway).

Now I’ve just replaced my main home server. It was a Pentium-D 2.8GHz machine with 1.5G of RAM and a couple of 300G SATA disks in a RAID-1. Now it’s a Pentium E2160 1.8Ghz machine with 3G of RAM with the same disks. Incidentally Intel suck badly, they are producing CPUs with names that have no meaning, and most of their chipsets don’t support more than 4G of physical address space [1]. I wanted 4G of RAM but the machine I was offered only supported addressing 4G and 700M of that was used for PCI devices. For computation tasks it’s about the same speed as the old Pentium-D, but it has faster RAM access, more RAM, uses less power, and makes less noise. If I was going to a shop to buy something I probably would have chosen something different to get support for more than 4G of RAM, but as I got the replacement machine for free as a favor I’m not complaining!

I expected that I could just install the new server and have things just work. There were some minor issues such as configuring X for the different video hardware (and installing the 915resolution package (which is only needed in Etch) to get the desired 1650×1400 resolution. But for the core server tasks I expected that I could just move the hard drives across and have it work.

After the initial install the system crashed whenever I did any serious hard drive access from Dom0, the Dom0 kernel Oopsed and network access was cut off from the DomU’s (I’m not sure whether the DomU’s died but without any way of accessing them it doesn’t really matter much). As a test I installed the version of the Xen hypervisor from Unstable and it worked. But the Xen hypervisor from Unstable required the Xen tools from Unstable which also required the latest libc6, and therefore the entire Dom0 had to be upgraded. Then in an unfortunate accident unrelated to Xen (cryptsetup in Debian/Unstable warns you if you try to use a non-LUKS option on a device which has been used for LUKS and would have saved me) I lost the root filesystem before I finished the upgrade.

So I did a fresh install of Debian/Unstable, this time it didn’t crash on heavy disk IO, instead it would lock up randomly when under no load.

I’ve now booted a non-Xen kernel and it’s working well. But this situation is not acceptable long-term, a large part of the purpose of the machine is to run virtualisation so that I can test various programs under multiple distributions. I think that I will have to try some other virtualisation technologies. The idea of running KVM on real servers (ones that serve data to the Internet) doesn’t thrill me, Tavis Ormandy’s paper about potential ways of exploiting virtual machine technologies [2] is a compelling argument for para-virtualisation. Fortunately however my old Pentium-3 machines running Xen seem quite reliable (replacing both software and hardware is a lot of pain that I don’t want).

In the near future I will rename the Xen category on my blog to Virtualisation. For older machines Xen is still working reasonably well, but for all new machines I expect that I will have to use something else – and I’ll be blogging about the new machines not the old. I expect that an increasing number of people will be moving away from Xen in the near future. It doesn’t seem to have the potential to give systems that are reliable when running on common hardware.

8 comments to The Future of Xen

I have been using Xen since 2005. But instead of choosing Linux as Domain0, I choose NetBSD, one reason was pf, another that it was more easy to custom compile a kernel, and third reason networking was easier to understand and setup. I still use the same setup, running NetBSD and some Linux in DomU’s. No problems, long uptime’s.

However Xen evolved and version 3 come, and I had version 2 running of Xen. NetBSD which is a small community, didn’t have Xen 3 support, less PAE (now they begin to have). Beginning to feel the need to upgrade I begun to look for alternatives. Should I have to go back to Linux, messing with kernels and other things i didn’ like?

Then I catched the news (2006/2007) that Sun was working on integrating Xen into Opensolaris. I have since then been testing the functionality of Xen on Opensolaris. Xen on Opensolaris is a different, and more pleasing experience, a little bit like NetBSD, than Xen on Linux (which I also use). I get an intuitive feeling for networking and the interfaces. And things like Crossbow, which I run a beta release of just now (http://opensolaris.org/os/project/crossbow/CrossbowBeta/), makes it more interesting to run Xen (xVM) on Opensolaris. And nothing beats stability. Nvidia works out of the box (“accelerated graphics”) 32/64 bit, Xen or no Xen (try that on Linux). And that’s in the CrossbowBeta out of the box, together with the Zone functionality. And if Xen still feels ugly, VirtualBox may be a solution, also on Opensolaris (you have to download it).

The box I’m running now, Amd64 asus a8v deluxe, no longer boots the default Linux kernel found in Ubuntu 8.04, because of changes in the Linux kernel (bye bye kvm testing). But why bother compiling another custom kernel, when Solaris boots, NetBSD boots? I want stability. I want Xen. And for that I need a stable system. And Xen need’s a stable and mature operating system, with a stable kernel to run well. Therefore I welcome Opensolaris and the project “Indiana” which will be released this month. There I will continue to use Linux, Ubuntu 8.04, maybe Debian, NetBSD, and perhaps some Windows, in a stable environment of Opensolaris xVM.

Not to mention also in the “package”; zfs, cifs, iscsi, vscan, dtrace … all making virtualization and testing more easier.

My current setup is to use Xen on servers and KVM on desktops (if the hardware support is present). For your Pentium-M laptop I’d think that VirtualBox might be the best solution since you have neither 64-bit support nor VT support, with the caveat that you may want to replace it with KVM should you ever upgrade.

I can sympathize with your Xen problems. Right now all of my Ubuntu DomU’s are unable to be upgraded from 7.10 to 8.04 due to a bug in the kernel that causes similar problems to what you mentioned. Perhaps it would shed some light on your issues:

However, it makes no sense to use Xen on a laptop due to Xen lacking power management facilities. If Xen ever gets CPU throttling support I would be quite happy, and perhaps if KVM can catch up in other areas it might be a reason to switch, even on the server (electricity is expensive). I haven’t replicated tests recently but in the past I found KVM to be unsuitable for servers. For one thing it requires hardware virtualization support which not all of my servers have. Beyond that I found the scheduling to be sub-par. Xen is the only virtualization software I’ve used that fairly allocates resources. This can be tested by generating a high load in one domain and testing the responsiveness of the others.

As with all software it’s important to find the right tool for the right job and to keep your eyes on the other tools people are using.

tbl: Thanks for the suggestion, it’s interesting to know how other Unix systems work with Xen. Generally I don’t have enough time to learn about the other distributions of Linux, let alone other Unix systems.

Stefano: My understanding of OpenVZ is that it’s like VServer in it’s design. This doesn’t work for some of the things I want to do as I want to run different kernel versions as DomU’s (or equivalent).

Jeff: Thanks, I should have tried that before upgrading to Unstable.

specialj: My laptop doesn’t get much travel nowadays, mostly it’s moved between desks and even when running Xen the batteries last the duration of the journey. But you are correct that it is a problem with Xen and a win for KVM.

Anon: I liked it when they had different names, Pentium 3, Pentium 4, Pentium D, and different clock speeds and cache sizes within that range. They stuffed it up a little by releasing both 32bit and 64bit chips under the name “Celeron D” and then abandoned it entirely in recent times.

geb: That sounds like a BIOS issue, so probably a new BIOS will fix it. Not that it matters for me, after all the pain of this upgrade I’m even less enthusiastic about further upgrades. 3G is enough for what I want to do though, so I should be able to keep using the same hardware for at least another 5 years for that server.