This HOWTO assumes that you have an inkling of what Xen does and just want to get it up and running to experiment. Or, you may be like me and need to get an entire LVS Linux cluster up and running for work, but don't have 6+ separate machines.

If you're interested in full details of Xen please visit the Xen website:

Briefly, however, Xen is a virtualization technique which allows several (Linux and BSD at this time) Operating Systems to run on a single piece of hardware. Xen is actually a para virtualization technique (read the real documentation for the full description), but is most similar to VMWare ESX (the full Enterprise Edition) in operation. Xen is not an emulation technique like Wine which uses a software layer to allow (in the case of Wine), MS Windows DLL's to "talk" with Linux programs. Xen is a specialized version of the Linux kernel which "grabs" all the hardware from the system during bootup and then allocates abstracted access to that hardware for the virtual Operating Systems (domains). (No flame wares meant to be started here, I simply want to categorize where Xen fits in the grand scheme).

I highly recommend the FAQ and Architecture papers for an overview and the Users' Guide at

for general installation instructions (non-Gentoo specific) and the full list of domain parameters and command/gui domain control
features. It's fascinating reading, but there is a lot to learn and I've tried to distill the essence of Xen and Gentoo-ify it here.

Happy Xen-ing and I hope this helps at least one person.

----

Prerequisites

1. Gentoo on x86 (or x86_64 installed 32 bit). The July '05 Xen release will be available for x86_64 full 64 bit and will clean and thin the Xen code. My understanding is that the current Xen code is off a 2.4 kernel or very early 2.6 kernel and that v3.0 will update much of
the code and thin it out.

2. (WARNING) glibc: Xen currently doesn't play well with NPTL (tls emulation is very slow), so use either.

USE="-nptl"
or
USE="nptl -nptlonly" and mv /lib/tls /lib/tls.disable

Note: recompiling glibc w/ different threading options may break other programs. I had to recompile openoffice, nautilus, etc... because the new glibc didn't have the proper header options. This may not have been caused by the nptl vs non-nptl, but by other things (ref the stage1 from stage3 tarball howto on these forums).

3. (WARNING) SATA: Several people reporting to xen-devel have been unable to get VIA and nVidia SATA to run on Xen. I have a VIA chipset on an MSI mb that I worked on for about 3 hours to with no success. The problem appears as a kernel lockup after SATA detection: drives are detected, but getting and handing off an interrupt fails (IRQ11 or 17 or something goes to IRQ0) and the kernel hangs. If anyone has success with Xen and SATA, particularly with these chipsets please write in.

There may be updates to these files or patches that need to go in the files directory, so move those into the proper locations as well. I'd like to be more helpful on exactly the files, but Xen is not in Portage yet. The above has worked for me as of 06/02/05.

Next, create the Xen ebuild in your portage overlay (look at make.conf for the portage overlay):

if you have a system w/o X: USE="-gtk -gtk2" b/c otherwise dependency for Python twisted will bring in X.

----

Xen Kernel(s)

There are two types of Linux kernels in Xen - privileged and unprivileged.

The privileged kernel is used to boot the system initially into Xen and contains all the hardware drivers and modules of a typical kernel. After booting into the privileged kernel (Xen0) this running "domain" is called Domain-0.

Unprivileged kernels, on the other hand have no kernel modules and obtain most/all of their hardware information from Xen and the Xen0 domain. (Note that this isn't strictly true and I'm still figuring out the details).

To make matters even more confusing a Xen0 kernel may be used as a XenU kernel. This is not how I explain the installation. I copy the entire xen0 sources to a xenU directory and keep my xen0 and xenU builds entirely separate. Choose for yourself, but I had extra disk space and I wanted to keep both xen0 and xenU kernels for experimentation.

To separate the xen0 and xenU kernels:

Code:

cd /usr/src

cp -Rp linux-2.6.11.10-xen0/ linux-2.6.11.10-xenU

Xen0 Kernel:

Code:

cd linux-2.6.11.10-xen0

make mrproper
cp ../2.6.11.config .config
make oldconfig
make menuconfig

where I usually store my current config (in this case 2.6.11.conf) in /usr/src. I'm using an existing kernel configuration file as
stated in the Xen User's Guide section 2.3.3.

Note: oldconfig doesn't work great sometimes and I have (over 4 installations) found that I have to manually say y/n/m over
the entire kernel. Still, it may save time.

Make sure the following options are used at a minimum for the Xen0 kernel

XenU kernels are reusable and not used during the physical system boot process. They are used by Xen when launching a new domain from Domain-0. Consequently there is no need to put the xenU kernels in /boot. After little deliberation and much knee jerk reaction I put my xenU kernel in /usr/share (LSB compliant?).

Code:

cd /usr/src/linux-linux-2.6.11.10-xenU
make mrproper
make xenU_defconfig
make

kernel - is the xenU kernel
memory - is the memory Xen allocates to the VM in MB
name - is the VM name (the domain name that shows up under xm list)
nics - number of virtual network cards in domain
ip - for static networking (best if it's on the same network as you usually
use)
disc - the ttylinux install is on a file which will be presented to the VM
operating system as a writable sda1
root - the root for ttylinux is on /dev/sda1 and is read only

Create Xen Runlevel

This isn't strictly necessary, but made things easier for me. I had some hardware problems with some pieces of the default runlevel (pcmcia card on my
laptop), and I wanted to experiment with Xen without X.

You should also have networking as Xen (tries) to automatically bring up a bridge from Domain-0 to other domains.
This worked for me in general. ttylinux doesn't have ssh, but does have a http daemon. Try hitting port 80.

TODO

Again, please forgive me for any inconsistencies, incompleteness etc... I figured having a start that people could try and
comment on was better than nothing. The fine people working in https://bugs.gentoo.org/show_bug.cgi?id=70161 have done a great job and it's time to get users on Xen and Xen in Portage.

As the I hope you find this helpful and welcome any feedback.

Here are a few things I will be working on:

1. Enumerating hardware problems and solutions (SATA, etc...)
2. Creating and using other domain images (I have worked with Red Hat ES 4 to good effect).
3. Troubleshooting domains and domain bootup.
4. Figuring out the true differences between Xen0 an XenU and how to get limited H/W drivers in XenU domains (iptables for example)._________________Success is the ability to go from one failure to another with no loss of enthusiasm. - Winston Churchill

Last edited by trad511 on Wed Jun 22, 2005 9:59 pm; edited 3 times in total

for the xen0 kernel, make sure to set
CONFIG_BLK_DEV_IDEDISK y
CONFIG_BLK_DEV_IDEGENERIC y
CONFIG_PARTITION_ADVANCED y
to prevent a non-bootable system failinf with
"VFS: Cannot open root device "306" or hda6
Please append a correct "root=" boot option
Kernel panic: VFS: Unable to mount root fs on hda6"
(description taken from http://kerneltrap.org/node/970 as I cannot reproduce my exact messages---any more *g*)

I got it. I guess that you have to have /boot mounted in order to get it in /boot/xen.gz

Sorry if I misunderstand, but are you talking about:

a. the initial copying of xen.gz to (a mounted) /boot, or
b. grub handoff to the kernel (as in grub.conf: kernel /xen.gz

If a. then I may have things in the wrong order. If b. then I don't understand.

Just trying to figure out where the HOWTO is confusing. Updating/beautifying now and will post in a couple days._________________Success is the ability to go from one failure to another with no loss of enthusiasm. - Winston Churchill

Domain 0 shouldn't need the frontend drivers in the kernel under most circumstances.

I was under the impression that the Domain 0 kernel had all the drivers and the other kernels were unprivileged - meaning that
they didn't have any special hardware drivers. I very well could be wrong, but went on the following from the Xen Users' Manual:

...one with a `-xen0' extension which contains hardware device drivers and drivers for Xen's virtual devices, and one with a `-xenU' extension that just contains the virtual ones.

2.3.3

Quote:

If you wish to build a customized XenLinux kernel (e.g. to support additional devices or enable distribution-required features), you can use the standard Linux configuration mechanisms, specifying that the architecture being built for is xen, e.g:

# cd linux-2.6.11-xen0
# make ARCH=xen xconfig
# cd ..
# make

You can also copy an existing Linux configuration (.config) into linux-2.6.11-xen0 and execute:

# make ARCH=xen oldconfig

You may be prompted with some Xen-specific options; we advise accepting the defaults for these options.

Note that the only difference between the two types of Linux kernel that are built is the configuration file used for each. The "U" suffixed (unprivileged) versions don't contain any of the physical hardware device drivers, leading to a 30% reduction in size; hence you may prefer these for your non-privileged domains. The `0' suffixed privileged versions can be used to boot the system, as well as in driver domains and unprivileged domains.

My experience and the documentation led me to believe that if you download Xen and install via Section 2.3.1 you would not need
any special configurations for drivers (does it compile them all????). I never did try to compile Domain 0 with no drivers (or did and
can't remember). Let me know if the Domain 0 configuration works without any drivers._________________Success is the ability to go from one failure to another with no loss of enthusiasm. - Winston Churchill

Where I currently have an unresolved problem (the reason I've not update the the domain/image section of the howto) is with a
RHEL-4 (CentOS-4) install image.

The way I installed the image was to boot the system with the installation CD and create a single partition installation (in my
case this was /dev/hda10). Because I was using this outside of Xen I updated my Gentoo grub.conf and rebooted into the new
installation no problem. Remember here that this installation will do hardware probing during install and then create an initrd based
on kernel modules specific to your installation. Also, the kernel installed with the installation includes all modules needed to run
the system (including iptables modules which I'll come back to in a moment).

To add this domain to Xen in the most simple manner I could think of I copied my /etc/xen/ttylinux configuration file to /etc/xen/rheltest.
Editing this file I used the XenU domain in /usr/share/xen/xenU-kernels/vmlinuz-2.6.11.10-xenU. Note that this is the exact same
kernel as used with ttylinux and that I changed a few networking parameters to make the domain unique.

Starting up this new domain I got a hung kernel boot. A search on the web stated that the RHEL initrd had to be used during the
boot process. So, ever in a rush to get this going, I added the initrd to the rheltest config file. The final file looks like:

(I'll post this later as I'm not able to access the machine right now)

Note that now I am using the standard XenU kernel with the RHEL-4 initrd, which doesnt' seem quite right....
But, starting the domain this time works.

Here's the problem: All the hardware drivers and services relying on them from the original RH install fail. No wonder as there
are no USB kernel modules now. There is no hardware access to /dev/rtc. So, during startup, the Xen RHEL domain throws up a
bunch of non-fatal errors, runs Kudzu to reconfigure, etc... Ultimately when you go through and allow Kudzu to remove all these
devices from your configuration (don't do this with networking) the system will boot cleanly (minus the rtc which is solved elsewhere).

This domain is usable, but my lack of knowledge is in how to allow the domain virtual access to the hardware it wants. Haven't
researched and experimented enough here so if anyone has some tips and tricks I'd appreciate it.

The other problem with this domain is that there are absolutely no kernel modules and thus none of the iptables modules exist.
Even if the domain is virtual we still want access to all the networking functionality. My hypothesis is that you need to selectively
compile in modules ot the XenU kernel - like the iptables module - in order for domains to be fully functional. Haven't had a chance
to test this out yet.

If anyone has experience knowing how exactly to work with the unprivileged domains and their drivers/modules please post._________________Success is the ability to go from one failure to another with no loss of enthusiasm. - Winston Churchill

Here's how it works. Domain 0 has access to all physical devices since it's privileged, after all. Domain 0 would have configured both the physical device drivers as well as the frontend drivers so that it can export them as virtual disk and/or virtual network devices to the unprivileged domains.

Unprivileged domains can be configured with backend drivers so that they may serve physical device access to other domains. The unprivileged domain must get access to the physical device via the pci= settings in the domain configuration. It must also have the physical device driver configured in the kernel.

A domain cannot:

export a netif (act as a xen network backend) and import a netif (act as a xen network frontend) at the same time

export a blkif (act as a xen block device backend) and import a blkif (act as a xen block device frontend) at the same time

Could anyone with a running xen0 please post (or mail me) a working .config for the linux-2.6.11.10-xen0 kernel? After about 10 different kernels which all reset during boot (somewhere at PS/2 .. mice ... ) instantly without giving me the chance to actually read at which point the kernel resets, I'm a little .. fed up with configuring by blind guessing (and I don't want to continue till I'm desperate)

Thanks in Advance

* ElCondor pasa*_________________Here I am the victim of my own choices and I'm just starting!

CONFIG_IDE and CONFIG_BLK_DEV_IDE vs CONFIG_BLK_DEV_IDEGENERIC was a little .. disturbing - since I tried to keep the kernel small and removed everything unnecessary (and a little more as we found out )

Thanks again for the help, xen runs fine now

* ElCondor pasa *_________________Here I am the victim of my own choices and I'm just starting!

I installed the xen ebuild per the instructions in the post, and I'm not sure what's going wrong, as I'm not a programmer and don't know Python =). Any help would be appreciated, and thanks in advance,

~braynyac_________________I never really understood how there could be things that would drive you insane just because you knew them until I ran into Windows.
- Peter da Silva

I installed the xen ebuild per the instructions in the post, and I'm not sure what's going wrong, as I'm not a programmer and don't know Python =). Any help would be appreciated, and thanks in advance,

~braynyac

Edit /etc/xen/xend-config.sxp and change the port number from 8001 to something else, or stop whatever else is listening on that port (execute ``netstat -lnp | grep :8001'' as root). Maybe xend is already running?

cgs, thanks for the help =) I installed Twisted 2.0 and Twisted-web 0.5, and I now have xend and xendomains started. Now I'm getting some other problems...When I start the service I get this message:

Code:

* Starting Xen control daemon ...
/usr/lib/python2.3/site-packages/xen/xend/server/SrvDir.py:3: DeprecationWarning: twisted.protocols.http has moved to twisted.web.http. See http://twistedmatrix.com/projects/web.
from twisted.protocols import http

Everything starts OK, but it's annoying. The Twisted web site says that this may happen due to a fork in the project and that it's ok to ignore, but I'd still like to prevent it from showing up each time I start the xend service.

Also, when I run

Code:

xm list

as root, I get this:

Code:

Traceback (most recent call last):
File "/usr/sbin/xm", line 9, in ?
main.main(sys.argv)
File "/usr/lib/python2.3/site-packages/xen/xm/main.py", line 808, in main
xm.main(args)
File "/usr/lib/python2.3/site-packages/xen/xm/main.py", line 106, in main
self.main_call(args)
File "/usr/lib/python2.3/site-packages/xen/xm/main.py", line 124, in main_call
p.main(args[1:])
File "/usr/lib/python2.3/site-packages/xen/xm/main.py", line 343, in main
doms.sort()
AttributeError: 'str' object has no attribute 'sort'

I installed the xen ebuild per the instructions in the post, and I'm not sure what's going wrong, as I'm not a programmer and don't know Python =). Any help would be appreciated, and thanks in advance,

~braynyac

Edit /etc/xen/xend-config.sxp and change the port number from 8001 to something else, or stop whatever else is listening on that port (execute ``netstat -lnp | grep :8001'' as root). Maybe xend is already running?

Which version of twisted do have installed? Refer to i think,, um... 82 and 83 or 84 and 85 of the xen ebuild bug report in the Gentoo bug list and make sure you have the right version.

cgs, I had Twisted 2.0.0 installed, as well as Twisted-web 0.5.0. With these I was getting the errors previously mentioned. I referred to the comments you mentioned, and unmerged both of those packages, then emerged Twisted 1.3.0. Now I get this when I start xendomains:

Code:

Traceback (most recent call last):
File "/usr/sbin/xm", line 9, in ?
main.main(sys.argv)
File "/usr/lib/python2.3/site-packages/xen/xm/main.py", line 808, in main
xm.main(args)
File "/usr/lib/python2.3/site-packages/xen/xm/main.py", line 106, in main
self.main_call(args)
File "/usr/lib/python2.3/site-packages/xen/xm/main.py", line 124, in main_call
p.main(args[1:])
File "/usr/lib/python2.3/site-packages/xen/xm/main.py", line 343, in main
doms.sort()
AttributeError: 'str' object has no attribute 'sort'

I also get the same errors when I try to run 'xm list'.

Again, I am now on Twisted 1.3.0, and I can get xend and xendomains started, but I can not create any new domains, or list any current domains. Thanks again!

Edit
I just stopped both xend and xendomains and then restarted. Both came up without any errors. However, when I try to get the status of the xendomains service or run 'xm list' I now get this:

As mentioned earlier, I had something running on port 8000 (Shoutcast), which caused xend to not start. So I changed the ports in /etc/xen/xend-config.sxp to 8003 and 8004 (top two lines). This allowed xend and xendomains to start, but would not let me run 'xm list'. I shut off Shoutcast, changed the ports back to their defaults, stopped and started xend and xendomains, and 'xm list' now works!

Anyone know why changing the default ports would cause xend to break?

Thanks!

~braynyac_________________I never really understood how there could be things that would drive you insane just because you knew them until I ran into Windows.
- Peter da Silva

did anybody wrote some gentoo-style init scripts for xen?
I put two (prolly quite ugly and buggy ) ones together, but the short-time-memoryless idiot I am, I forgot to copy them on my iBook, so I could post them, when I find someplace with internet access

#!/sbin/runscript
#
# /etc/init.d/xendomains
# Start / stop domains automatically when domain 0 boots / shuts down.
#
# chkconfig: 345 99 00
# description: Start / stop Xen domains. ### I got no ****ing clue what chkconfig is ^^"
#
# The original script offers fairly basic functionality. It worked on LSB-compliant
# Linux Distributions. This one is way more ugly, but works on Gentoo. ;P
#
# Original based on the example in the "Designing High Quality Integrated Linux
# Applications HOWTO" by Avi Alkalay
# <http://www.tldp.org/HOWTO/HighQuality-Apps-HOWTO/>,
### This one on what I've seen in other scripts and and some trying ;)
### Florian Ermisch <floh@eb104.tu-berlin.de>