Fedora is following the 3.0.x Xen line. Xen 3.0.0 was released in December of 2005 and is incompatible with guests using the previous Xen 2.0.x releases.

Quick Start

Setting up Xen and guests in Fedora 8 has some significant changes and improvements since the release of Fedora Core 6. The following guide will explain how to set up Xen and KVM, and how to create and manage guests using either the command line or GUI interface.

System Requirements

For Xen, GRUB, the default boot loader for is required[[FootNote(This is required because the system actually boots the Xen hypervisor and it then starts the Linux kernel. It does this using the MultiBoot standard.)]

For KVM, the system must have a CPU with virtualization support.

Sufficient storage space for the guest operating systems. A minimal command-line Fedora system requires around 600MB of storage, a standard desktop Fedora system requires around 3GB.

Generally speaking, at least 256 megs of RAM per guest plus 256 for the base OS. Practically speaking, it is hard to do work with virtualization with less than 1 GB of RAM.

Requirements for Para-virtualized Guests

Only Xen supports para-virtualized guests. KVM is full virtualization.

Any x86_64, or ia64 CPU is supported for running para-virtualized guests with Xen. For i386 hardware, a CPU with the PAE extension is required. Many older laptops (particularly those based on Pentium Mobile / Centrino) do not have PAE support. To determine if a CPU has PAE support, run the following command:

The above output shows a CPU that does have PAE support. If the command returns nothing, then the CPU does not have PAE support.

Fully-virtualized guests (HVM/Intel-VT/AMD-V)

To run fully virtualized guests in Xen or KVM, host CPU support is needed. This is typically referred to as Intel VT, or AMD-V. To check for Intel VT support look for the 'vmx' flag, or for AMD-V support check for 'svm' flag:

On some Intel systems, virtualization support needs to be enabled in the BIOS. Please ensure that this has been done before proceeding.

Installing the Virtualization Software

When doing a fresh install of Fedora 8, the virtualization packages can be installed by selecting Virtualization in the Base Group in the installer.

For an existing Fedora 8 installation, the Xen kernel, KVM, and other virtualization tools can be installed by running the following command:

su -c "yum groupinstall 'Virtualization'"

Enter the root password when prompted.

This installs python-virtinstkvm, qemu, and virt-manager. Optional packages in this group are xen, kernel-xen, gnome-applet-vm.

If kernel-xen is installed, there will be an entry in the file /boot/grub/grub.conf for booting the xen kernel. The xen kernel is not set as the default boot option.

To set GRUB to boot with kernel-xen by default, edit /boot/grub/grub.conf and set the default to the xen [[FootNote(Note that future kernel-xen packages can be set to the default kernel by editing /etc/sysconfig/kernel)]

This is an example /boot/grub/grub.conf configured to boot into the Xen hypervisor:

Booting into the Xen kernel is not necessary to work with KVM virtualization. It is only needed for Xen virtualiation.

Verify Virtualization is Enabled

Fedora 8 supports multiple underlying Virtualization platforms. Verifying that Virtualization is enabled depends on which platform is being used. For example, when using KVM the command to display all domains on the local system would be: virsh -c qemu:///system list. When using Xen,the command would be virsh -c xen:///system list.

To verify that Virtualization is enabled on the system, run the following command, where <URI> is a valid URI that libvirt can recognize. For more details on URIs: see http://libvirt.org/uri.html.

Note that for the default setup, networking for the guest OS (DomU) is bridged. This means that DomU's will get an IP address on the same network as Dom0. If a DHCP server provides addresses, it needs to be configured to give addresses to the guests. Another networking type can be selected by editing /etc/xen/xend-config.sxp

Configuring Remote Management

Fedora 8 adds the ability to manage virtual domains in a secure manner from remote hosts. To use these features, choose one of the following methods for the remote host to communicate with the hypervisor:

Create SSH keys for root, and use ssh-agent and ssh-add before launching virt-manager.

Set up a local certificate authority and issue x509 certs to all servers and clients. For information on configuring this option, refer to http://libvirt.org/remote.html.

Building a Fedora Guest System

With Fedora 8, installation of Fedora 8 guests using anaconda is supported. The installation can be started on the command line via the virt-install program or in the GUI program virt-manager.

If the system is booted into the Xen kernel, virt-manager will use Xen as the underlying platform. If the system is booted into the regular kernel, KVM will be used. To activate KVM when virt-manager is running in QEMU mode, check the "Use hardware acceleration" box in the wizard.

Building a Fedora Guest System using virt-install

The following questions about the new guest OS will be presented. This information can also be passed as command line options; run with an argument of --help for more details. In particular, kickstart options can be passed with -x ks=options.

What is the name of your virtual machine? This is the label that will identify the guest OS. This label will be used for various virsh commands and also appear in virt-manager the Gnome-panel Xen applet.

How much RAM should be allocated (in megabytes)? This is the amount of RAM to be allocated for the guest instance in megabytes (eg, 256). Note that installation with less than 256 megabytes is not recommended.

What would you like to use as the disk (path)? The local path and file name of the file to serve as the disk image for the guest (eg, /home/joe/xenbox1). This will be exported as a full disk to your guest.

How large would you like the disk to be (in gigabytes)? The size of the virtual disk for the guest (only appears if the file specified above does not already exist). 4.0 gigabytes is a reasonable size for a "default" install

Would you like to enable graphics support (yes or no): Should the graphical installer be used?

What is the install location? This is the path to a Fedora 8 installation tree in the format used by anaconda. NFS, FTP, and HTTP locations are all supported. Examples include:

Installation must be a network type. It is not possible to install from a local disk or CDROM. It is possible, however, to set up an installation tree on the host OS and then export it as an NFS share.

The installation will then commence. If graphics were enabled, a VNC window will open and present the graphical installer. If graphics were not enabled, the standard text installer will appear. Proceed as normal with the installation.

Building a Fedora Guest System using virt-manager

Start the GUI Virtual Machine Manager by selecting it from the "Applications-->System Tools" menu, or by running the following command as root:

su -c "virt-manager"

Enter the root password when prompted.

Open a connection to a hypervisor by choosing File-->Open connection...

Choose "qemu" for KVM, or "Xen" for Xen.

Choose "local" or select a method to connect to a remote hypervisor

After a connection is opened, click the new icon next to the hypervisor, or right click on the active hypervisor and select "New" (Note - the new icon is going to be improved to make it easier to see)

A wizard will present the same questions as appear with the virt-install command-line utility (see descriptions above). The wizard assumes that a graphical installation is desired and does not prompt for this option.

On the last page of the wizard there is a "Finish" button. When this is clicked, the guest OS is provisioned. After a few moments a VNC window should appear. Proceed with the installation as normal.

Building a Fedora Guest System using 'cobbler' and 'koan'

Cobbler is a tool for configuring a provisioning server for PXE, Xen, and existing systems. See http://cobbler.et.redhat.com for details. The following instructions are rather minimal and more configuration options are available.

Alternatively, cobbler can import a Fedora rsync mirror and create profiles automatically from there. Some of the imported distros will
be Xen profiles and some will be for bare metal. Usage of the Xen profiles will be required. See the manpage for details.

Bugs in the virt-manager tool should be reported in BugZilla against the 'virt-manager' component

Managing Virtual Machines from the command line with virsh

Virtual machines can be managed on the command line with the virsh utility. The virsh utility is built around the libvirt management API and has a number of advantages over the traditional Xen xm tool:

virsh has a stable set of commands whose syntax and semantics are preserved across updates to the underlying virtualization platform.

virsh can be used as an unprivileged user for read-only operations (e.g. listing domains, listing domain statistics).

virsh can manage domains running under Xen or KVM with no perceptible difference to the user

Bugs in the virsh tool should be reported in BugZilla against the 'libvirt' component.

Managing Virtual Machines from the command line with qemu-kvm

KVM virtual machines can also be managed in the command line using the 'qemu-kvm' command. See man qemu-kvm for more details.

Troubleshooting

SELinux

The SELinux policy in Fedora 8 has the neccessary rules to allow use of Xen with SELinux enabled. The main caveat to be aware of is that any file backed disk images need to be in a special directory - /var/lib/xen/images. This applies both to regular disk images, and ISO images. Block device backed disks are already labelled correctly to allow them to pass SELinux checks.

Log files

There are two log files stored on the host system to assist with debugging Xen related problems. The file /var/log/xen/xend.log holds the same information reported with 'xm log. Unfortunately these log messages are often very short and contain little useful information. The following is the output of trying to create a domain running the kernel for NetBSD/xen.

When reporting errors, always include the output from both /var/log/xen/xend.log and /var/log/xen/xend-debug.log.

If starting a fully-virtualized domains (ie to run unmodified OS) there are also logs in /var/log/xen/qemu-dm*.log which can contain useful information.

Finally, hypervisor logs can be seen by running the command

xm dmesg

Serial Console

Host serial console access

For more difficult problems, serial console can be very helpful. If the Xen kernel itself has died and the hypervisor has generated an error, there is no way to record the error persistently on the local host. Serial console lets you capture it on a remote host.

The Xen host must be setup for serial console output, and a remote host must exist to capture it. For the console output, set the appropriate options in /etc/grub.conf:

for a 38400-bps serial console on com1 (ie. /dev/ttyS0 on Linux.) The "sync_console" works around a problem that can cause hangs with asynchronous hypervisor console output, and the "pnpacpi=off" works around a problem that breaks input on serial console. "console=ttyS0 console=tty" means that kernel errors get logged both on the normal VGA console and on serial console. Once that is done, install and set up ttywatch to capture the information on a remote host connected by a standard null-modem cable. For example, on the remote host:

su -c "ttywatch --name myhost --port /dev/ttyS0"

Will log output from /dev/ttyS0 into a file /var/log/ttywatch/myhost.log

Paravirt guest serial console access

Para-virtualized guest OS will automatically have a serial console configured, and plumbed through to the Domain-0 OS. This can be accessed from the command line using

su -c "virsh console <domain name>"

Alternatively, the graphical virt-manager program can display the serial console. Simply display the 'console' or 'details' window for the guest and select 'View -> Serial console' from the menu bar.

Full Virt guest serial console access

Fully-virtualized guest OS will automatically have a serial console configured, but the guest kernel will not be configured to use this out of the box. To enable the guest console in a Linux fully-virt guest, edit the /etc/grub.conf in the guest and add 'console=ttyS0 console=tty0'. This ensures that all kernel messages get sent to the serial console, and the regular graphical console. The serial console can then be access in same way as paravirt guests:

su -c "virsh console <domain name>"

Alternatively, the graphical virt-manager program can display the serial console. Simply display the 'console' or 'details' window for the guest & select 'View -> Serial console' from the menu bar.

Accessing data on a guest disk image

There are two tools which can help greatly in accessing data within a guest disk image: lomount and kpartx.

Remember never to do this while the guest is up and running, as it could corrupt the filesystem

Note: always remember to deactivate the logical volumes with "vgchange -an", remove the partitions with "kpartx -d", and (if appropriate) delete the loop device with "losetup -d" after performing the above steps. The default volume group name for a Fedora install is always the same, it is important to avoid activating two volume group of the same name at the same time. LVM will cope as best it can, but it is not possible to distinguish between these two groups on the command line. In addition, if the volume group is active on the host and the guest at the same time, it can cause filesystem corruption.

Frequently Asked Questions

Q: I am trying to start the xend service and nothing happens, then when I do a virsh list I get the following:

A: You have rebooted your host into a kernel that is not a xen-hypervisor kernel. Yes I did this myself in testing :)

You either need to select the xen-hypervisor kernel at boot time or set the xen-hypervisor kernel as default in your grub.conf file.

Q. When creating a guest the message "Invalid argument" is displayed.

A. This usually indicates that the kernel image you are trying to boot is incompatible with the hypervisor. This will be seen if trying to run a FC5 (non-PAE) kernel on FC6 (which is PAE only), or if trying to run a bare metal kernel.

Q. When I do a yum update and get a new kernel, the grub.conf default kernel switches back to the bare-metal kernel instead of the Xen kernel

A. The default kernel RPM can be changed in /etc/sysconfig/kernel. If it is set to 'kernel-xen', then the Xenified kernel will always be set as default option in grub.conf

Getting Help

If the Troubleshooting section above does not help you to solve your problem, check the Red Hat Bugzilla for existing bug reports on Xen in FC6. The product is "Fedora Core", and the component is "kernel" for bugs related to the xen kernel and "xen" for bugs related to the tools. These reports contain useful advice from fellow xen testers and often describe work-arounds.