If everything is working, near the end of the output you will see <code>===== TEST FINISHED OK =====</code>. If things are not working, post the <i>complete, unedited</i> output of that command into a bug report.

== Networking ==

== Networking ==

Line 112:

Line 123:

== kvm ==

== kvm ==

−

See also the [http://kvm.qumranet.com/kvmwiki/Bugs KVM wiki page on reporting bugs].

+

See also the [http://www.linux-kvm.org/page/Bugs KVM wiki page on reporting bugs].

The output of any <code>qemu-kvm</code> command run by <code>libvirtd</code> is stored in <code>/var/log/libvirt/qemu/GuestName.log</code>.

The output of any <code>qemu-kvm</code> command run by <code>libvirtd</code> is stored in <code>/var/log/libvirt/qemu/GuestName.log</code>.

[[Testing_KVM_with_kvm_autotest|kvm-autotest]] is an excellent way of testing basic KVM functionality.

[[Testing_KVM_with_kvm_autotest|kvm-autotest]] is an excellent way of testing basic KVM functionality.

+

+

== Xen ==

+

+

Some useful information on how to debug Xen issues can be found in the [http://wiki.xen.org/wiki/Debugging_Xen Debugging Xen] Wiki page. If you think you found an actual bug, you may want to follow the steps outlined either on the [http://wiki.xen.org/wiki/Reporting_Bugs_against_Xen Reporting Bugs against Xen] Wiki page, or on [http://blog.xen.org/index.php/2013/06/04/reporting-a-bug-against-the-xen-hypervisor/ this] blog post.

+

+

The bugs that have been reported and are currently being tracked by the Xen developers are collected in the [http://bugs.xenproject.org/xen/ Xen Hypervisor Bug Tracker], so you may want to have a look there, to see if the bug you found is already being taken care of.

+

+

Some more useful information:

+

+

* log files are available at <code>/var/log/xen/</code>, for both HVM and PV guests (look for your guest name and domain ID)

+

* if your guest is crashing, we suggest you do the following:

+

** Set "on_crash=preserve" in your domain config file

+

** Copy the guest kernel's System.map to the host

+

** Once the guest has crashed, run <code>/usr/lib/xen/bin/xenctx -s System.map &lt;domid&gt;</code>

+

+

== General Tips ==

+

+

=== System Log Files ===

+

+

Always look in <code>dmesg</code>, <code>/var/log/messages</code> etc. for any useful information.

+

+

=== strace ===

+

+

<code>strace</code> can often shed light on a bug - e.g. if you run <code>virt-manager</code>, or <code>libvirtd</code> or <code>qemu-kvm</code> under strace you can see what files they accessed, what commands they executed, what system calls they invoked etc.:

+

+

<pre>

+

$> strace -ttt -f libvirtd

+

</pre>

+

+

If the program in question is already running, you can attach to it using <code>strace -p</code>.

+

+

=== gdb ===

+

+

<code>gdb</code> can often be useful to trace the execution of a program. However, in order to get useable information, you will need to install "debuginfo" packages. See the [[StackTraces]] page for more information.

+

+

=== SELinux ===

+

+

If you see "AVC denied" or "setroubleshoot" messages in <code>/var/log/messages</code>, your bug might be caused by an SELinux policy issue. Try temporarily putting SELinux into "permissive" mode with:

+

+

<pre>

+

$> setenforce 0

+

</pre>

+

+

If this makes your bug go away that doesn't mean your bug is fixed, it just narrows down the cause! You should include the AVC details from <code>ausearch -m AVC -ts recent</code> in the bug report, or if the message includes a <code>sealert -l</code> command then include the details printed by the command.

+

+

One common cause of SELinux problems is mis-labelled files. Try:

+

+

<pre>

+

$> restorecon /path/to/file/in/selinux/message

+

</pre>

+

+

If you are installing using an ISO on an NFS mount, you need to ensure that it is mounted using the <code>virt_content_t</code> label:

+

+

<pre>

+

$> mount -o context="system_u:object_r:virt_content_t:s0" ...

+

</pre>

+

+

If you are using libvirt storage pools, like nfs, or USB pass-through, you might want to check, or toggle one of the following SELinux booleans: virt_use_comm, virt_use_fusefs, virt_use_nfs, virt_use_samba, virt_use_usb.

+

+

<pre>

+

$> getsebool virt_use_nfs

+

virt_use_nfs --> off

+

$> setsebool -P virt_use_nfs on

+

</pre>

+

+

= Troubleshooting =

+

+

== Permission issues ==

+

+

Prior to Fedora 11/libvirt 0.6.1, all virtual machines run through libvirt were run as root, giving full administrator capabilities. While this simplified VM management, it was not very security conscious: a compromised virtual machine could possibly have administrator privileges on the host machine.

+

+

In Fedora 11/libvirt-0.6.1, security started to improve with the addition of [[Features/SVirt_Mandatory_Access_Control|svirt]]. In a nutshell, libvirt attempts to automatically apply selinux labels to every file a VM needs to use, like disk images. If a VM tries to open a file that libvirt didn't label, permission will be denied.

+

+

Fedora 12 saw things improve even more. As of libvirt-0.6.5, VMs were now launched with reduced process capabilities. This prevented the VM from doing things like altering host network configuration (something it shouldn't typically need to do). And as of libvirt-0.7.0, the VM emulator process was no longer run as 'root' by default, instead being run as an unprivleged 'qemu' user.

+

+

While all these changes are great for security, they broke previously working setups which depended on the relaxed VM permissions. Most issues have work arounds that come at the expense of security. Over time, many of these issues should be made to 'just work', but we aren't there yet.

To change the user that libvirt will run the QEMU/KVM process as, edit /etc/libvirt/qemu.conf and uncomment and change the user= and group= fields. For example, if wanting to run KVM as the user 'foobar', you would set the fields to <pre>...

{{admon/warning|Changing the SVirt/SELinux settings may have security implications.}}

+

+

SVirt can be disabled for the libvirt QEMU driver by editting /etc/libvirt/qemu.conf, uncommenting and setting <pre>security_driver='none'</pre> Then restart libvirtd with <pre>service libvirtd restart</pre>

+

+

=== Changing QEMU/KVM process capabilities ===

+

+

{{admon/warning|Changing the this setting has security implications.}}

Often times, VM slowness is caused because the VM is using plain QEMU and not KVM.

+

+

=== Ensuring system is KVM capable ===

+

+

Verify that the KVM kernel modules are properly loaded:

+

+

<pre>

+

$ lsmod | grep kvm

+

kvm

+

kvm_intel

+

</pre>

+

+

If that command did not list kvm_intel or kvm_amd, KVM is not properly configured. See [http://www.linux-kvm.org/page/FAQ#How_can_I_tell_if_I_have_Intel_VT_or_AMD-V.3F| this KVM wiki page] to ensure your hardware supports virtualization extensions. If it doesn't, you cannot use KVM acceleration, only plain QEMU is an option.

+

+

If your hardware does support virtualization extensions, try to reload the kernel modules with:

+

+

<pre>

+

su -c 'bash /etc/sysconfig/modules/kvm.modules'

+

</pre>

+

+

Retry the above lsmod command and see if you get the desired output. If not, or if the kvm.modules command produces an error, check the output of:

+

+

<pre>

+

dmesg | grep -i kvm

+

</pre>

+

+

If you see 'KVM: disabled by BIOS', please see the [http://www.linux-kvm.org/page/FAQ#.22KVM:_disabled_by_BIOS.22_error| relevant KVM wiki page]

+

Any other error message is probably a bug, and should be reported.

+

+

If all that works out fine, you want to make your that your VMs are actually using KVM

=== Is My Guest Using KVM? ===

=== Is My Guest Using KVM? ===

Line 122:

Line 263:

Often people are unsure whether their qemu guest is actually using hardware virtualization via KVM.

Often people are unsure whether their qemu guest is actually using hardware virtualization via KVM.

−

Firstly, check that KVM is available:

+

Firstly, check that libvirt thinks KVM is available:

<pre>

<pre>

Line 130:

Line 271:

</pre>

</pre>

−

and that the guest is configured to use KVM:

+

If that does not return anything, try this command to further identify what might need fixing to enable KVM support:

+

+

<pre>

+

$> virt-host-validate

+

</pre>

+

+

Next, check that the guest is configured to use KVM:

<pre>

<pre>

Line 138:

Line 285:

</pre>

</pre>

−

Next, look in <code>/var/log/libvirt/qemu/${guest}.log</code> to check that <code>/usr/bin/qemu-kvm</code> is the emulator that was executed by libvirt and that there are no error messages about <code>/dev/kvm</code>.

+

If that does not return anything, you want to make <domain type='kvm'> and <emulator>/usr/bin/qemu-kvm</emulator>, using the command:

−

+

−

If you are running a recent Fedora guest, you could try:

+

<pre>

<pre>

−

$> grep CONFIG_KVM_GUEST /boot/config-$(uname -r)

+

virsh edit ${guest}

−

CONFIG_KVM_GUEST=y

+

−

$> dmesg | grep KVM

+

−

Booting paravirtualized kernel on KVM

+

</pre>

</pre>

+

+

Next, look in <code>/var/log/libvirt/qemu/${guest}.log</code> to check that <code>/usr/bin/qemu-kvm</code> is the emulator that was executed by libvirt and that there are no error messages about <code>/dev/kvm</code>.

If you want to get really funky, you can check whether <code>qemu-kvm</code> has <code>/dev/kvm</code> open:

If you want to get really funky, you can check whether <code>qemu-kvm</code> has <code>/dev/kvm</code> open:

Line 158:

Line 302:

</pre>

</pre>

−

== xen ==

+

== Serial console access for troubleshooting and management ==

−

If a guest is crashing you can obtain a stack trace by doing the following:

+

Serial console access is useful for debugging kernel crashes and remote management can be very helpful.

−

* Set "on_crash=preserve" in your domain config

+

Fully-virtualized guest OS will automatically have a serial console configured, but the guest kernel will not be configured to use this out of the box. To enable the guest console in a Linux fully-virt guest, edit the /etc/grub.conf in the guest and add 'console=tty0 console=ttyS0'. This ensures that all kernel messages get sent to the serial console, and the regular graphical console. The serial console can then be access in same way as paravirt guests:

−

* Copy the guest kernel's System.map to the host

+

−

* Once the guest has crashed, run <code>/usr/lib/xen/bin/xenctx -s System.map &lt;domid&gt;</code>

+

−

== General Tips ==

+

<pre>

+

su -c "virsh console &lt;domain name&gt;"

+

</pre>

−

=== System Log Files ===

+

Alternatively, the graphical <code>virt-manager</code> program can display the serial console. Simply display the 'console' or 'details' window for the guest & select 'View -> Serial console' from the menu bar. <code>virt-manager</code> may need to be run as root to have sufficient privileges to access the serial console.

−

Always look in <code>dmesg</code>, <code>/var/log/messages</code> etc. for any useful information.

+

== Graphical console access ==

−

=== strace ===

+

In order to get a graphical console on your guest you can either use 'virt-manager' and select the console icon for the guest, or you can use the 'virt-viewer' tool to just directly connect to the console:

−

+

−

<code>strace</code> can often shed light on a bug - e.g. if you run <code>virt-manager</code>, or <code>libvirtd</code> or <code>qemu-kvm</code> under strace you can see what files they accessed, what commands they executed, what system calls they invoked etc.:

+

<pre>

<pre>

−

$> strace -ttt -f libvirtd

+

virt-viewer guestname

</pre>

</pre>

−

If the program in question is already running, you can attach to it using <code>strace -p</code>.

+

== Accessing data on guest disk images ==

−

=== gdb ===

+

{{Admon/caution | If the guest image might be live, you must only use read-only access, otherwise you risk corrupting the disk image.<br><br>It is always safe to use <code>guestfish --ro</code>}}

−

<code>gdb</code> can often be useful to trace the execution of a program. However, in order to get useable information, you will need to install "debuginfo" packages. See the [[StackTraces]] page for more information.

+

The [http://libguestfs.org/guestfish.1.html guestfish] program lets you manipulate guest disk images without needing to run the guest:

−

+

−

=== SELinux ===

+

−

+

−

If you see "AVC denied" or "setroubleshoot" messages in <code>/var/log/messages</code>, your bug might be caused by an SELinux policy issue. Try temporarily putting SELinux into "permissive" mode with:

+

<pre>

<pre>

−

$> setenforce 0

+

su -c 'yum install guestfish'

+

guestfish -d NameOfGuest -i --ro

+

&gt;&lt;fs&gt; ll /

+

&gt;&lt;fs&gt; cat /boot/grub/grub.conf

</pre>

</pre>

−

If this makes your bug go away that doesn't mean your bug is fixed, it just narrows down the cause! You should include the AVC details from <code>ausearch -m AVC -ts recent</code> in the bug report, or if the message includes a <code>sealert -l</code> command then include the details printed by the command.

+

See <code>man guestfish</code> and [http://libguestfs.org/ the libguestfs website] for information and examples. guestfish can also be scripted.

−

One common cause of SELinux problems is mis-labelled files. Try:

+

[http://libguestfs.org/virt-rescue.1.html virt-rescue] is an alternative libguestfs tool which you can use to make ad hoc changes. [http://libguestfs.org/virt-edit.1.html virt-edit] can be used to edit single files in guests, eg:

<pre>

<pre>

−

$> restorecon /path/to/file/in/selinux/message

+

virt-edit NameOfGuest /boot/grub/grub.conf

</pre>

</pre>

−

If you are installing using an ISO on an NFS mount, you need to ensure that it is mounted using the <code>virt_content_t</code> label:

+

= Known issues =

−

<pre>

+

== Audio output ==

−

$> mount -o context="system_u:object_r:virt_content_t:s0" ...

+

+

Audio has always been difficult to get working with libvirt, but the recent security changes have actually provided the mechanisms to make it work. The primary problem is that the VM is not sending sound output to your user's pulseaudio session. There may be a pulseaudio option to work around this issue, but I've managed to make it work with:

This will eventually be solved out of the box by having the VNC graphical client receive audio from the VM and play it as the current user. Some code exists to handle this for virt-viewer/virt-manager, but it isn't 100% complete yet. For more info, see [https://bugzilla.redhat.com/show_bug.cgi?id=595880 gtk-vnc bug 595880], [https://bugzilla.redhat.com/show_bug.cgi?id=536692 libvirt SDL audio bug 536692], [https://bugzilla.redhat.com/show_bug.cgi?id=508317 libvirt VNC audio bug 508317]

+

+

== SDL Graphics ==

+

+

QEMU needs access to your $XAUTHORITY file in order to use SDL graphics.

(stick that string somewhere in the <devices> block, remove any other <graphics> devices)

</pre>

</pre>

−

[[Category:Virtualization]] [[Category:Debugging]]

+

* [[SELinux/FAQ#How_do_I_enable_or_disable_SELinux_.3F|Set selinux to permissive]]. For more info, see [https://bugzilla.redhat.com/show_bug.cgi?id=609279 bug 609276]

+

+

* Give VM user access to your $XAUTHORITY file. The default VM user in Fedora 12+ is 'qemu', so you can provide read access with <pre>setfacl -m u:qemu:r $XAUTHORITY</pre> If you get an 'operation not supported' error, you can optionally provide less discerning read access with <pre>chmod +r $XAUTHORITY</pre> Beware, this probably has security implications. If that does not work, you can optionally change the VM user to either root (behavior of older Fedora versions), or [[#Changing_the_QEMU.2FKVM_process_user|to your own regular user]]

Libvirt's default behavior of dropping QEMU/KVM process capabilities prevents PCI device assignment from working correctly. See [https://bugzilla.redhat.com/show_bug.cgi?id=573850 bug 573850] for more info. I only managed to get this working with the following steps:

Effective bug reporting

Reporting bugs effectively is an important skill for any Fedora user or developer.

Narrowing down the possible causes of the bug and providing the right information in the bug report allows a bug to be resolved quickly. Filing a bug report with little useful information can mean that your bug lays unresolved, possibly until it is closed automatically when the distribution version reaches "end of life".

You can also check what virtualization capabilities are available on your machine by running:

$> virsh capabilities

Guest Configuration

When filing a bug related to problems seen in the guest, include full details on the guest configuration including CPU architecture, RAM size, devices etc. This is most easily done by including the output of virsh dumpxml MyGuest or, in the case of qemu, the full qemu command line.

Virt Manager

Virt Manager stores a logfile in ~/.virt-manager/virt-manager.log.

Examine the log file and include any pieces that look like they might be useful in the bug report. If in doubt, attach the whole file to the bug.

You can also run virt-manager from the command line using virt-manager --no-fork and check whether any relevant messages were printed there.

virt-install

virt-install stores a log file in ~/.virtinst/virt-install.log.

Run virt-install using the --debug option to get detailed debug spew.

In order to gain access to a serial console during the install, you can use -x "console=ttyS0". Using a serial console combined with a VNC install can be very useful for debugging e.g. --nographics -x "console=ttyS0 vnc"

libvirt

Any program using libvirt can be debugged using the LIBVIRT_DEBUG=1 environment variable e.g.

Xen

Some useful information on how to debug Xen issues can be found in the Debugging Xen Wiki page. If you think you found an actual bug, you may want to follow the steps outlined either on the Reporting Bugs against Xen Wiki page, or on this blog post.

The bugs that have been reported and are currently being tracked by the Xen developers are collected in the Xen Hypervisor Bug Tracker, so you may want to have a look there, to see if the bug you found is already being taken care of.

Some more useful information:

log files are available at /var/log/xen/, for both HVM and PV guests (look for your guest name and domain ID)

if your guest is crashing, we suggest you do the following:

Set "on_crash=preserve" in your domain config file

Copy the guest kernel's System.map to the host

Once the guest has crashed, run /usr/lib/xen/bin/xenctx -s System.map <domid>

General Tips

System Log Files

Always look in dmesg, /var/log/messages etc. for any useful information.

strace

strace can often shed light on a bug - e.g. if you run virt-manager, or libvirtd or qemu-kvm under strace you can see what files they accessed, what commands they executed, what system calls they invoked etc.:

$> strace -ttt -f libvirtd

If the program in question is already running, you can attach to it using strace -p.

gdb

gdb can often be useful to trace the execution of a program. However, in order to get useable information, you will need to install "debuginfo" packages. See the StackTraces page for more information.

SELinux

If you see "AVC denied" or "setroubleshoot" messages in /var/log/messages, your bug might be caused by an SELinux policy issue. Try temporarily putting SELinux into "permissive" mode with:

$> setenforce 0

If this makes your bug go away that doesn't mean your bug is fixed, it just narrows down the cause! You should include the AVC details from ausearch -m AVC -ts recent in the bug report, or if the message includes a sealert -l command then include the details printed by the command.

One common cause of SELinux problems is mis-labelled files. Try:

$> restorecon /path/to/file/in/selinux/message

If you are installing using an ISO on an NFS mount, you need to ensure that it is mounted using the virt_content_t label:

$> mount -o context="system_u:object_r:virt_content_t:s0" ...

If you are using libvirt storage pools, like nfs, or USB pass-through, you might want to check, or toggle one of the following SELinux booleans: virt_use_comm, virt_use_fusefs, virt_use_nfs, virt_use_samba, virt_use_usb.

Troubleshooting

Permission issues

Prior to Fedora 11/libvirt 0.6.1, all virtual machines run through libvirt were run as root, giving full administrator capabilities. While this simplified VM management, it was not very security conscious: a compromised virtual machine could possibly have administrator privileges on the host machine.

In Fedora 11/libvirt-0.6.1, security started to improve with the addition of svirt. In a nutshell, libvirt attempts to automatically apply selinux labels to every file a VM needs to use, like disk images. If a VM tries to open a file that libvirt didn't label, permission will be denied.

Fedora 12 saw things improve even more. As of libvirt-0.6.5, VMs were now launched with reduced process capabilities. This prevented the VM from doing things like altering host network configuration (something it shouldn't typically need to do). And as of libvirt-0.7.0, the VM emulator process was no longer run as 'root' by default, instead being run as an unprivleged 'qemu' user.

While all these changes are great for security, they broke previously working setups which depended on the relaxed VM permissions. Most issues have work arounds that come at the expense of security. Over time, many of these issues should be made to 'just work', but we aren't there yet.

Changing the QEMU/KVM process user

Changing the QEMU/KVM process user has security implications.

To change the user that libvirt will run the QEMU/KVM process as, edit /etc/libvirt/qemu.conf and uncomment and change the user= and group= fields. For example, if wanting to run KVM as the user 'foobar', you would set the fields to

...
user='foobar'
group='foobar'
...

Then restart libvirtd with

service libvirtd restart

Changing SVirt/Selinux configuration

Changing the SVirt/SELinux settings may have security implications.

SVirt can be disabled for the libvirt QEMU driver by editting /etc/libvirt/qemu.conf, uncommenting and setting

security_driver='none'

Then restart libvirtd with

service libvirtd restart

Changing QEMU/KVM process capabilities

Changing the this setting has security implications.

Libvirt by default launches QEMU/KVM guests with reduced process capabilities. To disable this feature, edit /etc/libvirt/qemu.conf, uncomment and set

clear_emulator_capabilities=0

Then restart libvirtd with

service libvirtd restart

KVM performance issues

Often times, VM slowness is caused because the VM is using plain QEMU and not KVM.

Ensuring system is KVM capable

Verify that the KVM kernel modules are properly loaded:

$ lsmod | grep kvm
kvm
kvm_intel

If that command did not list kvm_intel or kvm_amd, KVM is not properly configured. See this KVM wiki page to ensure your hardware supports virtualization extensions. If it doesn't, you cannot use KVM acceleration, only plain QEMU is an option.

If your hardware does support virtualization extensions, try to reload the kernel modules with:

su -c 'bash /etc/sysconfig/modules/kvm.modules'

Retry the above lsmod command and see if you get the desired output. If not, or if the kvm.modules command produces an error, check the output of:

dmesg | grep -i kvm

If you see 'KVM: disabled by BIOS', please see the relevant KVM wiki page
Any other error message is probably a bug, and should be reported.

If all that works out fine, you want to make your that your VMs are actually using KVM

Is My Guest Using KVM?

Often people are unsure whether their qemu guest is actually using hardware virtualization via KVM.

Serial console access for troubleshooting and management

Serial console access is useful for debugging kernel crashes and remote management can be very helpful.

Fully-virtualized guest OS will automatically have a serial console configured, but the guest kernel will not be configured to use this out of the box. To enable the guest console in a Linux fully-virt guest, edit the /etc/grub.conf in the guest and add 'console=tty0 console=ttyS0'. This ensures that all kernel messages get sent to the serial console, and the regular graphical console. The serial console can then be access in same way as paravirt guests:

su -c "virsh console <domain name>"

Alternatively, the graphical virt-manager program can display the serial console. Simply display the 'console' or 'details' window for the guest & select 'View -> Serial console' from the menu bar. virt-manager may need to be run as root to have sufficient privileges to access the serial console.

Graphical console access

In order to get a graphical console on your guest you can either use 'virt-manager' and select the console icon for the guest, or you can use the 'virt-viewer' tool to just directly connect to the console:

virt-viewer guestname

Accessing data on guest disk images

If the guest image might be live, you must only use read-only access, otherwise you risk corrupting the disk image.

It is always safe to use guestfish --ro

The guestfish program lets you manipulate guest disk images without needing to run the guest:

virt-rescue is an alternative libguestfs tool which you can use to make ad hoc changes. virt-edit can be used to edit single files in guests, eg:

virt-edit NameOfGuest /boot/grub/grub.conf

Known issues

Audio output

Audio has always been difficult to get working with libvirt, but the recent security changes have actually provided the mechanisms to make it work. The primary problem is that the VM is not sending sound output to your user's pulseaudio session. There may be a pulseaudio option to work around this issue, but I've managed to make it work with:

PCI device assignment

Libvirt's default behavior of dropping QEMU/KVM process capabilities prevents PCI device assignment from working correctly. See bug 573850 for more info. I only managed to get this working with the following steps: