Non-root GPU passthrough setup

If you want to use Linux as your main operating system and don’t want to do compromises like using a dual-boot solution with Windows there is an alternative called GPU passthrough. You basically pass through your GPU into a virtual machine s.t. your guest can fully utilize it according to your desires. The bottleneck of doing this is negligible (in my tests the 3DMark benchmark results were almost equal).

There are already a lot of guides and tutorials about how to make a GPU passthrough setup with QEMU. Among the best resources I could find are:

Although considering all of those and putting a lot of effort into setting up my own setup I had to deal with many annoying and very time consuming issues.
One of those issues is that most of the available guides just start QEMU with root privileges. This means a QEMU breakout directly leads to full control over your host system.

This article is meant as a frequently updated guide (last update: 09.07.2016) and (hopefully) complete walkthrough for everyone who is considering to make a non-root GPU passthrough setup with QEMU.
It was successfully tested with an up-to-date Archlinux and Xubuntu 15.10 (Wily), but should be compatible with any other distribution having a newer kernel version (preferably versions >= 3.9).

GPU passthrough with QEMU

Under normal circumstances there should be no risks doing this, however, I don’t take any responsibilities whatsoever in case you still manage to somehow brick your system by following this guide.

Hardware requirements

This setup has the following requirements on your hardware:

IOMMU compatible hardware (referred to as VT-D for Intel and VI for AMD):

CPU

Mainboard and bios (reading the mainboard’s online manual can help).

Two GPUs: one will be used for the host system and the other for the guest. An integrated GPU i.e. iGPU will do for the host.

One monitor with two different inputs like DP and HDMI. Two inputs are necessary, because you need to switch between your host and guest video output. You can also use a KVM switch or utilize a second monitor (the latter is recommended).

Basic preparations

For this guide we will use QEMU (which makes use of KVM). One graphics card will be exclusively allocated to the guest system while the other takes care of running our host system. To achieve this we can make use of the Linux VFIO driver (available in kernel versions >= 3.9?).

Make sure your system is up to date:

1

2

sudo apt-get update

sudo apt-get upgrade

Enable IOMMU support in your bootloader: ensure the default grub config contains “intel_iommu=on” if you have an Intel cpu and “amd_iommu=on” for an AMD cpu:

1

2

3

/etc/default/grub

[...]

GRUB_CMDLINE_LINUX_DEFAULT="[...] intel_iommu=on"

then do

1

sudo grub-mkconfig | sudo tee /boot/grub/grub.cfg

and ensure /boot/grub/grub.cfg is properly updated.

Dedicated GPU isolation

Dedicated GPU isolation can be done by using pci-stub with VFIO (available since 3.9+) or directly using vfio-pci (available since 4.1+). The latter is recommended since you don’t need to rebind the GPU with VFIO after each boot. Ensure it is installed by running:

1

modprobe vfio-pci

No output means everything is ok. If there is an error you need to install pci-stub (c.f. related sections in Puget system’s guide).

At this point there are at least two possibilities to load the correct modules:

In this case the important ids are: 10de:17c8 and 10de:0fb0. Further, we can then add those ids to the VFIO modprobe config so that the VFIO-PCI drivers will be bound to those devices on boot:

1

2

/etc/modprobe.d/vfio-pci.conf

options vfio-pci ids=10de:17c8,10de:0fb0

Attention:
in case you have two gpus with same ids this won’t work. “Another issue that users encounter when sequestering devices is what to do when there are multiple devices with the same vendor:device ID and some are intended to be used for the host.” – please consider reading and following this specific part here: http://vfio.blogspot.de/2015/05/vfio-gpu-how-to-series-part-3-host.html

Finally, for Debian run:

1

sudo update-initramfs -u

and for Archlinux with a normal linux kernel run:

1

mkinitcpio -p linux

in order for the changes to be applied on boot. At this point you should restart your system and verify with “lspci -nnk” if the entry “Kernel driver in use: vfio-pci” is available for the dedicated GPU.

In case you have made it this far most things should be ready for the next step: setting up QEMU and telling it to forward our dedicated GPU.

QEMU setup

Install QEMU (assuming you have a x86-64 system):

1

sudo apt-get install qemu-system-x86-64

First we will need a QEMU image for our VM (unless you want to forward a SATA controller or use another storage device for which I won’t go into detail at this point). Many guides use virtio controller and virtio drivers. Although they are improving performance, my advice is to avoid them at least for now since they unnecessarily complicate things. You can create a disk with 60 GB space and qcow2 format (makes use of compression) with:

1

qemu-img create -f qcow2 qemu_vm.qcow2 60G

Alternatively, you can use a raw image here, however, doing so won’t improve performance much and will eat up more space than necessary (I have to verify this claim first though). The next thing is taking care of the boot process. To be able to boot disks with GPT partitions we need EFI. Further, this requires UEFI (which is a replacement for the old bios version). UEFI support is provided by using OVMF and can be downloaded pre-compiled from Gerd Hoffman’s site https://www.kraxel.org/repos/jenkins/edk2/. Install it using:

1

rpm -i edk2.git-ovmf-x64-*.rpm

If you use a Debian based distribution like Ubuntu you can use the following commands to extract it:

Adjust the number of virtual cores, physical cores and threads with the “-smp” option for your CPU accordingly.

We use “kvm=off” to trick Nvidia cards into believing they are not passed through (else the card might reject working with errorcode 43).

In above configuration my dedicated GPU’s PCI ids were 02:00.0 and 02:00.1 you should lookup yours by executing “lspci” and adjust them accordingly.

Using “-vga qxl” makes QEMU use the QXL paravirtual graphic card as the main emulated VGA device. This gives us a window where we can directly see what’s going on in the VM. Many guides recommended directly making use of the passthrough and switch to the dedicated GPU’s output. However, I had no signal i.e. my screen just stayed black. This method makes it way easier to setup the OS first which then in turn will use the correct drivers. Further, we can avoid passing through any USB devices yet since they would lead to deadlocks during a failure within the vm.

Instead of an iso image you can also directly pass any device (e.g. passing a USB stick with -cdrom /dev/sdX where sdX is the correct device name works just fine).

Hint:Already at this point you can verify if the passthrough is working by starting QEMU and writing “info pci” into the QEMU console:

1

2

3

4

5

QEMU 2.5.0 monitor - type 'help' for more information

(qemu) info pci

[...]

Bus 0, device 5, function 0:

VGA controller: PCI device 10de:17c8

Here you can see that my device with the id 10de:17c8 was correctly mapped into the vm.

If you have fully installed your desired OS and booted into it you should make sure that the passed-through GPU is correctly identified (e.g. in the device manager of Windows or by using lspci in Linux). In my case I had to download and install the newest Nvidia GPU drivers in order for the GPU to be correctly identified in Windows 10. Once this step was done I was able to switch to the other monitor port and was finally able to see the guest’s dedicated GPU output :).

Hint:If you want to manage the VM with your dedicated GPU’s video output you can just click into the QXL window and then switch to your other monitor input.

Attention:A Linux guest can refuse to show output on the dedicated GPU when QXL is running (you can verify this by using “xrandr”). In order to fix this you can deactivate the QXL device s.t. only the dedicated GPU will be known to the guest:

1

2

# Use an emulated video device (use none for disabled).

OPTS="$OPTS -vga none -device qxl"

However, doing so will require passing through a mouse/keyboard to control the vm.

Update:
By appending “-device qxl” you can disable the QXL video output while still being able to control the VM exactly the same way as in the given hint above (thx to Reddit user /u/SxxxX for pointing this out).

The next step is to forward the keyboard and mouse for better responsiveness (QXL is too slow). Here you can either pass through a complete USB controller or rather pass through a specific USB device. Since I was unable to pass through a USB 3.0 controller I had to pass through specific devices (if you want to try to pass through a complete USB controller you can refer to Step 7 in Puget system’s guide).

Once you have all ids you can forward those devices in our QEMU startup script with:

1

2

3

4

5

6

# Passthrough USB devices.

OPTS="$OPTS -usb"

# USB mouse

OPTS="$OPTS -usbdevice host:046d:c051"

# USB keyboard

OPTS="$OPTS -usbdevice host:046d:c316"

Sound

We will now take care of getting working sound. To find out more about QEMU’s supported sound devices consider:

1

2

3

4

5

6

7

8

9

10

qemu-system-x86_64 -soundhw ?

Valid sound card names (comma separated):

sb16 Creative Sound Blaster 16

es1370 ENSONIQ AudioPCI ES1370

ac97 Intel 82801AA AC97 Audio

adlib Yamaha YM3812 (OPL2)

gus Gravis Ultrasound GF1

cs4231a CS4231A

hda Intel HD Audio

pcspk PC speaker

Adjust our startup script to use the Intel HD Audio (hda) device:

1

2

# Emulate a sound device.

OPTS="$OPTS -soundhw hda"

We further need to specify an audio driver we would like to use. Let’s give ALSA a try:

1

2

3

4

5

# Select a QEMU sound driver and specify its settings.

VM_SOUND=""

VM_SOUND="$VM_SOUND QEMU_AUDIO_DRV=alsa"

VM_SOUND="$VM_SOUND QEMU_ALSA_DAC_BUFFER_SIZE=512"

VM_SOUND="$VM_SOUND QEMU_ALSA_DAC_PERIOD_SIZE=170"

Alternatively you can also use Pulseaudio (it doesn’t have the best reputation though and I don’t recommend using it until really necessary…):

1

2

3

4

# Select a QEMU sound driver and specify its settings.

VM_SOUND=""

VM_SOUND="$VM_SOUND QEMU_AUDIO_DRV=pa"

VM_SOUND="$VM_SOUND QEMU_PA_SAMPLES=128"

To get more information about the above used sound parameters and some further sound driver information consider:

1

qemu-system-x86_64 -audio-help

Finally, we need to rewrite the QEMU startup script to consider the sound environment settings:

1

sudo $VM_SOUND qemu-system-x86_64 $OPTS

If you have crackles in your sound using ALSA try adjusting the buffer and period sizes.

Question:Do you have any advice about how to improve the input (microphone) quality at this point?

Whereas the above configuration worked instantly for Xubuntu, using Archlinux lead to horrible sound seemingly because of permission issues while setting the ALSA buffer and period sizes.
To solve this you can alternatively use the ac97 device and leave out any ALSA parameters in the environment variable.

1

2

3

4

# Emulate a sound device.

OPTS="$OPTS -soundhw ac97"

# Select a QEMU sound driver and specify its settings.

VM_SOUND="QEMU_AUDIO_DRV=alsa"

Attention:Windows won’t detect your sound device correctly. You have to use exactly the right drivers which are Intel drivers in this case (there are many drivers and you can waste a lot of time by choosing wrong ones here). You can find the correct AC 97 drivers for QEMU here: Intel 82801AA AC97 Audio (don’t download the first driver since this is adware. Use the second link with the zip file instead).

Network

The network seemed to work directly out of the box i.e. I could utilize about 5-6 MB/s with my 50 Mbit/s internet connection so I didn’t investigate further here. However, there are some things you can do to improve the performance if necessary (c.f. Performance Tweaks Section).

Synergy

Synergy allows you to share the mouse and keyboard across multiple operating systems which is very helpful when we pass through USB devices. I recommend using Synergy as a server within the guest system while running it as a client on the host system.

Attention:
Don’t use Synergy versions <1.7 (they have a bug that doesn’t allow you to press multiple keys at once. This is very annoying when typing capitalized letters…). You should download a version >= 1.7 (for your host and guest) at http://www.synergy-project.org/nightly.

Make the guest’s Synergy accessible by forwarding its default port (24800 is default):

Hint:In my case I had to add “setxkbmap de nodeadkeys” to fix an issue with the German keyboard layout and Synergy.

Now you can configure the Synergy server and client by setting a proper client name. Finally, you should be able to connect to localhost on the client side while Synergy is running on both systems.

Hint:During gaming you want to lock the mouse to one specific monitor. You can do this by pressing the Scroll-Lock button. If you don’t have a Scroll-Lock button on your keyboard you can specify another button in the Synergy server settings. In my case I have bound a mouse button for this.

If you are using Synergy as a server on your host and as a client in your guest (e.g. when you don’t want to use USB passthrough) and experience weird mouse movements ingame try setting “Send relative mouse moves” in the Synergy server.

Congratulations if you have made it this far!

Your basic GPU passthrough setup should work by now. We can now take care of removing the requirements of QEMU to be started as the root user.

Permissions for non-root GPU passthrough

At this point you should have a setup that is working as root. The next steps in this chapter will help to improve the overall QEMU security. Please keep in mind that using a passthrough setup even as non-root is risky since the GPU can have a lot of control over the system. Further, we are granting the VM access to other devices like USB devices which doesn’t really improve the situation. Thus, please don’t expect this chapter to be a guarantee for a really bulletproof setup. All we will do here is to decrease the attack surface as much as possible.

Basic permissions

We’ll create the user “qemu_vga” with a disabled password (s.t. you have to use “su”/”sudo” as an admin) for the purpose of running QEMU:

Ensure that our user has write access to our vm image and the OVMF variables file:

1

2

sudo chown qemu_vga:qemu_vga qemu_vm.qcow2

sudo chown qemu_vga:qemu_vga OVMF_VARS-pure-efi.fd

You have at least two options regarding the QEMU startup script:

Make sure the QEMU startup script is non writable by the qemu_vga user.

Never execute the QEMU startup script with your main user, but rather only execute it with “sudo -Hu qemu_vga /path/to/the/qemu_script.sh”.

If you neglect those points a possible breakout would mean the qemu_vga user would be able to write arbitrary commands into the startup script which would get executed by your user once you start the VM. For this guide I assume you use option 1, however, using option 2 would be okay, too. Just don’t forget to remove the redundant “sudo” command in the QEMU script if you decide to go with it.

Next, if you want to use QXL (recommended for testing) you need to give our qemu_vga user access to the XServer by modifying our QEMU script:

1

2

3

4

#!/bin/bash

xhost+SI:localuser:qemu_vga>/dev/null

OPTS=""

[...]

Attention:Granting XSever access to the user qemu_vga is a bad idea from many point of views regarding security. You should only do this while testing and using QXL. You can avoid any windows by using “-nographic” together with the “-vga none” option (c.f. Miscellaneous Section).

Device permissions

We need set proper permissions for the VFIO and USB devices by granting all users in the kvm group access. We will add a VFIO rule and our USB device ids (c.f. USB passthrough in the previous chapter) to the Linux udev rules like this:

Sound permissions

The next important step is to get the sound running. One option is to use the ac97 and ALSA without parameters solution (as described in the first Chapter). However, I wanted to be able to further customize ALSA with parameters to further improve the sound quality. This was definitively the most difficult part of the setup. Unfortunately, the following steps only worked for Xubuntu. When starting our setup at this point with the ALSA driver you might get the following error:

1

2

(qemu) alsa: Requested buffer size 512 was rejected, using 1881

alsa: Requested period size 170 was rejected, using 940

After a lot of research and even after reading QEMU-ALSA sourcecode I was unable to find the reason behind it. Alternatively trying Pulseaudio (as described in the previous chapter) lead to the following error:

Attention:
You might need to add “export PULSE_SERVER=unix:/tmp/pulse” to the QEMU startup script in addition (thanks to Sarnex).

Don’t forget to kill and restart the Pulseaudio daemon:

1

2

pulseaudio -k

pulseaudio -D

Attention:I later noticed that after rebooting my system there was a bug where the Pulseaudio daemon wouldn’t startup. Adding “pulseaudio -D” to the Application Autostart in Xubuntu solved this problem though.

By doing so I got Pulseaudio running. What absolutely stunned me was the fact that solving the Pulseaudio issue also solved the ALSA issue described before. I know that most Ubuntu versions (including Xubuntu) automatically come with Pulseaudio, too. However, the connection between ASLA, Pulseaudio and this error was very unclear to me.

While the only working solution on Archlinux was using ALSA with ac97, all sound device and driver combinations worked successfully under Xubuntu.

If you have reached this point you should have a basic GPU passthrough setup working with our qemu_vga user.

Performance tweaks

Hyper-V enlightments

You can activate several paravirt features within Windows by setting some CPU parameters referred to as Hyper-V enlightments.

However, setting hv parameters will lead to an NVIDIA error 43 with newer driver versions! Hence, we can use hv_vendor_id=Nvidia43FIX as a workaround. The latter only works with a newer QEMU version like 2.5.0 though.

“If you have an Intel CPU with APIC virtualization (starting with Ivy Bridge-E and Haswell), you should avoid using hv_vapic. It has more overhead than APICv, causing about 10-12% more VM exits.” – according to Reddit user /u/glowtape.

Since the version in my repository didn’t support this feature yet I compiled QEMU 2.5.0 on my own. You can find some very rudimentary advice on this in the Miscellaneous Section below.

Unfortunately, I don’t have any information on the real performance gain here. If you have any please feel free to contribute them here.

Disk tweaks

You can improve the disk IO performance by making use of the virtio drivers. You can either install them already during Windows installation by providing an ISO with the drivers as a secondary CDROM drive or by installing them afterwards.

Creates an iothread to move the IO away from the main QEMU thread, then specifies a disk drive without interface and ties it to a virtio SCSI controller. The relevant virtio driver to install in Windows is the SCSI Passthrough one.

Many thanks to Reddit user /u/glowtape for pointing this out and providing the basic parameters!

In my tests the speed performance jumped from roughly 300 MB/s reading speed to about 600-700 MB/s with a qcow2 image file.

Memory tweaks

Preallocation

Similar to VMware or other virtualization software you can preallocate enough memory for the vm:

1

2

3

# Assign 4 GB memory to the vm.

OPTS="$OPTS -m 4000"

OPTS="$OPTS -mem-prealloc"

Hugepages

You can make use of hugepages to further improve ram performance. Please consider:

Where qemu-ifup and qemu-ifdown are helper scripts. To make use of this you further need a bridge called br0 on your interface with internet access.

Big thanks to Reddit user /u/glowtape for providing the scripts and information necessary for this section!

More

Please let me know if you know of any further tricks to improve the machine performance here.

Miscellaneous

Rebinding

Apparently, it is also possible to assign the GPU to your host again once it has been used in the VM. I haven’t tested this yet but according to Reddit user /u/glowtape you can just bind and unbind the VFIO drivers. This has the drawback that you have to stop XServer for rebinding.

Please refer to his startup script to see how it can be done (links to the qemu-ifup and qemu-ifdown scripts can be found in the Network section above).

OVMF

You can adjust the standard resolution during boot (e.g. when using QXL) in the bios settings (press del key on boot). Go to “Device Manager” – “OVMF Platform Configuration” – “Change Preferred” and commit changes. This is helpful when using QXL for any tasks like OS management or OS installation.

QEMU settings

In case you don’t want the qemu_vga user to start a QEMU window or just don’t want to grant it any XServer rights you can use

1

2

3

# Use an emulated video device (use none for disabled).

OPTS="$OPTS -vga none"

OPTS="$OPTS -nographic"

Monitor switching

I don’t use it since I have to press a button on the monitor to switch the input. However, there are enough guides describing how you can utilize “xrandr” if your monitor supports automatic signal switching.

Request:Please feel free to contribute if you have a working solution.

First install virt-manager (this automatically will install dependencies like libvirt):

1

sudo apt-get install virt-manager

We then need to convert our whole QEMU-CLI startup script to a XML representation:

1

2

# Convert to virt-manager...

echo-ne"/usr/bin/qemu-system-x86_64 $OPTS">vm.args

Already at this point I had to play around with the parameters since virsh was not able to convert all of them correctly. Once you have created the “vm.args” file though you can create a XML file by doing:

1

virsh domxml-from-native qemu-argv vm.args > VM_SETTINGS.xml

You can then load this XML into the virt-manager by doing:

1

2

sudo cp VM_SETTINGS.xml /etc/libvirt/qemu/

virsh define /etc/libvirt/qemu/VM_SETTINGS.xml virt-manager

Further, having installied libvirt and virt-manager might require you to setup some privileges:

Apparmor settings:

1

2

3

/etc/apparmor.d/abstractions/libvirt-qemu

[...]

/dev/vfio/* rw,

then restart Apparmor with:

1

sudo /etc/init.d/apparmor restart

Qemu permissions (make sure the number for the VFIO device in the last line is correct):

1

2

3

4

5

6

7

8

9

/etc/libvirt/qemu.conf

[...]

cgroup_device_acl = [

"/dev/null", "/dev/full", "/dev/zero",

"/dev/random", "/dev/urandom",

"/dev/ptmx", "/dev/kvm", "/dev/kqemu",

"/dev/rtc","/dev/hpet", "/dev/vfio/vfio",

"/dev/vfio/1"

]

Finally, you should have an entry in your virt-manager or virsh. Nevertheless, several bugs with passing through the GPU device at this point ruined the fun for me here.

Compile QEMU 2.5.0

Make sure all QEMU dependencies are met.

Lookup the correct make parameters for your distribution, e.g. on Launchpad like here.

Remove any existing QEMU packages with your package manager.

Compile everything and run “make install”.

Attention:There is no “make uninstall”. You can make use of checkinstall to create a deb package first and install this one instead.

Share directories and files

Install samba and use the QEMU parameter:

1

2

# Share a folder with our vm.

OPTS="$OPTS -smb /directory/to/share"

Then you can access the shared directory over netshare at the location: \\10.0.2.4\qemu

Saving the machine state

Apparently there is no way like in VMware or Virtualbox to save the machine state. Executing “savevm test” in the QEMU window lead to the following error:

1

Device 'pflash1' is writable but does not support snapshots.

This error can be fixed by converting the OVMF variables file into the qcow2 format:

However, this only lead to a new error stating that the state of our passed through VFIO device couldn’t be saved:

1

2

(qemu) savevm test

State blocked by non-migratable device '0000:00:05.0/vfio-pci'

I guess the VFIO save state support is lacking. The question is whether there is any kind of feature request so far?

Update:
The Reddit user /u/SxxxX has pointed out that this seems to be a very complex issue both on the GPU side and software side of things. Nevertheless, we can hope that a clever solution / workaround to this problem can and will be found.

Automatic setup script

In addition to this guide an automatic script for guidance through the complete setup would be awesome. Please leave a comment or contact me if you are interested in setting something up.

Discussion

Other virtualization software

Using VMware for passthrough is only possible when you use VMware vSphere and special Nvidia cards like Quadro cards.
I could run VMware (unlike Virtualbox) with no problems in parallel to QEMU though.

There is unRAID which also uses QEMU and apparently can be used for passthrough purposes.

I have read some articles about XEN successfully supporting GPU passthrough, too.

Complicated solutions

During my research I have seen many things being done in a very complicated way. One examle is that you don’t need pci-stub if your kernel already supports vfio-pci. Further, keep things simple by using short parameters like:

-hda (only if you don’t need performance gains through using virtio drivers)

-cdrom

-smb

Issues

Overall the VM is really feeling nice and responsive. However, there are some issues that I encountered so far and am trying to fix.

Mouse microlags:
My mouse stutters in some games although being passed through (also without Synergy). This happens under a Windows or Linux guest.

Running QEMU parallel to Virtualbox is a nogo:
This seems to be an issue since Virtualbox and QEMU both share KVM. No solution in sight here?

Conclusion

This whole setup was definitively one of the more/most frustrating things I have done so far. However, we need more articels and scripts to decrease the global frustration level while making awesome things like this. I hope this guide helps you to reduce your stress level and to create your own setup.

In conclusion I can only say: It is absolutely worth the struggle. It can really make up for not using dual-boot and gets you a step further away from using Windows as the main OS.

Last but not least:
Many THANKS, KUDOS and SHOUTOUTS to all people who have created those awesome guides, have provided useful information or have struggled and felt the pain of setting up something like this. Finally, my biggest thanks go to my colleague Dario Weißer for doing the setup with Archlinux and helping me with many issues.

Please feel free to comment if you find any mistakes, if you want to share your own experiences or if you know about any other missing cool things.

Hey BlueBit, unfortunately I don’t understand your question. From my point of view a system account is a root account. Using a root account for a rootless setup seems to be pointless. Do you mean some kind of chroot or something similar?

However I did have problems when trying to switch the disk IO performance boost using -drive insead of -hda Windows 10 wouldn’t boot, got the bios command line instead. Did I have to use this from install?

And using:

OPTS=”$OPTS -vga none”
OPTS=”$OPTS -nographic”

Prevents windows 10 from loading, my graphics card doesn’t spring to life as it normally dose with qxl enabled. Still trying to figure that one out.

I had also tried using hugepages, but that just caused instability in my system resulting in odd system crashes.

Thanks for your feedback dugite-code. You should be able to boot your disk normally after changing to the “-drive” option. I didn’t need to install anything during the Windows installation. As mentioned in the article installing the drivers after having Windows installed should be enough (in addition to setting the -drive option of course).
I had a very similar issue with “nographic”. Strangely, changing/decreasing the memory size of the vm was enough to fix this problem. Currently, I am just tolerating a QXL window since this isn’t really bothering me and seems to have no visible impact on the vm performance. Unfortunately, I haven’t used hugepages yet so I can’t comment on that.

Great work, I just found out I’ve been working on a very similar configuration recently.

————-
First we will need a QEMU image for our VM (unless you want to forward a SATA controller or use another storage device for which I won’t go into detail at this point).
————-
This part… I passed through an entire SSD (with only the VM on it) with the virtio drivers. Sequential R/W performance is OK-ish but the 4K transfer rate is absolutely appalling (<1MB/s), about 20x lower than it should be. Do you think passing an entire SATA controller would give better performance?

I read that enabling iothreads(aka x-data-plane) for the disk could really improve performance but I'm unable to do this through libvirt (I will try the QEMU-only option)…

Thanks Wei. I don’t think that passing through the complete SATA controller will make a big difference, but you should definitely try it out. I haven’t used direct storage passthrough for my storage devices that much yet. However, as you suggested adjusting virtio settings in the non-libvirt startup script seems to be the best approach here.

Hi there. Thanks for the guide, it helped me to convert from virt-manager to a qemu script, which offers much more flexibility.

Problem: Everything is working great, passing through a Nvidia GTX960 to a Windows 10 guest, except that with a few minutes of use the video begins corrupting and tearing, eventually leading to a video card driver restart and general guest-OS Windows instability.

Has anyone encountered this? It’s like the video memory isn’t getting cleared effectively, you can see artifacts from previous screens appearing within the tears.

The Nvidia card looks good otherwise, and reports healthy in Windows (no strange error code 43 or 31 or anything), but the system is so unstable that it’s pretty unusable.

Hey seg, thanks for posting.
During testing and until now I had no problems with my graphics card whatsoever. To me this rather sounds like a hardware problem. You should try to benchmark it further on your host system and you should also maybe check the HDMI/DP cable. Further, you could try your graphics card on a Linux guest just to exclude any driver errors in Windows.

Really great guide!
I just have a question regarding CPU usage. That is, when running my guest machine under Ubuntu, the CPU usage is very high on host.
This sounds about right if I were running something very CPU stressful but this happens when all I am doing is watching a Youtube video. Does this seem right to you, or is it just me? Thank you!

Thanks Kyler. That’s definitively unusual. As you stated, if your guest machine is idling there shouldn’t be much CPU load on your host. Either you have a “weak” CPU or something else isn’t working as it should.

Now that you say you get 98-99%, I think the problem in my case may be the old and noisy Samsung 80gb sata I hdd (from my first pc, around 2006) that I used for the gpu passthrough experiment, while my native windows sits on a modern WD caviar blue.

The performance was pretty much the same even after moving the image to a modern HDD.
I’ve been thinking lately, what if I get the performance hit because of the pci-e lanes?
I have the asus Z87-K, which has one pcie 3.0 x16 slot (x16 mode) and one pcie 2.0 x16 (x4 mode). I placed the gtx750 ti in the 3.0 slot and the R7 240 in the 2.0 slot.
My Xeon 1231 v3 has only 16 lanes (1×16, 2×8, 1×8/2×4), so my gtx750 ti actually runs in x8 mode and the r7 240 in x4 mode. Could this configuration affect my performance, or my 750 ti is not powerfull enough in order to feel the difference between x16 vs x8 mode?
Also, I only have 8 gigs of ram, so I can only allocate 6 gigs max to the vm. I don’t know if this lowers the performance in cinebench gpu test and metro last light, but I’l definitely buy another 8 gb stick.

I had to install the proprietary nvidia drivers and blacklist them (don’t really know if necessary blacklisting them, I did it to be sure) in order for the vfio to grab my gpu.

I gave the VM 6gb ram and 3 cores plus hyperthreading and installed cinebench. Did the gpu benchmark and obtained 92 fps, while in the native windows 10 I obtained 110 fps, so I get 80% performance on the VM.
Keep in mind that I haven’t done yet any performance tweak, memory permissions, nothing, just installed the VM, fixed the sound and that’s it.

Thanks for the review. That’s interesting and unfortunately I can’t really explain why you have such a drastic performance drawback. With my setup and 3DMark I get about 98-99% of the points compared to the results on a native setup even without doing any optimizations.

Hey, I got it working, only problem is I made the qcow2 file only 20 gb, since this is only a test, and installed windows 10, which occupied nearly all of it. Can I extend the qcow2 file without reinstalling windows or I need to start all over again?

This guide was very useful to make sense of all of this, thanks.
Though I think it may not work on my system… I can install Windows10 in the qxl dialogue and my GTX 980 will show up in device manager named “NVIDIA GeForce GTX 980” with device status: “A driver(service)for this device has been disabled. An alternate driver may be providing this functionality. (Code 32)” does not detect any other displays, installing the driver causes windows to bluescreen with system_service_exception and system_thread_exception until I disable my GPU passthrough in the bootscript, boot in, uninstall drivers from add/remove programs, and then re-enable passthrough in boot script

Hey David,
thank you for your comment. A quick search showed that code 32 seems to be GPU passthrough unrelated. At least I couldn’t find any comments regarding this error in connection to passthrough setups. Hence, it might be the case that the issue can be found elsewhere. Please keep us updated if you make any progress on this problem.

Hi and thanks so much for this guide. It took me about 30-40 minutes to setup my Windows 7 with passthrough following your steps one by one. Everything worked fine, even sound and mouse/keyboard. This VM lets me play games like NFS Most Wanted 2012 and Rivals or Dying Light with max settings without any problems whatsoever. Also no stability problems at all.

it is great to hear that everything worked out fine on the first try! My setup also runs perfectly stable. I can strongly recommend checking out the performance tweaks section. For example using VIRTIO SCSI is a must do when using SSDs :).

Really nice how to, i spent my last days optimizing my VM, i found a problem with the virtio-scsi drive, i was not possible to install windows 10 with the last drive version, i had to use the virtio-win-0.1.110.iso

Thanks I have just recently successfully tested the virtio drivers “virtio-win-0.1.102.iso” from the Fedora page (https://fedoraproject.org/wiki/Windows_Virtio_Drivers#Direct_download as linked in the article) with Windows 10. Here I just had to select “Win 8.1” during the manual driver installation in the device manager (this was post Windows installation). In my tests the speed performance jumped from roughly 300 MB/s reading speed to about 600-700 MB/s with a qcow2 image file. Currently, the latest virtio-scsi driver is provided by “virtio-win-0.1.112.iso” though.

There is two things that may replace snapshots for some use-cases. First if you going to do something potentially harmful in VM then you can create QCOW2 image that will work as backing file (-b option). So all read going to real disk (HDD, partition or other QCOW image), but writes going into backing file. Also you can always just stop/cont VM anytime and when it’s not used you can even unload it from RAM on host just by changing CGroup rules.

About your issues with mouse. If you have lags when passing real USB device this is really weird issue, but first thing worth to try is to tell QEMU emulate USB mouse/keyboard instead of PS/2 one it’s using by default: “-device usb-kbd -device usb-mouse”.

Also except you want extra mouse buttons working you can be fine without passing actual USB device and use QEMU window instead. If you running with “-vga none” mouse capture via window is disabled, but you can run it with “-vga none -device qxl” instead so mouse capture keep working while emulated display device won’t work.

I would be very curious if this works as well. I am having the same issue in spite of the fact I am forwarding an entire USB host PCI device which all my inputs are plugged into.
The lag not enough that it causes issues with regular gaming, but on my Oculus Rift it is very noticeable.
Possibly related, the USB DAC on the Rift, and an external DAC both have noticeable static they don’t have native.

Thanks for posting lrvick.
Unfortunately, I haven’t tried out SXX’s suggestions yet (fixing a problem like this can be quite time consuming).
Please be kind enough to post again if you find a working solution to this problem.