How to “cure” screen flickering in VirtualBox on Ubuntu with a Windows 10 guest and 3D support enabled

TL;DR: The goal of this article is to explain why VirtualBox machines on Linux host flicker if your host has an nVidia video card and the VM has 3D support enabled. Also, for 64 bit Linux, a patched library file is provided which can be installed “on top” of an existing VirtualBox installation.

The patched library should be installed in the /usr/lib/virtualbox/ directory, as explained below.

Seems that VirtualBox has a strange problem. No, really. If you use an nVidia graphics card on your Ubuntu machine and wish to use a Windows (or, as far as I have read, basically any) virtual machine on VirtualBox for tasks that Ubuntu is not (yet) good enough and wish to enable the “experimental” 3D acceleration support, you will be unpleasantly surprised to find out that this 3D support is virtually unusable, flickering and blinking at will.

As it turns out, this problem was noticed several years ago in this forum. Basically, it’s a bug report with some folks offering a solution which involves changing a single line in the OpenGL rendering code. Unfortunately, the VirtualBox folks refused to patch this, as this breaks some functionality for an unnamed “paying customer” of theirs. This means that those affected by the problem have several options left:

a) Use a non-nVidia card, sending nVidia folks where Linus Torvalds once sent them (in this video).

b) Use the non-proprietary, free driver called nouveau with all the performance hit this implicates. Seriously, who needs 3D acceleration when it’s close to 0 (although I hope the situation will improve over time)?

c) Compile VirtualBox from source by applying the patch that can be found here.

I went through the hassle of doing that oxymoron thing of installing a VirtualBox VM on my VirtualBox to recompile a patched VirtualBox so I could patch my VirtualBox installation and use VirtualBox with 3D support and no flickering on an nVidia graphics card with the proprietary driver. 🙂

Basically, you need one file from the patched and recompiled VirtualBox, namely VBoxOGLrenderspu.so.

Should you happen to run VirtualBox 5.1.8 (STRICTLY THAT VERSION) on an x64 CPU (that is, basically, 64-bit VirtualBox version 5.1.8) running on Ubuntu, you may download the patched library in its binary, ready-to-use form here.

Then, assuming you saved it in your Downloads folder, execute these commands in terminal:

Should you need me to recompile VirtualBox with this patch for any other VirtualBox version that might come out later, just let me know. Or, you might wish to learn it yourself (note: this requires some knowledge of command line and Linux/Debian package management) by reading the official building guide here.

Well, as this is a VirtualBox issue, not a guest issue per se, there are no workarounds in the guest OS, excluding disabling 3D acceleration at all.

With nVidia drivers on the host, a race condition happens, causing buffers to be swapped (i.e. picture to be shown) before rendering is completed by the client, hence the flickering.

The only workaround is to change (comment out actually) a single line in VirtualBox source and recompile it. After recompilation is done, it is actually possible to just replace a single freshly compiled file, VBoxOGLrenderspu.so, which is located in /usr/lib/virtualbox/. No need to uninstall the “original” VirtualBox and install the self-compiled one.

For version 5.1.8 (64 bit), you may fetch a copy of the VBoxOGLrenderspu.so file from the link in the article. The bad thing is that as soon as a new version of VirtualBox comes out, VirtualBox also has to be recompiled to match the new version. I think the latest VirtualBox version now is 5.1.12, so I am going to recompile that so I can update my PC. With the “patched” .so file, VirtualBox works as a charm with 3D enabled and proprietary nVidia drivers.

Even if you get a fatal compile error, it does not mean your VBoxOGLrenderspu.so is not compiled yet. Just check out your build tree. Should you wish to use the official VirtualBox build guide, do it on a VM running Ubuntu 12.04 LTS, as that’s officially supported.

In any case, it’s always better to do such builds on a VM – this way, even if you manage to screw things up, you at least don’t screw up your main system. 🙂

Sorry, I was a bit busy with my work, so had no time to even approve the comments. ?

Thanks to showp1984 for also posting a recompiled VBoxOGLrenderspu.so file. I am doing recompilations as soon as I update VirtualBox on my PC. As I’m still running 5.1.14, haven’t done 5.1.18 recompilation yet. As to the next versions, I will try to keep in line with them. ?

@showp1984 thank you, i tried it and can confirm its workng. No more flickering issues. I noticed your .so file compared to original is much bigger (~500kb) and i had impression patch was small (1 line or something..)…

The size of the binaries depends on many factors, including whether debug information is included etc. Also, my build environment is based around Ubuntu 16.04, while the “original” VirtualBox is built around Ubuntu 12.04. Anyway, is the size that matters or the bug that is fixed? 🙂

Yeah, that’s because I didn’t strip away debug symbols. Those account for a large amount. I also built with the highest optimization setting, which generates larger files as well, because I generally don’t care about a few megs 🙂

Ive now started getting different graphics corruption, that I suspect is the same bug 🙁
Its hard to describe but I guess its in the new graphics systems for Win10 – new app systems sometimes in the Icons on the new add/remove apps system – and another example is if you install for example the Xbox Live app (not any gaming but the overview app that shows what you have been up to etc. If you for example install a game like the Solitaire app, there are graphics corruption in various places , (using the latest Win10 Pro 1703/15063.332) parallell diagonal lines etc in images. I hope its not just me ….

I can confirm this works for 5.1.8_Debian r111374. I’m running Debian 8.7 64 bit and have VirtualBox 5.1.8 installed from the jessie-backports repo. Just drop the library in place of the old one, and it works like a charm.

I’ve been so frustrated by this problem every single time I need to boot my Windows 10 VM, and I’m glad someone found a fix for it. Thanks for sharing!

I can not confirm that the 5.1.22 patch worked on my Debian jessie x64 host. VBox crashes at login screen with Windows 10 (creators update).
Maybe because it was build with Fedora? Does someone already has a patch for 5.1.22-115126, build on Debian or Ubuntu x64?

Thank you for this guys, at least i know now where to look to solve my flickering issues…

I’m sorry that it didn’t work for you, Fr4ncis.
I wonder what differences between the distros cause the crash, but it will remain unknown for now since I don’t have time to investigate.
I do, however, have access to a remote Ubuntu 16.04 (xenial) x64 machine, and I had some time while recompiling a big project on another machine, so there you go:https://drive.google.com/open?id=0B30DlLqffmOlTEhJMXRIX0ZtalE
The _deb_ tar.gz contains the patched library, this time built on Ubuntu.
I really hope that it works for you, but I didn’t even test it. Best I can do from here is to cross my fingers.

Hello Itzik,
still had the same crash at login time, but i found a solution!
It crashes only if you start the vm in “headless mode”. So starting in “normal mode” seems to work fine and also the screen flickering disappears. This workaround is okay for me.
Thank you so much for your help and time, and I hope to see further patches until VBox team is fixing that…
Thanks, Fr4ncis

I was just updated to VirtualBox 5.1.26 and was subjected to the flickering hell once again. Since there was no version of the VBoxOGLrenderspu.so for this version yet I rebuilt it myself with the patch and though I might save some hassle for those of you on Ubuntu OSes. Here is the link to the file for VBox 5.1.26 ONLY https://drive.google.com/file/d/0B-bQmaa9GDxeQmdORXg0Yl9nQWs
Thanks and enjoy!
Julian

@AcklenX, what kind of errors did you get from VirtualBox 5.1? did you at least file a bug report in hope that you will not be stuck with 5.0 (which is no longer supported by Oracle) forever?

I compiled the patched library for you on an Ubuntu 16.04 (amd64) machine, but it is NOT tested. So make backups of the two files before replacing with the ones from the archive.
You can find vboxogl_v5.0.40_deb_x64.tar.gz
in my Drive share: https://drive.google.com/open?id=0B30DlLqffmOlTEhJMXRIX0ZtalE
There are two files in this tar instead of one, since after applying the patch I noticed that these two files were recompiled. Maybe they are both affected by the patch. I don’t know.

Well, basically, you have to set up a build environment on your macOS box and compile VirtualBox yourself.
Unfortunately, we don’t have any macOS machines available and thus are not able to do this ourselves.

Big very thanks!
VirtualBox 5.2.2 is out.
version for 5.2.0 not work with 5.2.2, but 5.1.30 works with 5.2.2.
It solved my problem too.
WIN8x64 guest / Ubuntu 16.04 host
It is a pity that this is not added to the official Vbox release.

thanks for your article. I encountered the same problems with Ubuntu 17.10 (Gnome, Xorg) after adding a graphic card (Nvidia) to my Intel-Xeon-GPU. I wanted to run a virtual machines (Windows) dedicated to this card allowing and to speed up 3D.

So far I encountered the following: after a lot of general problems with 3D apps under VirtualBox in the past due to the fact, that it is only possible to give the machine 256 MB RAM (!!!) I switched for those apps to vmware-player. With it, it is no problem to give the machine enough RAM from the graphic card to use e.g. Autodesk Fusion 360. The latter never ran on VirtualBox without problems (artifacts). So I moved all those apps to Vmware-player vms and completely turned off 3D in VirtualBox. As far, as this problem is solved.

@Axel what is your message?
Saying you’re using VirtualBox 5.6 (you meant 5.26 i presume) therefore the bug is not solved??? Using some version means only that, nothing else… what’s have to do with bugs?

Forgot to mention:) I am using 5.26 and at me it no longer flickers using the above patch.

Starting with 5.2.8 the fix is already included in the official release (controlled by the new environment variable CR_RENDER_FORCE_PRESENT_MAIN_THREAD). Bottom line, simply install the new release and will be flicker free (no more patches required) took them few years…