Nvidia driver, CUDA tools and libraries

Oh no, another Nvidia driver repository? Why?

This driver reflects my personal view for the way the driver should be packaged for Fedora and CentOS/RHEL. It’s somewhat different from ELRepo repositories for RHEL/CentOS and from RPMFusion packages for Fedora.

Repository installation

To install the repository on a supported Fedora distribution, run as root the following command:

Starting from Fedora 25, the Nvidia software packages are available for installation by default also in Gnome Software.

Please note that the driver will show up only if your system matches one of the PCI ID supported by the driver. Otherwise, only the other Nvidia programs (mostly for CUDA development) will show up in the software center.

What’s different?

First of all the packaging is a lot simplified; more stuff is compiled from source, smaller packages and more options. This packages try to comply as maximum to the Fedora Packaging Guidelines; which means they have debuginfo packages, default Fedora’s GCC compile time options (where possible) and standard locations for binaries, data and docs.

What follows below, is a detailed explanation of all the “differences” from the various Nvidia driver packages that I was able to spot on the web and a detailed description on how to install components, etc.

Nvidia drivers

Packaging

nvidia-settings, nvidia-persistenced, nvidia-xconfig and nvidia-modprobe are compiled from source.

All RPM filters except for GL and OpenCL libraries have been removed, so there is no weird dependency option in the SPEC file. RPM pulls in all correct requirements on its own. This is to avoid pulling in the Nvidia drivers instead of the Mesa libraries or in place of the new open source OpenCL support that’s in Fedora.

Simplified packaging with much simpler and readable SPEC file.

Dependency on libva-vdpau-driver. So in Totem, or any other libVA supported application you can benefit from VDPAU acceleration.

Sources are generated with a script and inserted individually in the various packages; so it can be easily reproduced just by changing the version and rerunning the script.

nvidia-xconfig is not required on anything that uses the modular X.org directives, as it writes too much in the configuration file (keyboards, monitors, etc.) and the required entries should be written in separate configuration files under /etc/X11/xorg.conf.d. The package is still available as it’s required to speed up some configuration like multi-monitor setups with SLI Mosaic enabled from the command line, but not installed by default.

The NVIDIA OpenGL-based Framebuffer Capture (NvFBCOpenGL) libraries (NvFBC and NvIFR) are private APIs that are only available to NVIDIA approved partners for use in remote graphics scenarios (i.e. Steam In-Home Streaming hardware encoding); so they are packaged in another small package called nvidia-driver-NvFBCOpenGL.

The nvidia-settings package now builds the external libXNVCtrl.so library that can be used to control the graphic cards through the NV-CONTROL extension. This library updates the old and obsolete one in Fedora based on drivers version 165.

Starting from version 343.13, the nvidia-settings binary is compiled with GTK3 instead of GTK2 on Fedora and RHEL/CentOS 7+.

The driver can be installed separately from the nvidia-settings utility, so if you simply want a working driver and do not care about details, your experience should be as close as possible to the one with open source drivers.

Versioning

ELRepo ships 32 bit compatibility libraries in a separate package with x86_64 as the architecture and “32bit” in the name. 32 bit libraries should be like in RPMFusion, with an i686 package installable in parallel with the x86_64 one. There are no other packages in the distribution that are built for x86_64, with “32bit” in their name that contain i686 binaries (!), so Nvidia drivers should not be an exception. So no separate “32bit.x86_64” package for 32 bit libraries also on CentOS/RHEL; just install nvidia-driver-libs.i686.

Versions are not hidden; all packages have the same driver version.

No alternatives system, only the latest version which integrates CUDA support is available. For older releases nouveau works great; and anything below a GeForce 8xxx it’s in my opinion too low end to play anything modern. And Quake 3 and Doom 3 work greatly with nouveau, so that’s not a case!

CUDA support

CUDA libraries/tools for the driver are split into subpackages. There’s no need to install all the CUDA libraries and tools on a system that has only one adapter and is used for occasional gaming or for simple office use. This can save ~120 MB worth of installed libraries. nvidia-persistenced falls in this category as it’s not needed on a normal laptop or gaming system.

Complete packaged CUDA stack has been added for all supported distributions, all the packages provide/require/obsolete the relevant packages in the Nvidia CUDA repository; so you can enable this repository along with the official Nvidia CUDA one (x86_64 systems only).

Kernel modules

Multiple choice of kernel module packages; akmod (RPMFusion) for Fedora and binary kmod (Kernel ABI whitelists) for CentOS/RHEL. In addition to this, on both distributions dkms packages are available. This way all cases and personal preferences are covered for both distributions.

Starting from Nvidia driver version 334.16, the Nvidia DDX driver for X can also rely on the nvidia-modprobe command in the system to create devices and set permissions, so the new optional package has been added.

The nvidia module has a soft dependency on the nvidia-uvm module, making sure the module is loaded when installing the nvidia-driver-cuda package, but making sure that these modules are not included in the initrd (thing that would happen with systemd configuration (module-s-load.d). UDev rules make sure the module has proper permissions.

On Fedora, the kernel modules are compressed with XZ, like all the other kernel modules.

Default configuration

Dracut options are depending on the distribution; so no more “vga=normal is an obsolete option” at boot. Each distribution gets its own specific GRUB options for booting.

96 DPI is written in the default xorg.conf config file. Why? Gnome 3 by defaults hard-codes a 96×96 DPI resolution, most of the free drivers do (intel, nouveau, etc.) as the EDID is almost never reliable (please see the excellent Adam’s Jackson post where he explains this). As an example, if you install the Nvidia drivers on a RHEL/CentOS 6 laptop where you used to have nouveau installed (96 DPI hardcoded), the fonts gets 90% of the time supersize and ugly as Gnome 2 and the Nvidia driver do not hard-code 96 DPI like Gnome 3.

Make X.org NVIDIA Files section to be loaded latest in case there are other packages providing a custom Files section.

Starting from Fedora 21, all driver X.org configuration can be managed by simply adding/removing X.org configuration snippets in /etc/X11/xorg.conf.d.

Use new OutputClass directive on Fedora 21 X.org server 1.16 (and later) to load the driver and do not rely on an edited /etc/X11/xorg.conf file. This also removes editing of the xorg.conf file from the package scriptlets. This does not hardcode the 96 DPI resolution.

Add the IgnoreABI directive by default on Fedora rawhide builds.

Kernel modesetting and Wayland support

Kernel mode setting on the nvidia-drm module has been disabled by default for various reasons. First of all, Wayland support in the drivers require a patched Wayland which has been refused upstream, and then the driver itself does not expose an FB driver for the console, so you won’t see any difference in the terminal output, you will still be limited to VGA.

The Wayland libraries are still included in the Fedora builds, as all the dependencies are there but they are not used. On CentOS/RHEL 7 packages, they are not included as this would result in missing dependencies.

Vulkan support

Vulkan is now part of Fedora, so on supported Fedora releases, the Vulkan loader and libraries can be installed and you do not need to do anything to enable support in the drivers. CentOS and Red Hat Enterprise Linux do not have Vulkan yet. I’m not sure if it’s worth installing it by default along with the drivers, though.

Distribution and Nvidia driver version support

Here is a rundown of Nvidia supported drivers and options split by distribution. Basically, CentOS/RHEL will always get a Long Lived branch release if possible, Fedora always a Short Lived branch release, and unreleased distributions will always get a Beta driver.

Optimus laptops

The driver should install and operate cleanly whether you are installing it on a system which has one or more discrete Nvidia cards or an Optimus laptop with an Intel and a Nvidia card. Nothing to do to enable or configure Optimus.

This is up to the point that when the drivers are installed, you can even turn off Optimus on or off in your system Bios (if your laptop allows that) and the only difference you should see is that there’s an additional VGA card enabled in your system (check with lspci) and that the Nvidia control panel switches between a PRIME Display, like in this picture:

And a normal RandR managed one, like in this one:

Everything else should not be different from your normal experience.

Limitations with the Nvidia driver

The limitations are the same as provided by the Nvidia driver, this means that if you are running it on an Optimus laptop, the Intel card can never power off. Which means higher power consumption, unfortunately. If you have an Optimus laptop and absolutely need the proprietary drivers, my suggestion is still to disable Optimus in the Bios.

Limitations with the OSS stack

On the contrary, if you use the OSS stack (nouveau/intel) the second card can be powered off if there’s no application running on it or display directly connected to one of the card’s outputs. That’s the best reason to use the OSS drivers at all if you you’re not doing serious gaming or 3D work:

You also got the nifty selection menu about running your game on the discrete card on Gnome, which is really cool:

It will power up the video card just before launching the process. Launching a program through that menu entry is like starting it from the command line with the DRI_PRIME variable declared. For example, the same as above would be:

As you can see, the discrete video card is turned on. For Steam, you still need to edit each of your game to run on the Nvidia card:

SLI systems

SLI is now enabled by default with the Auto profile, there’s nothing to do if you have a SLI system. If you need any different SLI option (AA, SFR, etc.), just override it in X.org configuration files.

Nouveau fallback

With the new expanded OutputClass support for X, as carried out by Hans, it’s now super easy to switch to the OSS stack if the proprietary Nvidia driver somehow does not work. No user space component is touched, as soon as the Nvidia kernel module is not loaded (check on /sys/module/nvidia), the desktop starts with the normal OSS components you get with a normal installation. Thanks to all the work done on libglvnd, the libraries loaded are the correct one for the driver you are running.

This means that the performance of the Nvidia card would be abysmal, but still you would get a nice desktop and browser to Google around for answers on how to fix it :).

Sample installation

Here is an example. Let’s assume you have a freshly installed Fedora 25 system with a recent Nvidia GPU and you want to:

As you can see, this system has dkms enabled kernel module and libraries for running 32 bit applications. The amount of data to download for the drivers is really small compared to packages that contain CUDA libraries and tools. All packages have an higher Epoch set; so they should never be upgraded on your system when you enable this repository along the RPMFusion or ELRepo ones.

Package installation

If you are booting the system in UEFI mode; as a prerequisite to installing any external module (not built into the kernel package), you have to disable UEFI Secure Boot in the system configuration. All modules contained in the kernel package are signed with keys that are generated during build and deleted when packaging. If you want to preserve Secure Boot, you need to sign the modules yourself and import the keys into your hardware module. Doing so is out of scope here; if you need a decent guide just follow Red Hat’s guide for signing kernel modules.

First of all remove all the Nvidia drivers you might have on your system due to RPMFusion, ELRepo, or the Nvidia CUDA repository. The packages should already take care of this for you, as they should be completely compatible; but better be safe than sorry. This is usually accomplished with the following root command:

yum-y remove \*nvidia\*

yum -y remove \*nvidia\*

Then, to install the Nvidia driver and its control panel utility in CentOS/RHEL with the binary kABI (Kernel ABI whitelist) module (this is the default on CentOS/RHEL), perform the following command:

yum-yinstall nvidia-driver nvidia-settings

yum -y install nvidia-driver nvidia-settings

To do the same in Fedora, perform the following command:

dnf -yinstall nvidia-driver nvidia-settings kernel-devel

dnf -y install nvidia-driver nvidia-settings kernel-devel

Requirement on kernel-devel is required as otherwise the package kernel-debug-devel is pulled in automatically in place of the normal non-debug package. There is bug opened on dnf/libsolv for this.

Specific driver installations

For both Fedora and CentOS/RHEL distributions it’s possible to install additional packages and / or variant of the basic kernel modules. This paragraph contains some examples. Make sure you have the RPMFusion repository enabled if you plan to use akmod kernel modules on Fedora or EPEL if you plan to use DKMS modules on CentOS/RHEL.

akmod kernel module variant (Fedora):

dnf -yinstall nvidia-driver kernel-devel akmod-nvidia

dnf -y install nvidia-driver kernel-devel akmod-nvidia

DKMS kernel module variant (Fedora/CentOS/RHEL):

yum/dnf -yinstall nvidia-driver kernel-devel dkms-nvidia

yum/dnf -y install nvidia-driver kernel-devel dkms-nvidia

To add 32 bit libraries on a 64 bit system (for games or applications like Steam):

yum-yinstall nvidia-driver-libs.i686

yum -y install nvidia-driver-libs.i686

Additional driver configuration to your system

To add additional configuration to your system, just create the /etc/X11/xorg.conf file if it does not exist (by default it exists only in Red Hat Enterprise Linux / CentOS 6 systems). For example:

In this example we have 2 video cards with one monitor each, so we enabled SLI, Base Mosaic to have multi monitor support on SLI and make a layout with the second GPU monitor on the right of the first one. Also, we fix the DPI to 96×96, which is the hardcoded default in Gnome and in Open Source drivers.

Configuration for headless systems

Your system might only be used for CUDA development and not require the X server to be running the driver at all, so you might want to tweak the configuration a bit to make the system load (for example) the Intel driver as the main display and just the Nvidia driver for calculation. In this case, the Intel driver should load the modesetting driver, offload the rendering to the Nvidia driver and not use any monitor attached to the X server for the Nvidia driver.

So in this case, I would change /etc/default/grub to remove the nomodeset parameter to make the Intel KMS driver to load properly, regenerate the Grub config file, reboot and use this xorg.conf:

The device file /dev/nvidia0 is normally created when loading the Nvidia driver, so if the X driver is not loaded the device file fileis not created. You can use the nvidia-modprobe command that is in the package with the same name.
It contains a SUID binary that creates the device files and set the appropriate permissions when automatic device creation is not available. It is called directly by Nvidia libraries:

This requires some testing and adjustments with specifics to your setup, but is definitely possible to use the integrated Intel card and or rely on a system without X installed to run the CUDA components.

CUDA

Packaging

Previously in the repository was included the GPU Deployment kit. This was constructed with NVML (NVIDIA Management Library) headers, docs and samples from a separate tarball. The separate tarball was using a different version number than the drivers and was packaged in the nvidia-driver-NVML and nvidia-driver-NVML-devel packages. Installing these, the gpu-deployment-kit dependency provided by the CUDA repositories was preserved. Starting from CUDA version 8, the NVML header is provided by a CUDA subpackage (cuda-nvml-devel) and no longer provided as part of the GPU Deployment kit.

Included is also the NVENC (Nvidia Encoder) header, docs and code samples. Again, this uses a different version than the drivers.

All the libraries are split into subpackages, much like in the original Nvidia CUDA repository. This allows you to install and build software relying on specific components without the need to install all the CUDA toolkit just to satisfy a library dependency. With the new packaging organization, the original cuda-devel and cuda-extra-libs will pull in all the specific subpackages giving you the same situation you are accustomed to. Also, for the same reason, static libraries have been included in each respective devel subpackage.

In addition to the libraries bundled in the CUDA toolkit, also the cuDNN library for distributed neural networks is included in the repository. See the table below for details.

This will avoid you pulling in all the libraries as before just because you need a single library. This is useful for example for programs that leverage just some part of the CUDA toolkit, like the Nvidia Performance Primitives for image and signal processing in FFmpeg, and similar things.

I’m waiting for the 390.x series to stabilize a bit, after that I will push the update for CUDA 9.1 also to RHEL 7. At the moment the 390.25 is performing well with CUDA 9.1 but it has quite a few issues. There is 390.30, but the sources for the various components have not been released: http://www.nvidia.com/download/driverResults.aspx/131164/en-us

My HTPC is a i3-2100T and I used the integrated intel gpu till now, but as I experienced stuttering in Kodi, I have added a MSI nvidia GT1030 passive. While kodi no longer stutters, the audio I get to the TV over HDMI is very quiet. I need to boost it to 150% in pavucontrol, and turn up volume on TV double of what used to be normal before. I can’t find any setting to improve the audio level. The issue is that then I need to turn up and down TV volume as I switch inputs…

I think its likely the result of different hardware. Possibly nothing else you can do though I’ve never researched if anyone came up with something else. Pushing it past 100% is all I’ve ever used when necessary but like you said you’re using a TV which has its own volume controls, so that is the alternative.

Those three choices in the end are all I know. Use a different audio device, push past 100% and have possible snap, crackle, pop or still too low volume at times, or pass it to receiver, tv, etc which has its own volume control.

Hi. Thanks for the great work. I am running a desktop with AMD 6 core CPU and Nvidia GeForce 200 card. I recently installed the driver and X Server Settings after adding the repo on my Fedora 27. However, when I opened X Server Settings it notified me that “You do not appear using Nvidia X driver. Please edit your X configuration file (just run ‘nvidia-xconfig’ as root), and restart the X server.

The trouble is, after I installed nvidia-xconfig and run the command, I cannot log into my MATE desktop anymore. only after I deleted the file generated by nvidia-xconfig could I enter the system. Any suggestion as how to solve this issue? Thanks.

After upgrade to fc27 (kernel 4.14.11-300) the nvidia kernel modules are missing in /lib/modules/4.14.11-300.fc27.x86_64/extra/

I’ve tried reinstalling the nvidia packages, but unfortunately that does not solve the problem. How do i recompile the modules manually? My system is completely up-to-date, SELinux is enabled and afaik the other prerequisites are met.

One of my home PCs keeps having it’s akmodsbuild fail for the NVIDIA drivers. Could you please give me some tips for troubleshooting this, or do you know where I should ask?

To try to resolve I’ve installed mock (creating a mock group) and created a mockbuild user since that didn’t exist and there were warnings in the log. I’ve previously forced the build as root which may have mucked up permissions, so I recursively set them all back to ‘akmods:akmods’.

When I run a build manually it works, but never automatically.
For example, when run this works:
sudo -H -u akmods bash -c ‘nice /sbin/akmodsbuild –target x86_64 –kernels 4.14.11-300.fc27.x86_64 /usr/src/akmods/nvidia-kmod.latest’

Well, I think I’ve identified my problem and I’ve also learned that I really don’t understand how this works!

The PC in question runs some basic script I’d written to apply all security updates daily. Once a week a companion script does a full update and after success reboots the PC. I think it is doing so before whatever trigger runs which calls the build of a new NVIDIA kernel module.

I’m sorry to ask such a basic question, but how does the akmod build get triggered?

My update-reboot scripts run from systemd timers. Is there any easy way I can know when an akmod has finished being built (if one was required)?

For the moment I’ve just added a 5 minute sleep. The new kernel usually builds and installs in under 3 minutes.

After last update of nvidia-settings, I can’t use nvidia-driver anymore…
nvidia-settings show an error:
“You do not appear to be using the nvidia x server”.
I’ve tried to remove and reinstall nvidia driver from your repo… any suggestions?
File: /etc/default/grub
GRUB_CMDLINE_LINUX=”rd.driver.blacklist=nouveau rhgb quiet”

I also met the freeze issue. The Nvidia driver seems to be working but after input username/password on the GDM login screen the system become completely hangs with CPU fans run at full speed. Neither can I move the mouse cursor nor switch to other TTYs.

I finally solved this issue by disabling the nouveau driver fallback mechanism with sudo systemctl status nvidia-fallback. Not sure why this would cause trouble.

Giving this a spin on F27 and have discovered an issue that I would appreciate any input on.

Running your same repo on Centos 7 with driver version 384.98 the following command worked as expected when setting clock values. After the move to F27 and the latest 387.34, the command works ONLY for GPU0. Running the command against any other GPU in the system is silently ignored.

After reverting the F27 system to 384.98, I see that the same problem exists there. The Centos7 system is running kernel v3 and F27 is on kernel v4. Not sure if there could be some differences in the newer kernel that could prevent accessing these other devices.

Well, the command does not lead to the error. The commands install the packages, and then you’re trying to use some CMake based build to build something. What are you building? You need to make sure it finds CUDA installed in the system paths and the header files at /usr/include/cuda. Is it ethminer?

Ouch, I did not know. Considering that the Tesla driver is at 384.81, you can probably rebuild the RHEL packages which are at 384.98. Or figure out if the list of supported chips is something you can just “bolt in” in some header or similar.

I suspect our problem is that we need to update the VMware drivers (VIBs) underneath the vGPUs presented to Linux virtual machines to a newer version. So I’m going to have a colleague do that and try your drivers again.

I did this on freshly installed fedora 27 ( I have 1080 ti gpu)
sudo dnf install nvidia-settings kernel-devel dkms-nvidia vulkan.i686 nvidia-driver-libs.i686
and then reboot: The system gets stuck for ever.

Do all these drivers and features work with a plain install of Fedora 27 as of Dec 4 2017? I’m interested in using 4 monitors and doing GPU programming.

It seems that maybe wayland is NOT supported directly (meaning additional links and instructions must be followed). Since wayland is the default on F27 this is an important issue. I personally don’t want to follow other links which may or may not be up to date and work or not work and be easily uninstalled or not. I appreciate the work done on this driver, but I believe it is crucial to document what exactly the driver will work with. 🙂

After the kernel update to 4.13.16-302.fc27.x86_64 this seemed to resolve itself. Appears unrelated to the driver and only manifested (for some reason) on 4.13.16-300.fc27.x86_64. Managed to repro the issue with that particular version on two machines no less!

First of all, thanks for your effort.
Second, I’m the case when Secure Boot is desired to be preserved.
After installing the nvidia package, I
– locate the module (/lib/modules/…/extra/nvidia)
– unxz nvidia.ko.xz , sign it, xz it back
– modprobe it, reboot
– Details not showing that nvidia is working now, though lspci says nvidia is it is a kernel driver for 3d controller

Hi, I just tried to install nvidia drivers on a clean F27 system, and after the reboot the system becomes unusable due to gnome-shell hogging the CPU (load goes above 9.0). Even the login screen shows this behavior. Reverting back to nouveau makes system usable again. Any idea what’s going on? I have a GTX 750 Ti. Thanks!

Uh… I don’t know exactly what happened, but I decided to reinstall nvidia driver to get some additional info, and to my surprise it is working just fine now, no CPU hogs or such. Sorry for the noise, nothing to see here, move on 😉

I don’t know what you did, but the packages have explicit dependencies on each other version/release, so it’s not even possible that you get that result without forcing things.
The 387.22 packages are all already in the repository: https://negativo17.org/repos/nvidia/fedora-25/x86_64/

I added the repo and then installed the driver and graphics control panel from software center. But when i click on the control panel nothing happens. I mean it doesn’t open and how do i switch to nvidia drivers ?

ARGH! I installed your driver on F26 (GTX 650 Ti) and had ABYSMAL graphics performance. Slow, stuttering, graphics artifacts etc. I’ve been looking for a solution for months, to finally be able to play games on Linux.

Everything was supposed to be fine: the Nvidia control panel was working, direct rendering was ‘on’, showed the proper graphic device, didn’t complain about the kernel module not being loaded.

But all games ran awfully slow, especially compared to the fact that they ran nicely in Windows!

So, I had the following installed:
akmod-nvidia
kmod-nvidia-…
nvidia-driver-libs
nvidia-settings.

You know what solved my problem after searching for weeks and finding no solution?

Installing the seemingly optional nvidia-modprobe package… now my system works great, games run wonderfully.

What does it do and if it’s optional, why does it improve performance so much?

I just updated my f26 machine to the 387.22 nvidia drivers, and they produce errors / soft+hard lockups / udev crashes on my system. Any idea what could cause that? Downgrading the drivers to 384.90 again resolves the issue.
Comparing logs between booting with old and new drivers, the only difference are those log entries concerning lockups and crashes: https://gist.github.com/decathorpe/a5a896fbea931ad1a7d2ef6d4aa42847

Hi! I’m using Fedora 27 beta and after installing this repo the nouveau driver is used instead of nvidia. I ran on Gnome, Gnome Classic and Gnome Xorg, but the same driver still loads. Here there are commands I run:

Sorry to reply late. Yes, I’m using it. Few days before there was update which caused that driver if logged to xorg mode was loaded. I installed the mutter exactly from the provided link but nouveau driver loads.

OK! So managed to get the older kernels working with the nvidia driver again (though as I type this 4.13.12-200 f26 is working a bit oddly. Figure that out later.) 4.13.12-300 still does not work and still shows nouveau.

After more testing, have been unable to get nvidia drivers working at all now with wayland or xorg. Even removed gdm, mutter, wayland and all nvidia packages regardless of how I do it. Even with the linked mutter. Nothing at all.

Its interesting to note that nvidia-settings has error:
ERROR: Unable to find display on any available system

Is it how doable to do GPU Switching for Optimus laptops? Ubuntu has made Dirty hack which allows switching between Intel/Nvidia. I suppose that “offloading” would be THE way to do it like in Windows, but far as I know that is not happening in Linux.

Would be super helpful on Laptop to decide between Performance vs PowerSave. Batter time with only intel can be like 7 hours, but on nividia its max 2 hours, but commonly just 1 hour. (just my Dell Laptop measures)

I’m using fresh installed Fedora 26 and I get the following msg when I’m trying :
dnf remove *nvidia*
Error:
Problem: The operation would result in removing the following protected packages: kernel-core
how can I fix that?