Ethereum Mining with NVIDIA on Linux

Bitcoin has been getting a lot of attention in the media lately; primarily due to its association with recent high profile ransomware incidents. Bitcoin was first introduced back in 2009 as a digital currency and has been steadily gaining traction since, increasing almost 140% in value so far this year!

Bitcoin however, is not the only player in the game; Ethereum is another popular digital currency, currently sitting at 2nd place in total market value and is getting a lot of activity on currency exchanges.

An incentive to help keep the underlying “network” up and running (for both Ethereum and Bitcoin, amongst others) is a piece of software which can be used to validate transactions on the network and issue rewards for doing so. This is otherwise known as Mining.

My current Gaming PC seems to grab me a reasonable amount of Ethereum Tokens on 2+ year old hardware. Certainly enough to make me want to play around with actually mining some.

TLDR

Why use Linux?

The mining program I am using is currently a Fork of a Fork from the original C++ based version developed by the Ethereum Foundation. In its FAQ there seem to be known issues with NVIDIA, WDDM and Windows 10 which cause lower-than expected performance of NVIDIA cards when mining. A supposed fix is to use an older set of NVIDIA drivers along with a different build of the mining software specifically built for CUDA 6.5.

The machine I am using is my Gaming PC (which requires Windows), so I’m not prepared to jump through those hoops – running older drivers isn’t always conducive to a good gaming experience. My graphics card is NVIDIA and I’ve always had reasonable experiences using their Linux drivers so giving this a go on Linux seemed like a natural fit.

My hardware

I’ve not yet built myself a dedicated mining rig but these are the specs I’m working with:

Gigabyte H67N-USB3-B3 Motherboard

Intel Core i7-2600 CPU @ 3.40GHz

16GB RAM

NVIDIA GeForce GTX 970 – GM204 (GV-N970IXOC-4GD)

Current Driver Version: 375.66

Getting Started

In order to start mining Ethereum, you’ll first need a working Linux installation – My distro of choice is Ubuntu (16.04 LTS) Xenial.

When first installing and running Ubuntu I did have to make use of nomodeset in grub before anything would actually display on my screen – This seems to be card specific but it might help you get going! See here for more info about nomodeset.

Once you’re set up, you’ll need to install NVIDIA drivers. You can obtain Linux specific ones from NVIDIA directly and install those. As I’m using Ubuntu however, there should be some available in repos – apt-get install nvidia-375 at the time of writing this.

Establishing a baseline

Before we make any tweaks, we should establish what our baseline is on default settings. Keep in mind that this is specific to me and my hardware – your baseline may well be different depending on what equipment you use.

The version of ethminer I am using comes with a built in benchmark mode. To make these tests a fair comparison between tweaks, I’ve tested everything against block 3969748 (A random block I chose which represents the current difficulty of Ethash).

Running the benchmark for 10 trials, at 5 seconds each gives me an average hash rate of 18.31MH/s at the current difficulty. This is my baseline.

Enabling experimental features

By default, the NVIDIA drivers do not let you change any Experimental features (manual control of things) out of the box. In order to enable these, we need to add an option to our xorg.conf file. Before we make any changes here, you should back up your current config file if one exists. In my case this is /etc/X11/xorg.conf but your location may vary.

Once you know where your config file is, we need to add NVIDIA’s Coolbits option to the Devices section in the config. Using 28 in my case enables all experimental features – Even if we dont use them all 🙂

On this initial query, I can see that the default power limit for my card is 160 watts and that the maximum I can set it to is 180 watts.

Lets set it to 180w

simon@kirika:~$ sudo nvidia-smi -i 0 -pl 180
Power limit for GPU 0000:01:00.0 was set to 180.00 W from 160.00 W.
All done.

More power should mean better performance, Right? – Nope.

Performance States

I performed a benchmark at each power level my card supports, with a 5w increment between them.The graph above shows what I’d expect when we decrease our power limit below 160w; A steady decrease in hashes the lower the supplied power goes. At the top end of the scale though, I don’t actually gain any performance improvements what-so-ever.

After a bit of reading around on forums – This is because the mining application is providing a compute workload to the card, rather than something more graphical in nature. The default clock speeds for “Applications” appears lower than what the card is capable of and thus it gets capped into P2 Performance State; Hence the flatline on the graph.

To view what our current Application Clock is set to, we can take a look using nvidia-smi once again, and looking for Application Clocks and Default Application Clocks.

This output is also useful in that it shows what our current maximum clock speeds can be set to. It’s clear in the output above that our max clock differ from what Applications are permitted to use.

Armed with these values, I can proceed to change my Application Clocks to allow the GPU to clock higher when performing compute workloads. This higher performance state is known as P0 Performance State.

Lets run our benchmark again:You can just make out that the top-end of our graph is no longer a flatline which suggests we’re now using the full range of clock speeds available on the card. Albeit with little gain… But, I’m sure I can do better!

Overclocking

The final stages of tweaking are whats known as overclocking. The values I’ve chosen here are very much specific to my hardware. You might be able to get a more stable overclock, or have to use lower values – It entirely depends on your own hardware.

In order to achieve this, I’ll be using nvidia-settings which requires a working X server to be able to apply changes to our card. We tell nvidia-settings which X server to attach to with the -c :0 option and in my examples, we’re using display :0

We can also apply changes to multiple GPUs by changing the ID number in the command:

Figuring out how large an overclock your card supports will take a bit of trial, error and a lot of reboots – I started with 100MHz and increased slowly until the card became unstable/crashed. Once you know a stable number, increase it in smaller increments until you start having issues.

My card took a stable overclock at 240MHz. If I go as high as 250MHz, CUDA would sometimes fail to initialize, so make sure you test each value properly.

For the sake of being thorough, you can also apply an offset to Memory. The comments above still apply here with finding a stable value. In my case, an offset of 1,100MHz was stable, although this actually seemed to result in a frequency change of only 550MHz (which is half of 1,100). In my testing however, a Memory overclock had negligible impact on my results, but your mileage may vary so its still worth giving it a shot…

nvidia-settings -c :0 -a [gpu:0]/GPUMemoryTransferRateOffset[3]=1100

Once you know a good value for both of these, we can now look at adjusting the Application Clocks mentioned earlier to accommodate the new clock speeds for compute workloads.

Check the max values using nvidia-smi -i 0 -q -d clock like before, and set the new clocks accordingly.

With my card overclocked and stable, I did another round of benchmarks and you can clearly see the improvement compared to default settings. I mentioned previously that the memory overclock had a negligible impact on my results so I’ve omitted those from the graph.

Conclusion

I’ve managed to improve my hash rate by 18.4% – from 18.31MH/s to 21.68MH/s, by overclocking my GPU and increasing the power limit available to the card. Power consumption of the card has increased by approximately 12.5% when at 180 watts (instead of the default 160w), but it’s still worth me running at the higher power threshold.

If however, I wanted to keep power usage at similar (or lower) levels to what we started with, I can limit to 160w and still achieve 21.06MH/s with the overclock, which is still a 15% improvement over default settings! This also has the added advantage of keeping GPU temperatures under control.

For convenience, here are all the changes I’ve made in order to get this improvement (which includes some fan control options to keep on top of cooling):