Other gotchas

Who else is using the computer?

Temperature

The GPUs are set to slowdown at 93C and shutdown at 96C. Idle temperature should be about 50C.
To see the current temperature information in full run:

nvidia-smi -q -d temperature

It should never reach that temperature. If it does that's a big whoops.

I have noticed that the first GPU (GPU 0) gets hotter quicker, presumably due to its physical location. With this information, it might be preferable to use GPU 1 more of the time.

GPU and Memory Allocation

Multiple kernels and users can also run on one GPU. This may mean you don't have enough memory at some point. It is possible to set it so that a GPU can only be accessed by one user at a time. This may be necessary in the future to ensure there is enough memory for those big jobs.

Tensorflow actually seems to claim all the memory on all the GPUs so it might be a nice gentle thing for other users for you to make tensorflow be a bit nicer.

Hiding unused GPUs from your program

One suggestion is to use

nvidia-smi

to check which GPUs are available first then hide the one that you don't need from your program, using