I'm not an expert in with the NVIDIA drivers so for a definitive answer you would need to ask NVIDIA. However, in this case, I am fairly confident that you want to install the 64-bit drivers since the driver's bit-ness should match the OS. The fact that your application will run in 32-bits shouldn't matter.

I understand what you say, but I will be using the installed GPU for my executable and that means I should install the 32 bit version.? I am far more interested using the GPU during execution than in graphics.

Please also note that I am using the 64 bit CentOS system, but I am still compiling in 32 bit.