Double-precision routines (like gemv) in cuBLAS are only supported starting from 1.3 capable cards. By using a single-precision routine, it worked. There might be similar problems, using the library cuSPARSE.