You should also make sure null passwords are disabled in PAM configuration if you use empty passwords and don't want users to be able to login with username only.
–
Juraj HrubšaJun 25 '12 at 12:08

I personnaly create my user accounts with 'adduser --disabled-password' (this is Debian specific), which makes sure there's only two way to get to this account : via SSH pubkey auth, or via local root then 'su'. It's obviously safer, although not safer than the user care for its private key (at least the weak part is where it is due, users, users....
–
zerodeuxJun 25 '12 at 20:54

You don't need to use a password on login, but I do suggest using a passphrase on the private key on the user. This means you'll have to use a forwarding agent, but then you provide security on the user's end if their machine is accessed or compromised.

Other than that potential issue, you should really be increasing security as there is no password for an attacker to even attempt to crack.