Most likely the permissions to the files which hold your keys or to the above directories are too permissive. ssh will refuze to use the keys (even if they are valid) as long as the private key is readable by other users. You should chmod you key to 600 (rw-—).
See if this is the problem by looking over the output from ssh -vvv user@host