Disable SSH key authentification for one user

For now, I use key authentification for all my server users and it worked very well. Today I've just added a git service which job is to serve git repository for everyone : internal users of my server and external.

I've created a Git user for all the git purpose (daemon, cgit, etc...)
The problem is that when people want to clone one of my repository, because of the sshd_config, the server check the ~/.ssh/authorized_key file and I just know that it's unthinkable to stock all the public key of foreign users in this file.

So, is there a way to specify that only the Git user of my server has the right to connect without any key authentification and password ?