You can give each of the people you want to have access a public key which they can use to connect through the same SSH user. Forcing each key to only be able to run svnserve -t --tunnel-user=<username> gives each key a separate SVN identity. This is covered well in a blog post by James Gardner.