One method I've seen used at larger companies is products like Cyber-Ark that essentially take control of the passwords for the boxes and then have a set-up where you can request a password for a specific host for a time period. The application then gives you the password and changes it after the time period is up. An additional advantage to this approach is that you have a log of who had access to the server and when.

That way when a team member leaves they won't actually know any of the passwords. Obviously you need to combine this with monitoring tools to stop people adding unauthorised accounts or back-doors but if you've got enough accounts to manage it can be a good approach.