We have two sysadmins who know the passwords to all our systems. If the "unthinkable" happened (AKA "they both went under a bus") there is currently no way for the remaining team members to obtain administrator access to the systems.

What steps or procedures should we follow to protect against this, while still keeping the systems secure?

10 Answers
10

Have them write the critical passwords down, seal them in an envelope, and store them in a safe, safety deposit box, or some other location where the physical security is high. You want to make sure only someone like the CEO or some other trustworthy person could access this, and make sure they can only access it a way that will leave physical evidence.

That would make changing the critical passwords quite complicated, and it's a good practice to change those quite regularly. What about storing all the critical passwords in a software password safe (eg keypass, etc.), and storing only the master passwords in the safety deposit box ?
–
BrannNov 4 '09 at 15:38

@Brann, I guess it depends a lot on how you define critical in your situation. At least in my mind you would only use this procedure for something that cannot easily be reset without data loss by a competent tech with physical access. Assuming you have the domain admin account fixing service accounts isn't too hard, provided you also have good documentation. For a windows network I would do use this for one account with full Domain Admin privileges. Other less critical passwords could be stored in a password database like you suggested.
–
ZoredacheNov 4 '09 at 20:24

Have a procedure in place where every time they make a significant password change (like the recovery password on a DC, like the password for the original administrator account in the forest's first domain (which should be in the Enterprise Admins group), or the like that they record the account and the password twice. And those account/password combinations are immediately sealed up in an envelope and labeled appropriately. One envelope stays on-site in a secured location (like a safe). The other is secured wherever your offsite tapes/software is. In that case, you have recovery of all critical passwords.

Everyone else mentioned the necessity of physically writing the passwords down and storing them in a safe place. I will mention something equally important: Make sure the passwords are legible!!! Nothing is worse than looking at a disaster recovery password log and not being able to tell the difference between zeros, the letter O, ones, lower case 'l's and etc because your now deceased/disenfranchised/estranged SysAdmin has the handwriting of a tenured medical professor. Even "handsome" handwriting can carry ambiguities in various symbols. If the passwords are sufficiently complex, those ambiguities can lead to very long nights of password guessing OR give you a reason to learn how to set up Cain and Abel to brute force the system with the password on it using regex terms.

For example, I would write arrows pointing to the username and say "For all you Windows admins -- This username is actually case sensitive!!" or "This is the 12th letter of the English alphabet aka the letter "l" pronounced "ell"". Yes, I'm that paranoid.

In addition to having some sort of secured password list, document the process for gaining access to the system when no password is known...physical access, forcing a password change, "maintenance mode" etc.

Whatever the method might be for a given system, it's worth having that be a known process with understanding all around of what gaining that access takes (any downtime? impact of forcing a password change on a system?).

In addition, if you are paranoid about security - split up the password. Put control of say a third of the password to the CEO and a manager, another third to the CTO and a different manager, and the last third to the CIO and a manager. That way, at least three people must agree before your precious password can be reconstituted and no one person has the ability to hose you (should they be fired or something.)

I store all password in PasswordSafe, with the program and database stored on a server with access granted to only those senior managers who should have access, as per company dictates. The contents of the database are also exported into a document, which is then printed using a font that makes it easy to distinguish those ambiguous characters and the printed copy is stored off-site in a secure location. Never rely on handwriting for this kind of thing.

We type the passwords in and print them using a console or OCR font. After verifying, the document is closed without being saved. The passwords are then sealed in an envelope and stored in the CFO's safe of all places. Somewhat regularly, the CTO and CFO open the safe, verify it's still sealed, and have the admins make new envelopes.

Separately, the CTO maintains a backup admin accounts, with the passwords stored off-site with the backup tapes.

We haven't totally gotten the hang of what to do when a password is changed, because there are services that fail to start when an admin password is changed — but it happens rarely enough that it's not too big of a deal.

The envelope trick actually did come in pretty handy when the most senior sysadmin developed a terminal medical condition and passed before a full turnover of duties was complete. Unfortunately, with him knowledge of some of the cabling routes was irretrievably lost, and we have problems with this to the day — years and years later.

The best practice is that each service should have their own account with their own password. Services should never be running under the default admin account. These service accounts should be part of your documentation so when it is time to update them you know exactly what needs to be updated.
–
ZoredacheOct 29 '09 at 5:46

As a failsafe to be able to logon with the Domain Admin account in the event that your sysadmins disappear, create a special user account and delegate the ability to reset\change passwords to this account. Give this username and password to a trusted individual such as your boss, with instructions on how and when to use it. If your boss can change the domain admin password in the event of an emergency then you're half way to regaining control of your network.

You'll probably need to create a custom MMC console as well, but this can be stored in a secure network share.

Passwords are not the problem. Any incoming replacement sysadmin will need to prove his/her skill by knowing how to recover a root enabled account on each server. Sure it requires access to the physical servers, but it is an unthinkable occurrence anyway. It happened with me that an IT professional wanted an admin login for a server that was faulty - I simply told him "you'll need to put a screen and keyboard onto the server and create a new account for yourself" - because I was not willing to give this person of unknown abilities a password.

I hate to think what mess an incompetent person can cause if given all the admin passwords.

It is more important to document the architecture of the system. Including URL's and IP addresses. And what regular procedures must be followed to maintain the network. This sort of documentation must be kept safely - but security is less important than for passwords.

Any fool can Google for how to get root/admin access. That in no way indicates their skill level as an administrator. It simply shows they can use a search engine and are able to read. Alternatively, it shows they can use one of the many CDs others have created for this express purpose.
–
John GardeniersOct 29 '09 at 9:23

Why give anyone THE root/admin password to a domain/machine? Surely you'd be creating specific admin accounts for each person that needs them, for audit-ability if nothing else. Also, if you don't trust them enough to give them an admin account, you do trust them to create one themselves unsupervised?
–
GAThrawnOct 29 '09 at 11:39

It is true that any fool can google for how to get root access. But that is exactly my point. The root password is not somthing that one should worry about for when the "unthinkable" should happen. Keeping passwords available is just not as important as keeping documentation of the network architecture and administration tasks. I do concede that finding a password is not a good test of skill. What happened in my case was that, because the IT guy asked me for a root password, and I knew he had the server with him, I became suspicious of his true abilities.
–
simplrOct 29 '09 at 12:43

@simplr. Apart from the fact that the people replacing the senior Administrators could probably put their time to a better use than hacking into systems (after all, you've got to take the workload of the two key people who have left...), getting root access by overwriting a password is not the same as retrieving the password. While the approach you're suggesting can work in a "root access" situation, what about passwords protecting SSL certificates? What about encrypted files?
–
BrannNov 4 '09 at 15:43

@Bran: I agree, passwords are vital for encrypted files and such. In my situation I have no encrypted files, but I do have a lot of custom programs/scripts that no other administrator will be aware of. Should I drop dead not even the best expert could guess what needs to be done. Our business would suffer. But other answers seemed to be careless about documentation other than your comments and 'emgee' who wrote "... knowledge of some of the cabling routes was irretrievably lost, and we have problems with this to the day — years and years later." Other answers show signs of paranoia!
–
simplrNov 4 '09 at 22:37