Roberta Bragg, co-author of MCSE Training Guide (70-244): Supporting and Maintaining a Windows NT Server 4.0 Network, describes the syskey command and shows you how to protect your Windows NT 4.0 system by using it. Syskey protects the NT system by further encrypting the NT password database and can be used to prevent book without the knowledge of a special system password. Syskey is also used in Windows 2000.

From the author of

The Security Accounts Manager (SAM) stores the user passwords in a protected
database. The original Windows NT 4.0 database is protected by several
techniques:

Permissions on relevant Registry keys are set to allow only the operating
system access.

Permissions on the Registry folders and files are limited. When the
system is in operation, the SAM cannot be copied, or accessed directly except by
the system and administrators.

The passwords are obscured by a one-way function (OWF). This OWF is not
decryptable. However, anyone obtaining a copy of the database can use dictionary
and brute-force attacks in an attempt to crack or guess the passwords. In a
dictionary attack, the same OWF is applied to each word in a dictionary listing,
and then the result is compared to the obscured password. A match means the
password equals the dictionary word. A brute-force attack compares the password
OWF to an OWF of every possible combination of available characters.

Windows NT password-cracking programs have been available for several years.
(You can download an evaluation version of the famous LophtCrack tool from
http://www.@stake.com.)
Although to use them directly against the SAM requires Administrator privileges,
a backup of the SAM can be used offline by an attacker. (This is an excellent
reason to practice good physical security!) Microsoft developed Syskey to
protect the SAM from these types of attacks. Files that support Syskey as well
as the program SYSKEY.EXE were incorporated in Service Pack 3 and all later
service packs. Syskey uses a 128-bit key to encrypt the password portion of the
user database in the SAM. When it was introduced, existing cracking programs
could no longer be used to attack the password database.

Unfortunately, additional programs that can be used to provide a crackable
database to LophtCrack are now available. These tools, pwdump2 and pwdump3, must
be run by a member of the Administrators group in order to be successful.
LophtCrack 3.0 does not need these tools; LophtCrack 3.0 can be used directly
against a Syskey-protected SAM.

Nevertheless, you should use Syskey to protect the SAM for four reasons:

If you use appropriate security practices and limit administrative
accounts and require the use of strong passwords, you will mitigate the threat
of pwdump2 and Lophtcrack 3.0 being used interactively on your systems. Indeed,
if an administrative account has been compromised, there may be little need for
cracking passwords in the SAM at all because the administrative account can be
used to access any resources protected by DACLs.

You have no way of knowing what the attacker is able to deal with, nor
what weapons he has in his arsenal. Just because there are armor-piercing
bullets should not prevent me from wearing armor if I may be shot at. The
bullets fired at me may be of the regular kind, and I will survive the attack.

It is always a good idea to layer security on your system. Each problem
that you throw in an attacker's way decreases your risk of compromise. If
you make attacking your network difficult, many attackers will move on to
"lower hanging fruit."

For a nonadministrative user to use these tools against your SAM, he must
somehow obtain a copy of the SAM and use the tools offline. Good security
practices can reduce the possibilities of an attacker obtaining a copy of the
SAM. Servers, especially domain controllers, should be physically secured.
Emergency Repair Disks and backups of the Registry need to be physically
secured. The C:\WINNT\Repair directory (which holds a copy of the Registry when
the RDISK program is run to create an ERD) needs to be protected, and Registry
files can be removed from this location.

Implementation

The key used to encrypt the passwords is randomly generated by the Syskey
utility. This Password Encryption Key (PEK) is itself encrypted with a randomly
generated "System" key (Syskey) and stored in the Registry. Encrypting
the PEK prevents compromise of the encrypted passwords. If the PEK were stored
unencrypted in the Registry, it might be obtained and used to decrypt the
passwords. The Syskey must be present for the system to boot. However, now there
is a problem: how to protect the Syskey. This protection may be implemented in
one of three ways:

The Syskey is obfuscated and stored in the Registry. System can boot
without administrative action.

The Syskey is obfuscated and placed on a floppy disk that must be present
when the system reboots. The Syskey is not stored anywhere on the system. The
key is stored in a file call STARTKEY.KEY. Do not store the key on an ERD. To do
so would be to provide two items needed to attack your system in one location.
Do make copies of the disk. Without it you cannot boot your Windows NT
system.

A passphrase is entered and then used to create encrypt the Syskey. An
MD5 cryptographic hash (digest) of the Syskey is stored in the Registry. The
password must be entered during system boot to make the system usable.

In either the floppy disk choice or the password choice, the Syskey is not
stored anywhere on the system. Therefore, these choices are more secure. If the
floppy disk is lost or becomes corrupt, however, or if the password is
forgotten, the system cannot be booted.

To apply the additional security provided by using Syskey, follow the
procedure listed in Step by Step 1.

STEP BY STEP 1 Applying Syskey Protection to the SAM

Create a backup copy of the Registry prior to completing the additional
steps. Be sure to label the backup as pre-Syskey, and store it forever. The only
way to recover a Syskey-protected SAM if the Syskey is lost or corrupted is to
restore from this pre-Syskey backup of the SAM.

Check the service pack level. Apply the most current service pack.
(Service Pack 3 was the first service pack to incorporate Syskey.) Applying the
most current service pack adds the code necessary to use Syskey.

If you applied a service pack in step 2, you might want to make another
backup of the Registry. Label this one as post-SP and pre-Syskey.

From a command prompt, enter the Syskey.exe
command.

In the pop-up window, check the radio button to enable strong encryption.

Select the choice of Syskey operations by selecting the radio button
that matches your choice on the windows as shown in Figure
1.

If you have selected to enter a passphrase, do so now.

If prompted, provide a floppy disk.

Click OK.

A pop-up window will indicate success.

Reboot the system.

Make a new backup of the Registry and label it post-Syskey.

Repeat the process for each domain controller (the Syskey is not
replicated) or Windows NT 4.0 workstation that is to be protected.

The Syskey program may be used to change the Syskey option, or to generate a
new Syskey at a later time. It may not be used to bypass Syskey security. If the
key is stored on a floppy disk, the disk must be present. If the key is
passphrase-derived, the administrator much know the passphrase before being able
to rerun Syskey.