SANS Digital Forensics and Incident Response Blog

[Author's Note: This is the 3rd in a multi-part series on the topic of "Protecting Privileged Domain Accounts". My primary goal is to help incident responders protect their privileged accounts when interacting with comprised hosts, though I also believe this information will be useful to anyone administering and defending a Windows environment.]

Update: I have added several updates to this post due to the fact that the mimikatz tool (and others) are now able to dump cleartext passwords from the Kerberos security service provider (SSP). Because of this, the idea of disabling these SSPs that store encrypted passwords is no longer feasible. More details below, with updates in italics.

Last week I discussed the fact that the only way to prevent LM hashes from loading in memory for interactive logons is to use a password of 15 or more characters. Furthermore, the fact that LM hash Rainbow Tables can crack the corresponding passwords in just a few seconds means that attackers who obtain these LM hashes will know the clear-text passwords almost instantly.

Today we'll do one better. We'll look at a method for directly obtaining a user's clear-text password from memory. No intermediate cracking steps necessary!

In this article, I'll discuss the issue of encrypted passwords in memory, the mimikatz tool which exploits the issue, and finally what we can do to protect our accounts from this "feature".

Encrypted Passwords in Memory, You Say?

Yep. It turns out that by default, interactive logons store an encrypted form of the user's password in memory. The reason for this is to provide for single sign-on (SSO) to services that do not support native network authentication protocols (i.e., Kerberos, NTLM/LM challenge-response). In these cases, Microsoft conveniently stores an encrypted version of your clear-text password in memory to authenticate you to these services.

Now in case you are wondering if there's any real difference between an encrypted password in memory and a password hash in memory, there definitely is. A hash is considered a one-way function, meaning that once data is run through the hashing algorithm, you cannot reverse the output to get the input (i.e., to get the password). Encryption, on the other hand, allows for decryption. So the fact that there is an encrypted version of your password in memory implies that there is a way to decrypt it. This is not good. Now in the case of the LM hash, we saw last week that cracking this hash is trivial, so it is effectively the same as a clear-text password?also not good. But if you set a password to be greater than 15 characters, which breaks the LM hash, then only the NT hash is available and the NT hash of a strong password is much, much more difficult to crack?as a hashed password should be!

Getting the Password

This issue is relatively new?or at least relatively unknown until recently. Just a few weeks ago, the issue got a boost in visibility with this pentestmonkey blog regarding the tool mimikatz. (I have to thank Ed Skoudis for pointing this one out to me?thanks Ed!). More recently, Tim Tomes posted an interesting article on PaulDotCom showing how to use mimikatz with Metasploit. I assume it won't be long before there's a Meterpreter module for it as well. But prior to these articles, few people knew about this issue.

It appears that the discovery was made by the author of the tool mimikatz. The author, who goes by the nickname "Gentil Kiwi" (I couldn't find his real name), is French. With the help of Google Translate, his blog articles provide a good overview of the issue and generally how his tool is able to produce the clear-text password.

His first blog on this subject is cleverly titled "Pass the pass(word)". In this blog, he discusses the fact that Windows Vista and higher feature a Security Support Provider (SSP) for Terminal Services named TsPkg, which provides single sign-on functionality to terminal servers. This feature can also be added to Windows XP SP3, as discussed here. Furthermore, this TechNet article discusses how to enable this SSO feature for specified terminal servers via GPO. However, as the author of the tool stated in his blog, and as my testing confirms, Windows will store the password in memory whether this feature is "enabled" via GPO or not. In other words, by default it's always on.

A few weeks later, the author posted an update titled "Re-pass the pass(word)", where he discusses another SSP providing similar SSO capability, named WDigest. This time it's not only in Vista and higher, but in XP and Server 2003 also!

WDigest provides for "digest access authentication", an extension to HTTP specified by RFC 2069 and later in RFC 2617. Direct access authentication improves upon basic access authentication, which sends credentials in the clear. Wdigest specifically supports RFC 2617, as well as RFC 2831, which is a more generic authentication mechanism (useful beyond HTTP) called Simple Authentication and Security Layer (SASL). The general idea here is that these specifications provide challenge-response authentication, whereby a challenge is issued and the responding party answers the challenge using a hashing routine that takes the clear-text password as an input. The Wikipedia article on digest access authentication is very interesting and highlights a number of problems, including the fact that the server using this scheme may in fact store the client's password in the clear in order to verify the answer. Nevertheless, we are concerned about the client itself, which does have to store the password in the clear in order to answer a challenge from the server.

Update: After this article was published, the author of mimikatz, Benjamin Delpy, discovered that the Kerberos SSP also stores your password in memory and his tool will now dump the password from this module as discussed in his blog "Re-re-re-pass-the-pass". After further study on Kerberos, it appears that the reason for storing the password is to allow Windows to automatically renew the Keberos Ticket Granting Ticket (TGT) without the user having to retype the password every few hours. The TGT is effectively the LM/NT hash in the Kerberos model, except that it expires and must be renewed regularly, unlike password hashes which are valid until the password is changed.

So in each of these cases, the SSP stores the clear-text password in an "encrypted" form. The encryption is implemented using the standard Win32 functions LsaProtectMemory and LsaUnprotectMemory, which "encrypts/decrypts the specified memory buffer". The tool mimikatz is able to pull the encrypted credentials from memory and simply unencrypt them with the LsaUnprotectMemory function, reporting clear-text passwords to the console.

Testing

Let's see this in action. First off, have a look at the default Security Support Providers.

Here are the default settings on Windows 7:Update: Kerberos should also be highlighted:

And here are the default settings on Windows XP:Update: Kerberos should also be highlighted:

Now let's run the tool to see what mimikatz provides.

For this testing, MSAD2-RESPONDER2 has performed an interactive logon at the console while MSAD2-RESPONDER1 is logged on via a RunAs interactive logon.

The tool works as advertised. It provides the clear-text password for interactive logons. Notice that the tool also provides the LM and NT (aka NTLM) hashes as well?though in the case of MSAD2-RESPONDER2, there is no LM hash due to the long password.

Update: The new version of mimikatz would also dump the password from the Kerberos SSP module.

So what can we do to disable this "feature"? We can simply remove these two providers from the Registry location HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa.

Great! The passwords are no longer available because the questionable security support providers are no longer available. We still have hashes available due to the interactive logon though, so that still represents a significant risk. Nevertheless, we've at least incrementally improved our security posture by not revealing clear-text passwords.

Update: Unfortunately since Kerberos is also vulnerable, we cannot remove that SSP and still have a functioning modern Windows domain.

Conclusion

As was the case for my previous two articles in this series, the definitive way to avoid this issue is to avoid interactive logons. As discussed in my first article, always use network logons instead of interactive logons when connecting to compromised machines.

On there other hand, there will obviously be one or more trusted machines for which you will need to perform an interactive logon. For those machines, I suggest you remove the TsPkg and WDigest SSPs to avoid a blatent security risk.

Update: Removing SSPs is no longer valid advice since Kerberos also stores the password. Therefore, the most effective protection is to avoid interactive logons to any untrusted hosts. Other solutions such as one-time-passwords should also be considered.

That's all for now. Next week?access tokens!

Mike Pilkington is a Sr. Information Security Analyst and Incident Responder for a Fortune 500 company in Houston, TX, as well as a SANS Mentor and Community Instructor.

Mike Pilkington

Anders Thulin

There are simpler ways than using passwords longer than 14 characters.Simply use a character that is not covered by the OEM character set that LM passwords are pased on (ASCII in US, other character sets elsewhere). AltGr-E () and AltGr-M () are very useful for this.

Mike Pilkington

Hi Anders,You bring up a great point! You are very right, there are a lot of ALT characters which will also break the LM hash, and I should have mentioned that. Unfortunately though, there are also some issues with relying on ALT characters for this purpose. The following Microsoft TechNet article discusses the use of ALT characters to break the LM hash: http://technet.microsoft.com/en-us/library/dd277300.aspx#ECAAAfter first making the point that >14 character passwords will break the LM hash, it goes on to say the following: "Second, if the password contains certain ALT characters, the system will also not be able to generate an LMHash. This latter point is tricky, because while some ALT characters significantly strengthen the password by removing the LMHash, others significantly weaken it since they are converted into a normal upper-case letter prior to storage."So it turns out that there are some ALT codes which could do the opposite of what we intend. That stinks! Table 1 in the article does list the ALT codes which will break the LM hash, which is nice. However, that's pretty difficult to keep track of and would be really hard to train English-speaking users to follow in a password policy. Though for non-English speaking users, it would probably be much easier, and in fact may not even be necessary since I would think most non-English speakers probably use Unicode in their passwords to begin with?As kind of a side note''In testing I've tried quite a few variations of passwords with ALT codes in Windows XP and haven't had a great deal of luck. For example, I was able to create a new password in XP with ALT-7 ("''"), but that's not a Unicode character and thus it creates an LM hash (verified with WCE). On the other hand, I tried including ALT-0128 ("'") in a new password, but I couldn't get XP to accept it as a password in the Change Password dialog box. On Windows 7, it worked just fine though. It also worked if I set the password on the domain controller and then logged into XP with the new password. So possibly it's user error, but from what I could tell, XP (at least my non-international version) does not have full support for changing passwords with ATL codes.To sum up, you make a great point and I was definitely too absolute in saying that the "only" way to break LM hashes is with >14 character passwords. On the other hand, ALT characters do not provide a straightforward and fail-proof method for disabling the LM hash, so I still recommend >14 character passwords for consistently breaking the LM hash.Thanks!Mike

Eric Lukens

Are there was to get access to the password that don't involve using debug privileges? If not, why not restrict that instead of reducing the functionality elsewhere? Debug privileges can be set and enforced via Group Policy for domain-joined machines.

Mike Pilkington

I got a nice email from Benjamin Delpy, the author of mimikatz. One thing he mentioned is that Microsoft does not support the removal of the offending SSPs. This is not surprising of course, and by itself does not mean that you shouldn't consider removing them anyway''"particularly on sensitive systems. However, based on the recent issues with RDP described in MS12-020, now is not the time. Benjamin's understanding is that removing TsPkg will break Network Level Authentication (NLA) in RDP, which is currently a mitigation/workaround option for MS12-020.
Thanks for that information, Benjamin!

Jeff Louisma

Mike Pilkington

Hi Jeff. Great question. I don't know for sure, but I'd be shocked if it has changed. I had a conversation with a Microsoft engineer a while ago, mostly regarding the LM hashes being stored in memory, but this issue came up too. He made no indication that either of these issues were going to change anytime soon (LM hashes or encrypted passwords). However, I don't know for sure, so if anyone has Win 8 running, look in HKLM\\System\\CurrentControlSet\\Control\\Lsa\\Security Packages and let us know if TsPkg and/or WDigest are there.

Anton Karlan

And one more thing.I`am using Hyper-V on my Windows Server 2008 R2 and after I disabled wdigest and tspkg and reboot, when I try to connect to virtual machines I get an error.An authentication error has occurred. The Local Security Authority cannot be contacted. Remote computer: .I enabled wdigest first and then reboot my computer, and still get the same error.And then I enabled tspkg, and after reboot all work fine! So, obviously, tspkg necessary for Hyper-V normal working.

Mike Pilkington

Thanks for update Anton. That's very interesting about HyperV.Unfortunately the idea of removing vulnerable SSPs is no longer a viable option due to the fact that Benjamin Delpy discovered that even Kerberos stores your password in memory. Since there is virtually no situation where you'd want to remove Kerberos, we're pretty much stuck with the fact that the password will be in memory for machines you interactively logon to. Here's a link to Benjamin's blog on Kerberos (running it through Google Translate works well): http://blog.gentilkiwi.com/securite/rerere-pass-the-pass

Bert

On windows systems the password hash can be used for authentication (google for pass the hash), so wether you have lm or ntlm, cracking the hash becomes irrelevant and the hash itself becomes directly equivalent to the plaintext password. The LM hash is actually less useful than the NTLM one as the former can't be used for anything other than cracking in the default setup, and while cracking LM may be laughably quick these days its still slower than using the NTLM hash as-is.The fact you can also extract the plaintext from memory is just an amusing side effect''