and other brilliant error messages

Tag Archives: authentication

Business users increasingly expect full LAN access while working wirelessly around the workplace. On account of the perceived weakness of WPA cryptography many network administrators will tend to offer a separate guest network over wifi, but not the full corporate LAN. I was one of those people until I got this working six months ago. This solution supports Mac, PC clients, together with iOS devices (iPhone, iPad), and I would guess Linux too since it’s based on open standards.

There is a pretty comprehensive Cisco configuration example document on this subject, but there isn’t much information on the web apart from that. There are several drawbacks to that guide:-

it deals with Windows Server 2003 IAS rather than its more current replacement: NPS, as introduced by Windows Server 2008.

it assumes that the client will be running Cisco wifi client software (presumably it pre-dates the introduction of Windows’ own Wireless Network control panel introduced with XP Service Pack 1).

The access points authenticate using PEAP which is EAP inside a TLS session, therefore a working PKI is required for creating the NPS server’s certificate. If you don’t already have it, install the Active Directory Certificate Services role to one of your Windows domain servers. As I mentioned in my LDAPS guide, that whole process is somewhat outside the scope of this blog post but do heed Microsoft’s warning:

Warning Before you install a certification authority (CA), you should be aware that you are creating or extending a public key infrastructure (PKI). Be sure to design a PKI that is appropriate for your organization. See PKI Design Brief Overview for additional information.

Windows PKI Problems

DCs should auto-enroll for their own certificates once that’s up and running. However I had huge difficulties with this. I installed the Active Directory Certificate Services role on a Windows Server 2008 R2 Domain Controller. I soon discovered that none of the other 2008 R2 DCs could auto-enroll for certificates (the Event Log reported RPC server unavailable in the failure event which was quite misleading). I could not manually enroll for certificates using the Certificates MMC snap-in either.

This held me up for a long time, and I was able to find several sources stating that the solution was to add the Domain Controllers to the group CERTSVC_DCOM_ACCESS, which didn’t work for me. In the end I found the solution in this Technet answer by Joson Zhou. Somebody in the distant past had apparently tampered the Active Directory group membership of Builtin\Users (CN=Users,CN=Builtin,DC=domain,DC=com) and had removed Interactive and Authenticated Users. It had been like this for years with no adverse impact. Since a DC has no local accounts it will use this AD group as its definition of which users have User privileges. I suppose that up until this time no service that would have been affected had been installed on a DC.
I called a business contact to verify against an external Active Directory and, sure enough, those missing groups should indeed have been members. The moment I corrected this I was able to enroll for a DC certificate. Sadly the Technet thread is locked so I wasn’t able to thank Joson, and unfortunately it seems you can’t send private messages. I don’t understand why anyone would be inclined to interfere with default Active Directory groups – searching only reveals this post. The guy asks if it’s best practice to remove Authenticated Users from that group and is pretty roundly slapped down. Perhaps people did this with NT 4.0, which my domain dates from.

I also had a recurring issue in the Event Logs where these DCs would fail to pull down the Certificate Revocation Lists from Microsoft and fail to refresh the Trusted Root certificates. Microsoft KB931125 was the only thing that fixed that. However, while troubleshooting this problem I ended up installing the Trusted Root CA update pack (from here I think), assuming I’d be able to uninstall it later if it didn’t improve things. Unfortunately it cannot be uninstalled, and it caused me another huge hold-up later on.

Configuring Network Policy Server

Once your Domain Controller has a certificate you can install the Network Policy Server role. Each access point (or only your WLC if you use LWAPP access points) will need to be on a static IP address, and have an entry in the RADIUS clients section of the NPS management MMC. When generating Pre-Shared Keys note that it seems Cisco devices do not tolerate keys containing a dollar sign, at least not in their WebUI.

The behaviour for Windows computers is that before logon the PC will authenticate and connect using its computer account (providing connectivity to the domain), then when the user logs in the wifi network is re-connected in that user context. You will therefore probably need an NPS Network Policy item for both cases (NPS -> Policies -> Network Policies) which will allow you to apply different settings to each (timeouts, etc.):

In each of the policies be sure to select PEAP as the only EAP authentication type, with EAP-MSCHAP v2 as below, selecting your server’s certificate in the drop-down.

Make sure that your Connection Request Policies are not preventing connections (NPS -> Policies -> Connection Request Policies) – I think they are disabled by default.

Access Point Configuration

I won’t bother to describe the Cisco WLC configuration, since that is basically identical to Cisco’s own guide. However, one problem I had in testing was that I removed and re-added the RADIUS server settings on the WLC and I had forgotten that you need to specify your NPS server twice – once for Authentication, and separately once again for Accounting (assuming you’re using accounting), which is not the case in the Server Manager GUI on standalone access points:

Not everyone has the budget for a WLC, or you might, like me, have combination of a WLC at one site but not at others. It’s not practical to use a single WLC for multiple sites since all LWAPP traffic is trunked back to the controller – you wouldn’t want that load on your site-to-site links. Here is a sample multiple SSID configuration for a standalone Cisco 1131AG Access Point (AIR-AP1131AG-E-K9). I’m not going to provide Cisco access point GUI screenshots, but if you want them the first part of the following guide covers that well:http://blog.laurence.id.au/2010/03/running-peap-with-cisco-aeronet-1231g.html

My example access point config below relates to the following schematic. If you’re going to apply this config to a bare unit, substitute the IPs and VLANs to match your environment and remove the lines with keys or passwords.

The Final PKI Hurdle

Each time I tried to authenticate I got an Schannel Event Log error on the NPS server of “The message received was unexpected or badly formatted.” which exactly matched the symptoms described in Microsoft KB933430, although that was only intended for Windows Server 2003. This was confusing, but according to that article:

When asking for client authentication, this server sends a list of trusted certificate authorities to the client. The client uses this list to choose a client certificate that is trusted by the server. Currently, this server trusts so many certificate authorities that the list has grown too long. This list has thus been truncated. The administrator of this machine should review the certificate authorities trusted for client authentication and remove those that do not really need to be trusted.

I had been working on this issue sporadically and I had only got as far as getting the Enterprise CA online before going on holiday. Only later did I make the connection and remember the updated Trusted Root CA pack that I had loaded on in desperation. Consulting the Certificates MMC snap-in I discovered that the server had 304 trusted root CAs instead of nine! Windows Server 2008 and 2008 R2 do have a more generous storage allowance for sending CA certificates in the PEAP handshake but clearly 304 certificates was too much. Using another server as a reference machine I manually deleted all the superfluous CA certificates and I could finally authenticate via wifi!

Non-Domain Client Configuration

When connecting from non-domain machines or iOS devices, the issuing CA for the NPS server’s certificate will not be trusted. In iOS you will be prompted to accept it manually, but the situation is more complicated in Windows 7 (I haven’t tested older OS versions). In my tests the connection would be rejected upon providing the additional user credentials. You will need to export the CA server certificate by running the following command on your CA:

The Credentials provided by the server could not be validated. We recommend that you terminate the connection and contact your administrator with the information provided in the details. You may still connect but doing so exposes you to a security risk by a possible rogue server.

I found that I couldn’t connect even if I ignored this warning. To get around this, you will need to create the wireless network manually using the following settings:

Make sure to manually select your CA which should be listed as below:

Windows AD Domain Client Configuration by Group Policy

Domain workstations can have their wireless networking configs entirely managed by Group Policy, including network preference order, auto connect options, whether to cache credentials and more. Domain members also implicitly trust the Domain CA. Be aware that Windows XP workstations are configured by a separate policy to Windows Vista and Windows 7. Both Group Policies are located at Computer Configuration -> Policies -> Windows Settings -> Security Settings -> Wireless Network (IEEE 802.11) Policies. It’s all pretty self-explanatory, and the security and encryption settings are broadly identical to those in the screenshots above, except that you do not need to specify the CA.

Remember to add your designated wifi users and computers to the AD groups you created for the NPS Network Policies!

Being able to have people log into websites using credentials they already use is a huge advantage. Hosting those websites in DMZ zones away from domain controllers means that you need to do this via LDAP, but this isn’t secure since the query results are sent in plain text. Enter LDAPS, which from the research I did online seems to be the most misunderstood technology I have used to date, with very few documented examples.

It’s a pretty simple idea – wrap LDAP in a TLS session. The LDAP client connects to the server. The server sends over its certificate, which is used to authenticate it as being the actual server the client expects. This is critical since the webserver is delegating authentication to it. A TLS session is established, and the client binds to the directory with credentials sent in plain text, but TLS provides the security against eavesdropping.

What is absolutely baffling is how many big companies are completely oblivious to getting this working correctly. I deal with one of the big players in the email scanning space which offers LDAPS authentication so customers can use their normal Active Directory credentials to check their spam. I had bought a £500 VeriSign certificate expressly for this purpose, but when this certificate expired I discovered that this company still had connectivity to the domain controller. They are clearly not even checking the chain of trust on the certificate, even though they allow it to authenticate users! It would be relatively trivial to exploit this to impersonate a particular domain controller, insert an appropriate dummy user account, and hack any of their customers who use federated authentication. Security by DNS is not security (think man-in-the-middle, or cache-poisoning).

I have also worked with a supposedly major website CMS system which cannot get this right, despite claims by the vendor that it supports LDAPS. It categorically doesn’t – LDAP on a different TCP port is still LDAP.

Concept summary

There seems to be a lot of conflicting information out there on how this works and how the certificates are involved, so this is the overview as I understand it:

The LDAPS client initiates a connection to the DC on TCP636 specifiying the DC’s FQDN (IP address or hostname only are not sufficient).

The DC checks through its Local Computer certificate store for a certificate with the same FQDN, then signs using its private key as proof of ID. The certificate is sent to the client, so the client gets the DC’s public key.

The LDAPS client checks the chain of trust on that certificate. If the CA is not a trusted public one (like VeriSign) the client system will need to have the issuing CA’s own certificate pre-loaded into its Local Computer Trusted Root Certification Authorities certificate store. Remember, the web server is not in the Active Directory domain so it won’t implicity trust that CA.

Once the client trusts that the DC is genuine, the SSL session is established and the client can bind to the directory using plain text Active Directory account credentials.

Configuring your Domain Controller for LDAPS

This knowledgebase article outlines the steps the DC goes through to select a certificate, and includes details of how to create the CSR for external certification authorites: http://support.microsoft.com/kb/321051

The information there is pretty good, and there aren’t many gotchas. If you’re aiming for third party servers to bind to your AD then you will probably want an SSL certificate from a commercial CA. Be aware that you will need a certificate with a Subject Alternate Name (to have two hostnames registered) if you use a different internal FQDN for that server. The cheap public CAs like GoDaddy could not offer this the last time I enquired (admittedly about two years ago). If the website that will be binding to your directory is your own then you won’t want to buy a certificate. If you don’t already have it, install the Active Directory Certificate Services role to one of your Windows domain servers. That whole process is somewhat outside the scope of this blog post. Heed Microsoft’s warning from their Technet LDAP over SSL wiki:

Warning Before you install a certification authority (CA), you should be aware that you are creating or extending a public key infrastructure (PKI). Be sure to design a PKI that is appropriate for your organization. See PKI Design Brief Overview for additional information.

DCs should auto-enroll for their own certificates once that’s up and running. As per that Microsoft KB article that’s all you need to do. DCs will respond to LDAPS requests providing they can find a valid SSL certificate. I did run into some issues setting up my PKI which I have documented in another post.

If you’re using your own Active Directory domain’s CA, export the CA’s own certificate by running:

certutil -ca.cert c:\temp\domain-SERVERNAME-CA.cer

Configuring the DMZ webserver

Assuming your domain controller isn’t in public DNS, add its FQDN to the hosts file on your DMZ webserver.

Verifying your LDAPS server using OpenLDAP

Since in my case I was not developing the .Net code myself, I found it useful to use OpenLDAP to demonstrate to our developers that the server and firewall were definitely working as expected. I had been unable to get the normal Windows LDAP troubleshooting tool LDP.EXE to work and it’s not particularly verbose during failures (as it turns out, I hadn’t been using the FQDN). OpenLDAP is extremely easy to install on any Linux distro with a package management system. I ended up using my Synology NAS at home. The format of the issuing CA certificate needs to be slightly different though (base64 encoded). On the CA server, use:

I tried to use this binary build of OpenLDAP for Windows but it ignores ldap.conf because the people who compiled it forgot to define the variable sysconfdir in the configure script. I was able to use the excellent Process Monitor to track a failed filesystem read to a path featuring the static entry $SYSCONFDIR$ as ldapsearch.exe is invoked. Unfortunately as such you can’t add the CA cert location to the config.