and other brilliant error messages

Category Archives: Network

Simple VPN configurations can be deployed by Group Policy but EAP authentication settings cannot be configured like this, even using Windows 8.1 and Windows Server 2012 R2. Microsoft added some new PowerShell cmdlets to Windows 8.1 for configuring VPNs, but the worked examples do not appear to function for all the settings for PEAP connections, and they do not show a worked example of how you go about exporting and re-importing a connection’s XMLStream.

Defining the XML as a block within the script itself, even assigning it as data type XML does not seem to work. Not being particularly accustomed to PowerShell, the following script took a while to get right. I assigned it as a laptop startup script by GPO. If I need to modify the connection in future I can increment the version number since the script checks the local machine Registry for that, and will not install if the desired version marker is already present.

Just after doing an Active Directory migration to Windows 2008 R2 native mode I started experiencing odd DNS registration behaviour. The DNS for this domain is hosted in AD, so the migration meant that all DNS servers were now 2008 R2 (replacing 2003 R2). As a result, some 2008 R2 member servers decided to register IPv6 addresses for their automatic 6to4 tunnel interfaces in DNS which led to connectivity problems for clients (which are all Windows 7, and therefore IPv6 aware). What was really strange is that this was not consistent. Some servers were unaffected, despite being in the same subnet.

The workaround was to disable all IPv6 tunnelling over IPv4. Here are the netsh commands since they vary enough to be confusing…

Business users increasingly expect full LAN access while working wirelessly around the workplace. On account of the perceived weakness of WPA cryptography many network administrators will tend to offer a separate guest network over wifi, but not the full corporate LAN. I was one of those people until I got this working six months ago. This solution supports Mac, PC clients, together with iOS devices (iPhone, iPad), and I would guess Linux too since it’s based on open standards.

There is a pretty comprehensive Cisco configuration example document on this subject, but there isn’t much information on the web apart from that. There are several drawbacks to that guide:-

it deals with Windows Server 2003 IAS rather than its more current replacement: NPS, as introduced by Windows Server 2008.

it assumes that the client will be running Cisco wifi client software (presumably it pre-dates the introduction of Windows’ own Wireless Network control panel introduced with XP Service Pack 1).

The access points authenticate using PEAP which is EAP inside a TLS session, therefore a working PKI is required for creating the NPS server’s certificate. If you don’t already have it, install the Active Directory Certificate Services role to one of your Windows domain servers. As I mentioned in my LDAPS guide, that whole process is somewhat outside the scope of this blog post but do heed Microsoft’s warning:

Warning Before you install a certification authority (CA), you should be aware that you are creating or extending a public key infrastructure (PKI). Be sure to design a PKI that is appropriate for your organization. See PKI Design Brief Overview for additional information.

Windows PKI Problems

DCs should auto-enroll for their own certificates once that’s up and running. However I had huge difficulties with this. I installed the Active Directory Certificate Services role on a Windows Server 2008 R2 Domain Controller. I soon discovered that none of the other 2008 R2 DCs could auto-enroll for certificates (the Event Log reported RPC server unavailable in the failure event which was quite misleading). I could not manually enroll for certificates using the Certificates MMC snap-in either.

This held me up for a long time, and I was able to find several sources stating that the solution was to add the Domain Controllers to the group CERTSVC_DCOM_ACCESS, which didn’t work for me. In the end I found the solution in this Technet answer by Joson Zhou. Somebody in the distant past had apparently tampered the Active Directory group membership of Builtin\Users (CN=Users,CN=Builtin,DC=domain,DC=com) and had removed Interactive and Authenticated Users. It had been like this for years with no adverse impact. Since a DC has no local accounts it will use this AD group as its definition of which users have User privileges. I suppose that up until this time no service that would have been affected had been installed on a DC.
I called a business contact to verify against an external Active Directory and, sure enough, those missing groups should indeed have been members. The moment I corrected this I was able to enroll for a DC certificate. Sadly the Technet thread is locked so I wasn’t able to thank Joson, and unfortunately it seems you can’t send private messages. I don’t understand why anyone would be inclined to interfere with default Active Directory groups – searching only reveals this post. The guy asks if it’s best practice to remove Authenticated Users from that group and is pretty roundly slapped down. Perhaps people did this with NT 4.0, which my domain dates from.

I also had a recurring issue in the Event Logs where these DCs would fail to pull down the Certificate Revocation Lists from Microsoft and fail to refresh the Trusted Root certificates. Microsoft KB931125 was the only thing that fixed that. However, while troubleshooting this problem I ended up installing the Trusted Root CA update pack (from here I think), assuming I’d be able to uninstall it later if it didn’t improve things. Unfortunately it cannot be uninstalled, and it caused me another huge hold-up later on.

Configuring Network Policy Server

Once your Domain Controller has a certificate you can install the Network Policy Server role. Each access point (or only your WLC if you use LWAPP access points) will need to be on a static IP address, and have an entry in the RADIUS clients section of the NPS management MMC. When generating Pre-Shared Keys note that it seems Cisco devices do not tolerate keys containing a dollar sign, at least not in their WebUI.

The behaviour for Windows computers is that before logon the PC will authenticate and connect using its computer account (providing connectivity to the domain), then when the user logs in the wifi network is re-connected in that user context. You will therefore probably need an NPS Network Policy item for both cases (NPS -> Policies -> Network Policies) which will allow you to apply different settings to each (timeouts, etc.):

In each of the policies be sure to select PEAP as the only EAP authentication type, with EAP-MSCHAP v2 as below, selecting your server’s certificate in the drop-down.

Make sure that your Connection Request Policies are not preventing connections (NPS -> Policies -> Connection Request Policies) – I think they are disabled by default.

Access Point Configuration

I won’t bother to describe the Cisco WLC configuration, since that is basically identical to Cisco’s own guide. However, one problem I had in testing was that I removed and re-added the RADIUS server settings on the WLC and I had forgotten that you need to specify your NPS server twice – once for Authentication, and separately once again for Accounting (assuming you’re using accounting), which is not the case in the Server Manager GUI on standalone access points:

Not everyone has the budget for a WLC, or you might, like me, have combination of a WLC at one site but not at others. It’s not practical to use a single WLC for multiple sites since all LWAPP traffic is trunked back to the controller – you wouldn’t want that load on your site-to-site links. Here is a sample multiple SSID configuration for a standalone Cisco 1131AG Access Point (AIR-AP1131AG-E-K9). I’m not going to provide Cisco access point GUI screenshots, but if you want them the first part of the following guide covers that well:http://blog.laurence.id.au/2010/03/running-peap-with-cisco-aeronet-1231g.html

My example access point config below relates to the following schematic. If you’re going to apply this config to a bare unit, substitute the IPs and VLANs to match your environment and remove the lines with keys or passwords.

The Final PKI Hurdle

Each time I tried to authenticate I got an Schannel Event Log error on the NPS server of “The message received was unexpected or badly formatted.” which exactly matched the symptoms described in Microsoft KB933430, although that was only intended for Windows Server 2003. This was confusing, but according to that article:

When asking for client authentication, this server sends a list of trusted certificate authorities to the client. The client uses this list to choose a client certificate that is trusted by the server. Currently, this server trusts so many certificate authorities that the list has grown too long. This list has thus been truncated. The administrator of this machine should review the certificate authorities trusted for client authentication and remove those that do not really need to be trusted.

I had been working on this issue sporadically and I had only got as far as getting the Enterprise CA online before going on holiday. Only later did I make the connection and remember the updated Trusted Root CA pack that I had loaded on in desperation. Consulting the Certificates MMC snap-in I discovered that the server had 304 trusted root CAs instead of nine! Windows Server 2008 and 2008 R2 do have a more generous storage allowance for sending CA certificates in the PEAP handshake but clearly 304 certificates was too much. Using another server as a reference machine I manually deleted all the superfluous CA certificates and I could finally authenticate via wifi!

Non-Domain Client Configuration

When connecting from non-domain machines or iOS devices, the issuing CA for the NPS server’s certificate will not be trusted. In iOS you will be prompted to accept it manually, but the situation is more complicated in Windows 7 (I haven’t tested older OS versions). In my tests the connection would be rejected upon providing the additional user credentials. You will need to export the CA server certificate by running the following command on your CA:

The Credentials provided by the server could not be validated. We recommend that you terminate the connection and contact your administrator with the information provided in the details. You may still connect but doing so exposes you to a security risk by a possible rogue server.

I found that I couldn’t connect even if I ignored this warning. To get around this, you will need to create the wireless network manually using the following settings:

Make sure to manually select your CA which should be listed as below:

Windows AD Domain Client Configuration by Group Policy

Domain workstations can have their wireless networking configs entirely managed by Group Policy, including network preference order, auto connect options, whether to cache credentials and more. Domain members also implicitly trust the Domain CA. Be aware that Windows XP workstations are configured by a separate policy to Windows Vista and Windows 7. Both Group Policies are located at Computer Configuration -> Policies -> Windows Settings -> Security Settings -> Wireless Network (IEEE 802.11) Policies. It’s all pretty self-explanatory, and the security and encryption settings are broadly identical to those in the screenshots above, except that you do not need to specify the CA.

Remember to add your designated wifi users and computers to the AD groups you created for the NPS Network Policies!

I spent quite a while experimenting with L2TP over IPsec with my iPad 2, and surprisingly found no useful guides as to how to configure it. Judging by what I could find online, most people simply give up and use PPTP instead which has significant security vulnerabilities. Here’s a concise comparison of PPTP versus L2TP/IPsec which describes that weakness:http://www.ivpn.net/pptp-vs-l2tp-vs-openvpn.php

I had considered using Apple’s support for Cisco IPsec but that would have meant exposing the core switch where I work. It’s old enough to make that a bad idea. The Juniper Netscreen firewall only supports L2TP with certificates and not Pre-Shared Key so that was also ruled out. This post will outline how to configure Windows Server 2008 R2’s NPS/RRAS role to host L2TP/IPsec connections which will allow iPads and iPhones to connect securely into your Windows infrastructure without the need for additional client software.

Firstly, it’s likely that your NPS/RRAS server is behind a perimeter firewall. If this is the case you’ll need to grant IPsec traffic access from the public internet. Using details from this Technet post I created the following custom service object on the Netscreen firewall, and allowed it inbound to the RRAS server (IP protocols 50 and 51, UDP 500 and 4500). For initial testing though you should probably create a rule to allow all traffic to and from your test client.

I am going to assume a knowledge of both NPS and RRAS. For more information on those, other guides exist. As far as I have been able to discover, it seems that the iPad only supports Pre-Shared Key authentication for the IPsec tunnel, rather than certificates-based. The VPN connection settings GUI in Mac OS 10.6 for instance will allow either method, but not in iOS. It may be possible to force your way around this with the iPhone Configuration Utility (designed for applying corporate settings to iOS) but information is pretty scant. I did find a long forum thread about certificate auto-enrollment, and a Microsoft Directory Services team blog post, but I suspect they may relate more to 802.1x:https://discussions.apple.com/message/10402090http://blogs.technet.com/b/askds/archive/2010/11/22/ipad-iphone-certificate-issuance.aspx

The L2TP/IPsec Pre-Shared Key is configured by right-clicking on the top level of Routing and Remote Access in Server Manager -> Properties -> Security tab:

It’s useful to keep your VPN clients on a different subnet to your servers, however multihoming with several NICs can cause problems, particularly if your RRAS server is also a Domain Controller. You can define a subnet for this purpose in the IPv4 tab here, but you will need to remember to add a static route entry on your router pointing traffic for this subnet to the RRAS server.

In Server Manager -> NPS -> Policies -> Network Policies create a policy with the following settings, making sure to set the encryption settings. As this Microsoft KB article makes clear, these options actually ensure that IPsec gets used, with the different grades here representing different algorithm proposal combinations. The iPad supports the maximum encryption setting.

Lastly, the Mac OS X and iOS VPN client configuration is pretty self-explanatory. Make sure to use the Pre-Shared Key that you defined on the RRAS server (referred to here as Secret):

I would at this point like to thoroughly recommend iTap RDP as being the best iOS Remote Desktop client I have seen. It has NLA authentication support, a universal iPad/iPhone binary, and by far the most intuitive controls which really puts it ahead of the competition.

UPDATE – I was hoping to use this VPN configuration for all clients, but it seems that Mac OS clients cannot connect. Mac OS apparently didn’t use the standard L2TP UDP port 1701. Someone compiled a fix for Snow Leopard but I could not get it to work. It’s possible that this is all out of date information though.

UPDATE 2 – I did some more troubleshooting from home and discovered that when a tunnel is initiated from a second device on my home network while another tunnel is already up, all further connection attempts then fail for a long while, even when the RRAS server is rebooted. This would suggest that the Netscreen firewall at my work still considers the original session open, and thus it will eventually timeout after 30 minutes. This behaviour had disrupted my Mac OS X test results. Using verbose logging on the Mac and looking at the NPS log I could see that Mac OS X 10.6.8 VPN client does not accept the 128bit encryption setting. Permitting 56bit encryption allows Macs to connect, but perhaps older versions of Mac OS could have difficulties. I have updated the policy settings screenshot above.

UPDATE 3 – I realised that although NATed clients could connect, clients with public addresses could not. I have amended the destination ports for IP protocols 50 and 51 in the firewall IPsec definition screenshot (it had defaulted to 0-0 rather than 0-65535 for some reason). I have verified that this VPN works for Windows XP clients, Windows 7, Mac OS X 10.6, and Mac OS X 10.5, as well as iPhones (mine’s on iOS 3.1.3) and iPads. Once connected to the RRAS server you cannot interact with that server directly, so make sure that the RRAS server’s own DNS settings do not refer to itself as a primary (assuming it’s also a DNS server) – these DNS entries will be inherited by all VPN clients.

Included in the SSH specification is the hugely overlooked ability to tunnel traffic. Looking on the Synology forums and the Web in general I saw plenty of articles and blog posts on how to enable port forwarding on your router for all the various services these appliances can provide. Many of these services transmit their data in non-encrypted form, with passwords being sent in clear text. However, there is a better way – one that can allow you secure remote access to absolutely anything on your home LAN. I had used this method to connect to my hacked TiVo many years ago because its web interface didn’t have the option of SSL. I did find some Synology-specific info about it, but I suspect it was from much earlier DSM versions which perhaps didn’t have a decent SSH daemon, and so called for additional packages to be installed. I was sure that there must be a simple way to tweak the config of the existing daemon. Luckily I found this blog post which seems to be quite hard to find via Google.

The concept is simple. You connect via SSH with PuTTY, but in setting up the connection you define some port mappings. So let’s say you wanted to be able to remotely use the DSM software on TCP5000, you can define a local port of say 8500 and remap that to the private IP of your Synology NAS at home on port 5000 (so 192.168.1.202:5000 in my case):

Don’t forget to click Add. Also in the options navigate to Window > Translation and set the character set to UTF-8.

Once you’re connected in PuTTY, point a browser to http://localhost:8500 and you’ll be connected to the DSM at the other end of the SSH tunnel – and securely too.

If you’re using a Mac or Linux computer the normal SSH client can be configured similarly from the terminal, e.g.:ssh root@mynasip -L 8500:192.168.1.202:5000 -L 8123:192.168.1.202:8123

One final little mod is to enable coloured output in directory listings, and displaying of the current directory path in the shell prompt. Type:

I recently had a Catalyst 6500 Supervisor 2 fail and needed a complete replacement. It was one of a redundant pair and when the replacement unit arrived it had different firmware on it. Redundant pairs should have matched firmware, so I had to update it. Just in case this ever happens again, I wanted to record this information somewhere because it’s quite a tricky process.

Manually configuring VoIP handsets is a horrible job, and of course leaves you in a complete mess if you ever make any changes to those settings. I recently subnetted a building which was one flat network and wanted to prioritise VoIP traffic. Fortunately the Ericsson and Aastra handsets can be configured by DHCP, but typically it’s something you have to pester your telecoms provider for information about. Where I work this information was not volunteered when our Ericsson MD110 PABX was IP-enabled a few years ago. The reason I’m writing this up is that newer firmwares have changed this behaviour.

The handsets will boot from their bootrom firmware and then check their software firmware against the files on the ‘software server’, updating if out of date. This software server will typically be your DNA server. The files are served by IIS and are within the default Inetpub folder structure.

The handset settings are configured from DHCP on option 43, but this presents problems in infrastructures which contain other hardware which also depends on option 43. Fortunately, Microsoft Windows DHCP server allows configurable Vendor Classes – it can respond differently to requests based on their vendor class identifier string which they declare during DHCP discover and DHCP request, thus allowing re-use of the same option numbers. However, configuring this isn’t particularly intuitive hence my writing this up.

First right-click on your parent IPv4 tree in the DHCP MMC snap-in:

Now create as follows:

I discovered using Wireshark that newer firmwares since the Aastra rebranding identify themselves with the string Aastra IP-Phone whereas older ones use Ericsson IP-Phone. You should really create one for each, just in case you plug in an older handset one day.

Again by right-clicking on the parent IPv4 item select Set Predefined Options and complete as follows:

At the root level of your IPv4 tree, select Server Options. Add a new option, but use the advanced tab:

Since Vendor Classes can only be defined on a per DHCP server basis we can’t have dedicated voice VLANs per subnet unfortunately, to reflect how you separate your workstation VLANs. Since all your VoIP handsets will end up in the same VLAN, Ericsson added support for configuring two additional VLANs (using Tags 06 & 07). These additional VLANs will only get used if the handset is unable to get a DHCP lease on the VLAN defined in the preceeding tag.

As before you will need to define one set of option 43 settings for Aastra handsets, and another for Ericsson handsets:

As the phone starts up it will request and obtain a lease on the native VLAN for the switch port it is connected to, will then read the option 43 settings, release that IP, and request a new lease on the specified VLAN.

Additional configuration such as NTP server settings, daylight saving time corrections, CoS, PC port VLAN etc. can all be configured in the model-specific config files in Inetpub on the DNA server.