All posts tagged routing and remote access service

Over the last few weeks, I’ve worked with numerous organizations and individuals troubleshooting connectivity and performance issues associated with Windows 10 Always On VPN, and specifically connections using the Internet Key Exchange version 2 (IKEv2) VPN protocol. An issue that appears with some regularity is when Windows 10 clients fail to connect with error 809. In this scenario, the server will accept connections without issue for a period of time and then suddenly stop accepting requests. When this happens, existing connections continue to work without issue in most cases. Frequently this occurs with Windows Server Routing and Remote Access Service (RRAS) servers configured in a clustered array behind an External Load Balancer (ELB).

Network Address Translation

It is not uncommon to use Network Address Translation (NAT) when configuring Always On VPN. In fact, for most deployments the public IP address for the VPN server resides not on the VPN server, but on an edge firewall or load balancer connected directly to the Internet. The firewall/load balancer is then configured to translate the destination address to the private IP address assigned to the VPN server in the perimeter/DMZ or the internal network. This is known a Destination NAT (DNAT). Using this configuration, the client’s original source IP address is left intact. This configuration presents no issues for Always On VPN.

Source Address Translation

When troubleshooting these issues, the common denominator seems to be the use of Full NAT, which includes translating the source address in addition to the destination. This results in VPN client requests arriving at the VPN server as appearing not to come from the client’s original IP address, but the IP address of the network device (firewall or load balancer) that is translating the request. Full NAT may be explicitly configured by an administrator, or in the case of many load balancers, configured implicitly because the load balancer is effectively proxying the connection.

Known Issues

IKEv2 VPN connections use IPsec for encryption, and by default, Windows limits the number of IPsec Security Associations (SAs) coming from a single IP address. When a NAT device is performing destination/full NAT, the VPN server sees all inbound IKEv2 VPN requests as coming from the same IP address. When this happens, clients connecting using IKEv2 may fail to connect, most commonly when the server is under moderate to heavy load.

Resolution

The way to resolve this issue is to ensure that any load balancers or NAT devices are not translating the source address but are performing destination NAT only. The following is configuration guidance for F5, Citrix ADC (formerly NetScaler), and Kemp load balancers.

F5

On the F5 BIG-IP load balancer, navigate to the Properties > Configuration page of the IKEv2 UDP 500 virtual server and choose None from the Source Address Translation drop-down list. Repeat this step for the IKEv2 UDP 4500 virtual server.

Citrix ADC

On the Citrix ADC load balancer, navigate to System > Settings > Configure Modes and check the option to Use Subnet IP.

Next, navigate to Traffic Management > Load Balancing > Service Groups and select the IKEv2 UDP 500 service group. In the Settings section click edit and select Use Client IP. Repeat these steps for the IKEv2 UDP 4500 service group.

Caveat

Making the changes above may introduce routing issues in your environment. When configuring these settings, it may be necessary to configure the VPN server’s default gateway to use the load balancer to ensure proper routing. If this is not possible, consider implementing the workaround below.

Workaround

To fully resolve this issue the above changes should be made to ensure the VPN server can see the client’s original source IP address. If that’s not possible for any reason, the following registry key can be configured to increase the number of established SAs from a single IP address. Be advised this is only a partial workaround and may not fully eliminate failed IKEv2 connections. There are other settings in Windows that can prevent multiple connections from a single IP address which are not adjustable at this time.

To implement this registry change, open an elevated PowerShell command window on the RRAS server and run the following commands. Repeat these commands on all RRAS servers in the organization.

Join me this Thursday, April 9 at 10:00AM EDT for a Remote Access Q&A session hosted by Kemp Technologies. During this free live webinar, I’ll be answering all your questions as they relate to enterprise mobility, remote access, scalability and performance, security, and much more. Topics are not limited to Kemp products at all, so feel free to join and ask me anything you like! Register now and submit your questions!

Recently I wrote about Windows 10 Always On VPN device tunnel operation and best practices, explaining its common uses cases and requirements, as well as sharing some detailed information about authentication, deployment recommendations, and best practices. I’m commonly asked if deploying Always On VPN using the device tunnel exclusively, as opposed to using it to supplement the user tunnel, is supported or recommended. I’ll address those topics in detail here.

Device Tunnel Only?

To start, yes, it is possible to deploy Windows 10 Always On VPN using only the device tunnel. In this scenario the administrator will configure full access to the network instead of limited access to domain infrastructure services and management servers.

Is It Recommended?

Generally, no. Remember, the device tunnel was designed with a specific purpose in mind, that being to provide pre-logon network connectivity to support scenarios such as logging on without cached credentials. Typically, the device tunnel is best used for its intended purpose, which is providing supplemental functionality to the user tunnel.

Deployment Considerations

The choice to implement Always On VPN using only the device tunnel is an interesting one. There are some potential advantages to this deployment model, but it is not without some serious limitations. Below I’ve listed some of the advantages and disadvantages to deploying the device tunnel alone for Windows 10 Always On VPN.

Advantages

Using the device tunnel alone does have some compelling advantages over the standard two tunnel (device tunnel/user tunnel) deployment model. Consider the following.

Single VPN Connection – Deploying the device tunnel alone means a single VPN connection to configure, deploy, and manage on the client. This also results in less concurrent connections and, importantly, less IP addresses to allocate and provision.

Reduced Infrastructure – The device tunnel is authenticated using only the device certificate. This certificate check is performed directly on the Windows Server Routing and Remote Access Service (RRAS) VPN server, eliminating the requirement to deploy Network Policy Server (NPS) servers for authentication.

User Transparency – The device tunnel does not appear in the modern Windows UI. The user will not see this connection if they click on the network icon in the notification area. In addition, they will not see the device tunnel connection in the settings app under Network & Internet > VPN. This prevents casual users from playing with the connection settings, and potentially deleting the connection entirely. It’s not that they can’t delete the device tunnel however, it’s just not as obvious.

Simplified Deployment – Deploying the device tunnel is less complicated than deploying the user tunnel. The device tunnel is provisioned once to the device and available to all users. This eliminates the complexity of having to deploy the user tunnel in each individual user’s profile.

Disadvantages

While there are some advantages to using the device tunnel by itself, this configuration is not without some serious limitations. Consider the following.

IKEv2 Only – The device tunnel uses the IKEv2 VPN protocol exclusively. It does not support SSTP. While IKEv2 is an excellent protocol in terms of security, it is commonly blocked by firewalls. This will prevent some users from accessing the network remotely depending on their location.

Limited OS Support – The device tunnel is only supported on Windows 10 Enterprise edition clients, and those clients must be joined to a domain. Arguably the device tunnel wouldn’t be necessary if the client isn’t domain joined, but some organizations have widely deployed Windows 10 Professional, which would then preclude them from being able to use the device tunnel.

Machine Certificate Authentication Only – The device tunnel is authenticated using only the certificate issued to the device. This means anyone who logs on to the device will have full access to the internal network. This may or may not be desirable, depending on individual requirements.

No Mutual Authentication – When the device tunnel is authenticated, the server performs authentication of the client, but the client does not authenticate the server. The lack of mutual authentication increases the risk of a man-in-the-middle attack.

CRL Checks Not Enforced – By default, RRAS does not perform certificate revocation checking for device tunnel connections. This means simply revoking a certificate won’t prevent the device from connecting. You’ll have to import the client’s device certificate into the Untrusted Certificates certificate store on each VPN server. Fortunately, there is a fix available to address this limitation, but it involves some additional configuration. See Always On VPN Device Tunnel and Certificate Revocation for more details.

No Support for Azure Conditional Access – Azure Conditional Access requires EAP authentication. However, the device tunnel does not use EAP but instead uses a simple device certificate check to authenticate the device.

No Support for Multifactor Authentication – As the device tunnel is authenticated by the RRAS VPN server directly and authentication requests are not sent to the NPS server, it is not possible to integrate MFA with the device tunnel.

Limited Connection Visibility – Since the device tunnel is designed for the device and not the user it does not appear in the list of active network connections in the Windows UI. There is no user-friendly connection status indicator, although the connection can be viewed using the classic network control panel applet (ncpa.cpl).

Summary

The choice to deploy Windows 10 Always On VPN using the device tunnel alone, or in conjunction with the user tunnel, is a design choice that administrators must make based on their individual requirements. Using the device tunnel alone is supported and works but has some serious drawbacks and limitations. The best experience will be found using the device tunnel as it was intended, as an optional component to provide pre-logon connectivity for an existing Always On VPN user tunnel.

One of the many advantages of using Windows Server Routing and Remote Access Service (RRAS) as the VPN server to support Windows 10 Always On VPN connections is that it includes support for the Secure Socket Tunneling Protocol (SSTP). SSTP is a TLS-based VPN protocol that is easy to configure and deploy and is very firewall friendly. This ensures consistent and reliable connectivity even behind restrictive firewalls. The Citrix Application Delivery Controller (ADC), formerly known as NetScaler, is a popular platform for load balancing Always On VPN connections. In this article I’ll describe how to configure load balancing on the Citrix ADC for RRAS VPN connections using the SSTP VPN protocol.

Load Balancing SSTP

Previously I’ve written about some of the use cases and benefits of SSTP load balancing as well as the options for offloading TLS for SSTP VPN connections. Load balancing SSTP eliminates single points of failure and enables support for multiple RRAS VPN servers to increase scalability. It is generally recommended that the Citrix ADC be configured to pass through encrypted SSTP VPN connections. However, TLS offloading can be configured to improve performance and reduce resource utilization on VPN servers, if required.

Configuration

Load balancing SSTP on the Citrix ADC is straightforward and not unlike load balancing a common HTTPS web server. Below are specific settings and parameters required to load balance SSTP using the Citrix ADC.

Note: This article is not a comprehensive configuration guide for the Citrix ADC. It assumes the administrator is familiar with basic load balancing concepts and has experience configuring the Citrix ADC.

Service Settings

The load balancing service for SSTP VPN should be configured to use TCP port 443 and the SSL_BRIDGE protocol. If TLS offload is required, TCP port 80 and the HTTP protocol can be configured. Additional configuration is required on the RRAS server when TLS offload is enabled, however. Detailed information for configuring RRAS and SSTP for TLS offload can be found here.

Virtual Server Settings

The virtual server is configured to use TCP port 443. It is recommended to use SSLSESSION persistence.

The LEASTCONNECTION load balancing method is the recommend option for load balancing method.

Service Monitoring

Using the default TCP monitor (tcp-default) is not recommended for monitoring SSTP, as a simple TCP port check does not accurately reflect the health of the SSTP service running on the RRAS server. To more precisely monitor the SSTP service status, a new custom monitor must be created and bound to the load balancing services. Follow the steps below to configure a custom SSTP VPN monitor on the Citrix ADC.

Open the Citrix ADC management console and expand Traffic Management.

Select Monitors.

Click Add.

Enter a descriptive name in the Name field.

Select HTTP form the Type drop-down list and click Select.

Adjust the Interval and Response Time-out values according to your requirements.

Enter 401 in the Response Codes field and click the “+” button.

In the Response Codes field click the “x” next to 200.

In the HTTP Request field enter HEAD /sra_{BA195980-CD49-458b-9E23-C84EE0ADCD75}/.

Check the box next to Secure (not required if TLS offload is enabled).

Once complete, bind the new service monitor to the load balancing services or service groups accordingly.

TLS Offload

It is generally recommended that TLS offload not be enabled for SSTP VPN. However, if TLS offload is desired, it is configured in much the same way as a common HTTPS web server. Specific guidance for enabling TLS offload on the Citrix ADC can be found here. Details for configuring RRAS and SSTP to support TLS offload can be found here.

Certificates

When enabling TLS offload for SSTP VPN connections it is recommended that the public SSL certificate be installed on the RRAS server, even though TLS processing will be handled on the Citrix ADC and HTTP will be used between the Citrix ADC and the RRAS server. If installing the public SSL certificate on the RRAS server is not an option, additional configuration will be required. Specifically, TLS offload for SSTP must be configured using the Enable-SSTPOffload.ps1 PowerShell script, which can be found here.

Once the script has been downloaded, open an elevated PowerShell command window and enter the following command.

Re-Encryption

When offloading TLS for SSTP VPN connections, all traffic between the Citrix ADC and the RRAS server will be sent in the clear using HTTP. In some instances, TLS offload is required only for traffic inspection, not performance gain. In this scenario the Citrix ADC will be configured to terminate and then re-encrypt connections to the RRAS server. When terminating TLS on the Citrix ADC and re-encrypting connections to the RRAS server is required, the same certificate must be used on both the Citrix ADC and the RRAS server. Using different certificates on the RRAS server and the load balancer is not supported.

When configuring Windows 10 Always On VPN using Extensible Authentication Protocol (EAP), the administrator may encounter a scenario in which the client connection fails. The event log will include an event ID 20227 from the RasClient source that includes the following error message.

“The user [domain\username] dialed a connection named [connection name] which has failed. The error code returned on failure is 858.”

RasClient Error 858

RasClient error code 858 translates to ERROR_EAP_SERVER_CERT_EXPIRED. Intuitively, this indicates that the Server Authentication certificate installed on the Network Policy Server (NPS) has expired. To resolve this issue, renew the certificate on the NPS server.

A recent update to the Kemp LoadMaster load balancer may cause failed connections for Always On VPN connections using IKEv2. SSTP VPN connections are unaffected.

Load Balancing IKEv2

When using the Kemp LoadMaster load balancer to load balance IKEv2, custom configuration is required to ensure proper operation. Specifically, the virtual service must be configured to use “port following” to ensure both the initial request on UDP port 500 and the subsequent request on UDP port 4500 are sent to the same real server. This requires the virtual service to be configured to operate at layer 7. Detailed configuration guidance for load balancing IKEv2 on the Kemp LoadMaster load balancer can be found here.

Issues with LMOS 7.2.48.0

A recent release of the Load Master Operating System (LMOS) v7.2.48.0 introduced a bug that affects UDP services configured to operate at layer 7, which includes IKEv2. This bug breaks Always On VPN connections using IKEv2, resulting in failed connections. When this occurs, the administrator may encounter an error 809 message for device tunnel or user tunnel.

Update Available

Administrators who use the Kemp LoadMaster load balancer to load balance Always On VPN IKEv2 connections and have updated to LMOS 7.2.48.0 are encouraged to update to LMOS 7.2.48.1 immediately. This latest update includes a fix that resolves broken IKEv2 load balancing for Always On VPN. Once the LoadMaster has been updated to 7.2.48.1, Always On VPN connections using IKEv2 should complete successfully.

LoadMaster Configuration

Once the virtual machine has been provisioned, open a web browser and navigate to the VM’s internal IP address on port 8443 to accept the licensing terms.

Next, log in with your Kemp ID and password to finish licensing the appliance.

Finally, log in to the appliance using the username ‘bal’ and the password provided when the virtual machine was configured.

Azure Network Security Group

A Network Security Group (NSG) is automatically configured and associated with the LoadMaster’s network interface when the appliance is created. Additional inbound security rules must be added to allow VPN client connectivity.

In the Azure management console open the properties for the LoadMaster NSG and follow the steps below to configure security rules to allow inbound VPN protocols.

SSTP

1. Click Inbound security rules.
2. Click Add.
3. Choose Any from the Source drop-down list.
4. Enter * in the Source port ranges field.
5. Select Any from the Destination drop-down list.
6. Enter 443 in the Destination port ranges field.
7. Select the TCP protocol.
8. Select the Allow action.
9. Enter a value in the Priority field.
10. Enter a name for the service in the Name field.
11. Click Add.

IKEv2

1. Click Inbound security rules.
2. Click Add.
3. Choose Any from the Source drop-down list.
4. Enter * in the Source port ranges field.
5. Select Any from the Destination drop-down list.
6. Enter 500 in the Destination port ranges field.
7. Select the UDP protocol.
8. Select the Allow action.
9. Enter a value in the Priority field.
10. Enter a name for the service in the Name field.
11. Click Add.
12. Repeat the steps below for UDP port 4500.

Load Balancing SSTP and IKEv2

Refer to the following posts for detailed, prescriptive guidance for configuring the Kemp LoadMaster load balancer for Always On VPN load balancing.

Summary

Although Windows Server RRAS is not a formally supported workload in Azure, it is still a popular and effective solution for Always On VPN deployments. The Kemp LoadMaster load balancer can be deployed quickly and easily to provide redundancy and increase scalability for larger deployments.

Previously I wrote about Always On VPN options for Microsoft Azure deployments. In that post I indicated that running Windows Server with the Routing and Remote Access Service (RRAS) role for VPN was an option to be considered, even though it is not a formally supported workload. Despite the lack of support by Microsoft, deploying RRAS in Azure works well and is quite popular. In fact, I recently published some configuration guidance for RRAS in Azure.

Load Balancing Options for RRAS

Multiple RRAS servers can be deployed in Azure to provide failover/redundancy or to increase capacity. While Windows Network Load Balancing (NLB) can be used on-premises for RRAS load balancing, NLB is not supported and doesn’t work in Azure. With that, there are several options for load balancing RRAS in Azure. They include DNS round robin, Azure Traffic Manager, the native Azure load balancer, Azure Application Gateway, or a dedicated load balancing virtual appliance.

DNS Round Robin

The easiest way to provide load balancing for RRAS in Azure is to use round robin DNS. However, using this method has some serious limitations. Simple DNS round robin can lead to connection attempts to a server that is offline. In addition, this method doesn’t accurately balance the load and often results in uneven distribution of client connections.

Azure Traffic Manager

Using Azure Traffic Manager is another alternative for load balancing RRAS in Azure. In this scenario each VPN server will have its own public IP address and FQDN for which Azure Traffic Manager will intelligently distribute traffic. Details on configuring Azure Traffic Manager for Always On VPN can be found here.

Azure Load Balancer

The native Azure load balancer can be configured to provide load balancing for RRAS in Azure. However, it has some serious limitations. Consider the following.

Azure Application Gateway

The Azure Application Gateway can be used for load balancing RRAS SSTP VPN connections where advanced capabilities such as enhanced health checks and TLS offload are required. More information about the Azure Application Gateway can be found here.

Load Balancing Appliance

Using a dedicated Application Delivery Controller (ADC), or load balancer is a very effective way to eliminate single points of failure for Always On VPN deployments hosted in Azure. ADCs provide many advanced features and capabilities to ensure full support for all RRAS VPN protocols. In addition, ADCs offer much better visibility and granular control over VPN connections. There are many solutions available as virtual appliances in the Azure marketplace that can be deployed to provide RRAS load balancing in Azure.

Summary

Deploying Windows Server RRAS in Azure for Always On VPN can be a cost-effective solution for many organizations. Although not a formally supported workload, I’ve deployed it numerous times and it works quite well. Consider using a dedicated ADC to increase scalability or provide failover and redundancy for RRAS in Azure whenever possible.

When deploying Windows 10 Always On VPN, it may be desirable to host the VPN server in Microsoft’s Azure public cloud. Recently I wrote about Always On VPN deployment options in Azure, and in that post I indicated that deploying Windows Server and the Routing and Remote Access Service (RRAS) was one of those options. Although not formally supported by Microsoft, RRAS is often deployed in Azure because it is cost-effective, easy to manage, and provides flexible scalability.

Supportability

It’s important to state once again that although it is possible to successfully deploy Windows Server with RRAS in Azure to support Always On VPN, as of this writing it is not a formally supported workload. If the administrator makes the decision to deploy RRAS in Azure, they must also accept that Microsoft may refuse to assist with troubleshooting in this specific deployment scenario.

Azure Prerequisites

The configuration of RRAS is identical to on-premises, with a few additional steps required by Azure infrastructure.

Windows Server

RRAS can be configured on any Windows Server virtual machine supported in Microsoft Azure. As with on-premises deployments, Server GUI and Core are supported. Domain-join is optional. The server can be deployed with one network interface or two.

Public IP

A public IP address must be assigned to the VPN server’s external network interface, or the internal interface if the VPN server is configured with a single network adapter. The IP address can be static or dynamic. When using a dynamic IP address, configure a CNAME record in DNS that points to the name configured for the IP address in Azure. If using a static IP address, an A host record can be configured pointing directly to the IP address.

Network Security Group

A Network Security Group (NSG) must be configured and assigned to the VPN server’s external or public-facing network interface that allows the following protocols and ports inbound.

TCP port 443 (SSTP)

UDP port 500 (IKEv2)

UDP port 4500 (IKEv2 NAT traversal)

RRAS in Azure

Below are the infrastructure requirements for supporting Windows Server RRAS VPN in Azure.

Client IP Subnet

Static IP address pool assignment must be used with RRAS. Using DHCP for VPN client IP address assignment in Azure is not supported and will not work. The IP subnet assigned to VPN clients by RRAS must be unique and not overlap with any existing Azure VNet subnets. If more than one VPN server is deployed, each server should be configured to assign a unique subnet for its clients.

IP Forwarding

IP forwarding must be enabled on the VPN server’s internal network interface. Follow the steps below to enable IP forwarding.

1. In the Azure portal, open the properties page for the internal network interface for the VPN server.
2. Click IP configurations in the navigation pane.
3. Click Enabled next to IP forwarding.
4. Click Save.

Routing

Azure must be configured to route IP traffic from VPN clients back to the VPN server. Follow the steps below to create and assign a routing table in Azure.

1. Click Create Resource.
2. Enter “Route Table” in the search field and press Enter.
3. Click Route Table.
4. Click Create.
5. Enter a descriptive name for the route table in the Name field.
6. Choose an appropriate subscription from the Subscription drop-down list.
7. Select the resource group where the VPN server(s) reside.
8. Select the best location to deploy the route table resource from the Location drop-down list.
9. If the administrator wants to have the VPN client IP subnet route information published automatically, select Enabled for Virtual network gateway route propagation.
10. Click Create.

Once complete, follow the steps below to define the route for VPN clients.

1. Open the properties page for the route table.
2. Click Routes in the navigation pane.
3. Click Add.
4. Enter a descriptive name in the Route name filed.
5. Enter the IP subnet assigned to VPN clients in the Address prefix field.
6. Select Virtual appliance from the Next hop type drop-down list.
7. Enter the IPv4 address assigned to the VPN server’s internal network interface in the Next hop address field.
8. Click Ok.
9. Repeat the steps above for each VPN server configured in Azure.

Finally, follow the steps below to assign the route table to an Azure VNet subnet.

Note: Azure only supports the assignment of one route table per subnet. If a route table is currently assigned, the VPN client subnet route can be added to an existing route table, if necessary.

Summary

Administrators have many choices when it comes to support Always On VPN connections hosted in Azure. RRAS on Windows Server can be an effective solution, assuming you can live without formal support. If having a formally supported solution is a hard requirement, consider deploying Always On VPN using the native Azure VPN gateway or another third-part Network Virtual Appliance (NVA).

The Internet Key Exchange version 2 (IKEv2) VPN protocol is the protocol of choice for Windows 10 Always On VPN deployments where the highest levels of security and assurance are required. However, as I’ve written about in the past, often the default IKEv2 security settings are less than desirable. Before using IKEv2 VPN in a production environment the administrator will need to update these security settings accordingly.

Connection Failure

When configuring Windows Server Routing and Remote Access Service (RRAS) or a third-party VPN appliance to support IKEv2 using custom security policies, the administrator may encounter a scenario in which a connection cannot be established due to a policy mismatch error. When the connection attempt fails, an error will be recorded in the Windows Application event log from the RasClient source with Event ID 20227. The error message states the following:

“The user [username] dialed a connection named [connection name] which has failed. The error code returned on failure is 13868.”

Error Code 13868

Error code 13868 translates to ERROR_IPSEC_IKE_POLICY_MATCH. Essentially this error indicates that the IKEv2 security policy on the client did not match the configuration on the server.

Server Configuration

To view the current IKEv2 IPsec policy configuration, open an elevated PowerShell command window and run the following command.

Get-VpnServerIPsecConfiguration

Client Configuration

To ensure interoperability, the VPN client must be configured to use the same IKEv2 security policy as defined on the sever. To view a VPN client’s currently configured IKEv2 security policy, open an elevated PowerShell command window and run the following command.

Note: If this PowerShell command returns no output, the VPN connection is not using a custom IKEv2 IPsec security policy.

Updating Settings

Guidance for configuring IKEv2 security policies on Windows Server RRAS and Windows 10 can be found here.

Summary

IKEv2 policy mismatch errors can be resolved easily by ensuring both the VPN server and client are configured to use the same IPsec security policies. Use the PowerShell commands in the above referenced above to validate settings and make changes when necessary.