Use integrated identity information to create and manage identities and control access to enterprise resources. We provide identity and access management, single sign–on (SSO), access governance, and more.

Detect and respond to all potential threats quickly and decisively. By monitoring user activities, security events, and critical systems, we provide actionable security intelligence to reduce the risk of data breach.

Get affordable, high-performance disaster recovery. We protect your workloads and help you meet or exceed RPOs and RTOs of an hour or less, with mirroring-like performance at a price point approaching tape.

When troubleshooting SSLVPN issues, you need to identify the stage where the problem appears. What we will cover in this AppNote are the main areas when problems crop up: installation, initialization, connection, and accessing applications. At each stage, we will outline the key log files and tools, as well as the troubleshooting steps to identify what the issue is.

Troubleshooting SSLVPN Server Installation Issues

Whether you are installing the SSLVPN server on the Linux Access Gateway or on a separate box, the log files from the install are in the /tmp/novell_access_manager/ directory. They start with “sslvpn_inst_*” (where * includes the date and time of the install). Any errors shows at this level need to be addressed; otherwise, everything following may have issues.

The install script is a bash script that runs the install_sslvpn() and configure_sslvpn() methods. It copies over two rpms (see below), sets up the servlet tomcat environment, imports the device, and configures the device with the parameters that were specified during the install. A sample output of the install log file is shown below. Always check to make sure no exceptions were thrown in this file.

If the device is not imported into the Admin Console after the installation, look at the following:

Messages file in the /var/log directory, to make sure that the VCC and JCC services are up and running successfully
jcc-0.log.0 in the /opt/novell/devman/jcc/logs directory of the SSLVPN server. This file contains the log entries for the server communications module related to interaction of the SSLVPN server with the Administration Console, such as imports, certificates, and configuration.

Troubleshooting SSLVPN Server Initialization Issues

Assuming the SSLVPN server was installed correctly, and a valid configuration has been assigned,

1. Make sure that all the SSLVPN specific services are running correctly.

The SSLVPN server uses the following processes (available with a ‘ps aux’ output, or by just doing ‘pgrep sockd’ and making sure a valid PID is returned).

This gives an indication as to the state of the various processes – ideally, they should all be either “registered” or “running”. The only exception to this rule is with the servlet – it may report that it is not registered. This is actually normal when no request from a browser for the SSLVPN server has been processed by the servlet. Once a request is processed by the SERVLET, the state should go to registered.

This information is also available on the Health screen for the SSLVPN server in Admin Console.

If a problem exists at this stage, there are a few log files to look at for more details. The log files to analyze depend on the component that appears to be the problem. The main SSLVPN server log files are shown below:

/var/opt/novell/tomcat4/logs/catalina.out : This file contains all the information specific to the servlet component. When the servlet loads, the Tomcat environment must be set up correctly; any errors at this level will the displayed here. The servlet, as we saw above, also interacts with the Connection Manager, and the state of this interaction will be displayed here.

A sample (truncated) output of SSLVPN entries when the servlet is being registered is shown below. Check this file to make sure that the settings match what have been configured and that there are no errors or exceptions reported. If an administrator inputs incorrect information during the SSLVPN install (e.g., an invalid IP address syntax), the install will continue, but the servlet will not register correctly – and that will be visible from this file.

/var/log/messages : This file reports details of the SSLVPN server configuration (addresses, ports, policies) at initialization time, as well as the status of all the SSLVPN server daemons. Look at this file to make sure that all the services (VCC, OpenVPN, Dante/server and Policy) report as registered. If any of the services fail to register, an error will be reported here.

/var/log/stunnel.log : This file is specific to the Kiosk mode and doesn’t log much during initialisation. The following is what should be displayed when the service is successfully loaded, and we register with the Connection Manager.

/var/log/novell-openvpn.log : This file is specific to the Enterprise mode and shows the OpenVPN server initializing. As with stunnel in Kiosk mode, it needs to register with the Connection Manager but it also needs to initialize the Tun driver required to route traffic through the tunnel encrypted. Use this file to make sure the tun0 device opens and has the correct IP addresses assigned for routing, and that the port has the correct transport (UDP/TCP) and number.

It turns out that the IP address we are trying to bind to (10.1.16.5) is an old IP address, and not the IP address we had configured in the Admin Console. The SSLVPN server configuration, stored in /etc/opt/novell/sslvpn/config.xml on the SSLVPN server itself and read in by the various daemons, was not updated with the latest configuration from the Admin Console. Simply running the ‘sslvpnc –update’ on the server forces a re-read of the data from the Admin Console and fixes the issue.

Troubleshooting SSLVPN Server Connection Issues

When the SSLVPN server is configured and a user can connect successfully, the following green light will be displayed in the SSLVPN portal page:

The next stage is to try to determine what is causing this error. Based on the authentication flows covered above for the Kiosk and Enterprise mode, you need to find out at what stage the process breaks down. The following is a list of checks to be performed at each stage:

1. Make sure that the Web server defined in the Access Gateway configuration is at the IP address and TCP port of the servlet engine. A common configuration error is to leave the Web server TCP port at the default 80, resulting in a ‘504 Gateway Timeout’ error being reported on the browser when trying to connect to the SSLVPN servlet.

2. Make sure that the SSLVPN policy is setup for the SSLVPN protected resource. This policy needs to send the following things:

– Proxy cookie in the X-SSLVPN-PROXY-SESSION-COOKIE header
– Username and password in the Authentication Header
– User roles in the X-SSLVPN-ROLE header

If the servlet fails to receive any of this data, the connection will fail. By looking at the log entries in the ‘Log Entries’ section of the portal, the error details will indicate what the problem was. In this case, I had enough information to take a LAN trace of traffic to the SSLVPN servlet and confirm that the X-SSLVPN-PROXY-SESSION-COOKIE was missing from the request.

After the SSLVPN client binaries have been sent by the SSLVPN servlet to the SSLVPN client, they need to be installed and the components initialized. This may be a common source of errors for customers. The following screenshot shows a typical error:

In the above example, a download from a previous Enterprise mode connection had failed and certain resources were not released correctly. The log file indicated that the binaries being sent down were not fully downloaded and that the novl-sslvpn-service-install.exe was in use. By simply running the C:\Program Files\Novell SSLVPN Service\uninstall.exe program, the environment was cleaned up so that a subsequent connection from this SSLVPN client to the SSLVPN server allowed all files to download successfully.

4. Make sure that the CIC policies activated on the SSLVPN server can be executed successfully.

Before the connection is made to the SSLVPN server, a number of things need to be validated on the client side. Part of these checks involve detecting the version of the client if already installed, the mode of operation (and offering the Enterprise mode to users even if they do not have root privaledges), gathering the policy info for the user, the Client Integrity Checks (CIC) to be performed, and the certificate validation.

When an error occurs at each of these stages, the corresponding error displayed in the ‘Log entries’ section of the portal will point you to where the problem is. For example, I have a CIC policy defined that requires that a specific virus-checking program needs to be running on my host before the connection can take place. During the initial communication and handshaking with the servlet, a check for that virus-checking program will be carried out on the client. If the application is not located, or is not the right version, or runs on the wrong platform, the connection will fail to come up, and the following error will be reported.

The certificate exchange occurs when the SSLVPN tunnel is brought up. This occurs at different stages, depending on the mode of operation. With Kiosk mode, the tunnel is brought up when data is sent by the application to the remote server via the tunnel. With Enterprise mode, however, the tunnel is brought up after the client has initialized.

The server certificate that the client expects is the default one. If a third-party certificate is installed and added to the SSLVPN certificate device store, the SSL handshake will fail. Again, looking at the Log entries section of the portal, or hitting the ‘Save logs’ button on the same page to create the applet.log file, the following information will be visible:

Another common example where the certificate validation fails is when the SSLVPN client and server time is out of sync. The SSLVPN client tries to validate the timestamps on the certificates. If it cannot, an error is thrown in this applet file.

6. Make sure that connectivity to the SSLVPN server is available.

When the binaries are downloaded and initialized, and the versions, policies and certificates are validated, the final stages of the connection includes exchanging data with the SSLVPN server. The SSLVPN server listens out by default on TCP 7777 (Kiosk) or UDP 7777 (Enterprise mode). Data is sent to these ports during the connection phase in Enterprise mode. In Kiosk mode, the data goes to the port only when a user tries to access a remote application through the SSLVPN tunnel. If there are any intermediate devices such as firewalls or Network Address Translators that can intercept, redirect and/or drop requests, the outgoing requests from the SSLVPN client may not reach the server.

The trace below shows the final stages of a connection request to an Enterprise mode server where the UDP requests to port 7777 are being filtered by an intermediate firewall. The end result is that the SSLVPN client never gets any responses to its request, and the connection will fail.

By using ‘tcpdump -i any -s 0 -w ssslvpncon.cap’ you can take a trace of the traffic coming into the SSLVPN server. You need to make sure that all SSLVPN client requests to the SSLVPN server are visible and getting a response. If no requests are visible, take an equivalent trace on the client and make sure they are being sent. If they are, check whether firewalls or NATs are dropping the request or redirecting it to the wrong destination.

Troubleshooting SSLVPN server Application issues

Before looking at troubleshooting application level issues, let’s look at the tools and log files that you can use to troubleshoot the application layer:

Troubleshooting Tools

a) TCPDUMP, Wireshark: These tools (available on Windows and Linux) allow an administrator to decode the traffic in and out on the SSLVPN server. With these tools, you can confirm whether the encrypted data arrived at the SSLVPN server and whether this server proxied or forwarded the request to the back end application. Note that TCPDUMP can trace packets on the loopback interface so SOCKS data is also available for checking. Once the request goes to the application server, there may be enough information to decode the request and response to make sure it is valid – understanding the application data is outside the scope of this document. TCPDUMP is the main tool required for troubleshooting application layer problems.

b) netcat : When the SSLVPN server processes an incoming client request and must send the application data on to the server, it does so to a specific address and port combination. Using netcat on the SSLVPN server, you can make sure there is a listener on the back-end TCP- or UDP-based application. Using ‘netcat ‘ for remote TCP applications or ‘netcat -u’ for UDP based applications, you can confirm that the back-end application is listening for a request by checking for an ‘open’ response. Examples include:

c) SSLVPN portal page: The Policies tab on the SSLVPN portal page defines the applications and subnets that are accessible for this user. When a user cannot access a remote application through the SSLVPN tunnel, check the Policies tab to make sure that the user has a policy allowing access to the application required.

The Log Events tab on the same portal shows any errors that may have occurred accessing the session. Always check this tab for details. There may not always be useful information here – at which point you would analyze the files mentioned in the next section – but be aware of it. More importantly, it confirms the mode of operation being used for this session (Enterprise versus Kiosk) and allow you to look at subsequent log files to gather more useful data.

The Statistics screen shows the amount of data sent and received through the tunnel. It’s a simple way of checking whether the application data is actually going out the tunneled interface. If it is not incrementing as the application is being accessed, the policies may not be correct.

d) SSLVPN configuration ‘Debug Level’ setting: This sets a more verbose logging level for both the OpenVPN and stunnel daemons. With this setting enabled, more verbose information will be available in both the stunnel.log file and novell-openvpn.log files in SSLVPN server/var/messages directory.

Troubleshooting Files

The key troubleshooting files to look at depends on the mode of operation and the OS platform the SSLVPN client is running on. The following list of client-specific files has been broken down into Host OS and SSLVPN modes, and a small explanation is given with both.

applet.log: This contains the output of the ‘Log entries’ screen on the SSLVPN portal page. It reports the status on each phase of connection. It will often report errors when the SSLVPN client cannot connect to the server, but will report little after the connection has been made. An example of the SSL VPN Applet Logs might include the following:

polresolver.log: This is a very useful file in determining DNS details and whether the request has been sent through the tunnel. This file reports details regarding the destination host (IP address and port) that the SSLVPN client is trying to access, as well as the allowed/denied response from the policy engine. A sample entry from this log files might be:

SSLVPNINstall.log: This includes details regarding the installation of the SSLVPN binaries sent down by the servlet, and invoking the environment for the SSLVPN client applications to be SSL- enabled. It also includes priority of message in the file so you can gauge the severity of the problem. An example set of entries from this file might include:

OpenVPNInstall.log: Contains the same info as the SSLVPNINstall.log Kiosk mode file above. You can see details about the installation of the OpenVPN client and the subsequent environment setup.

openvpn.log: Contains all the details regarding the Enterprise mode setup. This includes details about the SSLVPN server you need to communicate with, the certificate exchanged, and ciphers selected for the SSL session. It also includes very useful information about the Tun interface setup, including DNS details, MTU of interface, and the hosts and subnets that the user can access through the SSLVPN server. An example of the entries found in such a file might include:

The Kiosk mode files for this platform are identical to that of the corresponding Linux files above. The exceptions are that the polresolver.log, socksclient.log and stunnel.log files are located under the C:\Documents and Settings\\Novell\SSLVPN\log directory. The equivalent of the above SSLVPNINstall.log is simply the install.log file and is located under the C:\Documents and Settings\\Novell\SSLVPN\install directory.

For the Enterprise mode files, the same files exist as those for the Linux platform above, but again, the directories are different. The applet.log, and openvpn.log files are located in C:\Documents and Settings\\Novell\SSLVPN\log.

There is one new file – the SSLVPN service instalation log is located under C:\Program Files\Novell SSLVPN Service. This file displays the details about the installation of the OpenVPN service and the subsequent environment setup.

Troubleshooting Steps

Now that you understand the tools and files available for troubleshooting issues, let’s look at steps to carry out when troubleshooting SSLVPN application-level issues. Note that up to now, no data has been sent across the SSLVPN tunnel at all, even though the connection is successful. Only when you open an application on the client that requires access to an SSLVPN protected application server, will data be sent across the SSLVPN tunnel.

It’s assumed you’ve followed the steps described above to rule out issues with the SSLVPN installation, initialization, or connection layer. Then, the first step is to identify the mode of operation and whether the client has picked up any errors. This data is available by looking at the Log Events tab of the SSLVPN portal page and checking to see whether the problem is with the Kiosk or Enterprise mode. This page may also give some useful data regarding any errors experienced on the SSLVPN tunnel.

Kiosk Mode Issues

If the SSLVPN connection is in Kiosk mode, this rules out a lot of routing issues. The first thing to do is to identify the back-end resource that the user is trying to go to. Then:

1. Make sure no errors exist in the applet.log for the user.
2. Make sure that the command failing is not a ping command. Ping uses ICMP raw sockets, and since we only support TCP and UDP sockets, ping requests will never succeed.
3. Check the polresolver.log on the client to make sure the connected user is allowed access to the host or network that appears to be problematic.

The log file will report an ‘allow’ or ‘deny’ state for that user. For example, here’s what you would see in the log file if the user was not allowed access to the application (ICMP in this case):

At this stage, you can assume that the request has been passed down to the stunnel client, to be transmitted out the LAN. One way to confirm this is to do the following:

1. Look at the ‘Statistics’ screen on the SSLVPN portal page and make sure that the packets ‘Sent’ and ‘Received’ counters are incrementing. Or, you can use TCPDUMP or Wireshark on the client, look at the TCP connections to the SSLVPN server port, and make sure they are going out the right LAN interface.

2. Check the routing table on the SSLVPN client and make sure that the next hop for the SSLVPN server IP address is the correct interface.

3. Check the stunnel.log file on the SSLVPN client and make sure that no errors occur there. First you need to launch the client application that needs to talk to the remote server behind the SSLVPN. Then the tunnel will be activated, and any errors here will be displayed in this file. The following shows an issue where the SSL handshake fails because of an invalid server certificate:

If the packets ‘sent’ are incrementing but the packets ‘received’ are not, you need to look at the server side in more detail.

1. Run ‘tcpdump -i any -s 0 -n -w sslvpn.cap’ on the SSLVPN server. Check the resulting sslvpn.cap file with Wireshark and make sure the requests from the SSLVPN client are arriving at the LAN interface on the server. If this is failing, check a network diagram for any intermediate device that could be processing this request, such as a firewall. Make sure this firewall is capable of forwarding the SSLVPN destined TCP segments to the SSLVPN server.

2. Look at the above sslvpn.cap trace and check to see if requests are being sent on the loopback interface to the SOCKS server (TCP port 1080). If this is failing, make sure that iptables or the SuSEFirewall2 service is not running on the SSLVPN server and blocking the requests. Run ‘rcSuSEFirewall2 stop’ to disable this service temporarily for checking.

3. Assuming you are still in a ‘connected’ state from the SSLVPN client, run the ‘tail -f /var/log/stunnel.conf’ command on the SSLVPN server console and try launching the client application again. Ideally, you should see some data displayed on the SSLVPN server console indicating that the stunnel daemon received data:

4. Compare the output of your file to determine at what stage the SSLVPN tunnel connection fails. If no errors are reported here, then the data going through the tunnel appears to be fine. You will either have an application-level issue at this stage or a possible routing issue.

To check for routing issues, try and simulate the request to the remote application server from the SSLVPN server itself. The easiest way of doing this is to use the netcat tool described in the tools section above to talk to the remote TCP or UDP port. If a netcat to the remote application from the SSLVPN server fails, then there is a routing or possible firewall issue on the internal network. Sort out this routing issue so that requests from the SSLVPN server reach their destination, and that the responses from this destination arrive back at the SSLVPN server

To check application level issues, one needs to be familiar with the application itself. If the application we’re tunneling is not one of the applications known to have issues with the Kiosk mode described in the Introduction section of this document (e.g. NETBIOS based, Applications opening TCP connections on both directions), then take a LAN trace using tcpdump and try and compare the request/response combination going through the SSLVPN server, with that when the user goes direct to the back end application.

A simple test to narrow the issue down to a specific application and not the SSLVPN tunnel is to create a simple traffic policy allowing a user ssh into a back end server. This is a basic application and assuming all works fine, we can rule out issues with the SSLVPN tunnel and focus on the problem application.

Enterprise mode issues:

If the SSLVPN connection is in Enterprise mode, routing issues become much more prevalent. The initial troubleshooting steps will be similar to that for the Kiosk mode above.

1. Make sure no errors exist in the applet.log for the user.
2. Check the polresolver.log on the client to make sure that the connected user is allowed access to the host or network that appears to be problematic (as in the Kiosk mode above). The log file will report an ‘allow’ or ‘deny’ state for that user. For example, here’s what you would see in the log file if the user was not allowed access to the application (ICMP in this case).

At this stage, we can assume that the request has been passed down to the OpenVPN client, to be transmitted out the LAN. One way to confirm this is to do the following:

1. Look at the ‘Statistics’ screen on the SSLVPN portal page and make sure that the packets ‘Sent’ and ‘Received’ counters are incrementing. Or, using tcpdump or wireshark on the client, look at the TCP connections to the SSLVPN server port and make sure they are going out the right LAN interface.

2. Check the routing table on the SSLVPN client and make sure that the next hop for the SSLVPN server IP address is the correct interface.

3. Check the openvpn.log file for errors. The following example below shows a ‘route: netmask doesn’t match route address’ error that is easily missed. This error prevented any requests from going out the tunnel, because the route was invalid. This occurred because the traffic rule subnet address was incorrectly configured in the Admin Console, and this invalid subnet was sent down to the SSLVPN client.

If the packets ‘sent’ are incrementing but the packets ‘received’ are not (very common!), then one must look at the server side in more detail.

1. Run ‘tcpdump -i any -s 0 -n -w sslvpn.cap’ on the SSLVPN server. Check the resulting sslvpn.cap file with Wireshark and make sure the requests from the SSLVPN client are arriving at the LAN interface on the server. If this is failing, check a network diagram for any intermediate device that could be processing this request, such as a firewall. Make sure this firewall is capable of forwarding the SSLVPN destined TCP segments to the SSLVPN server.

2. Look at the /proc/net/dev file and check to see if the tun0 interface ‘Receive’ data is incrementing. This will indicate whether the tun0 OpenVPN interface is receiving any data.

3. If the LAN (eth0 interface) receives data, but the corresponding tun0 counters do not increment, check whether routing is enabled on the SSLVPN server. Run ‘cat /proc/sys/net/ipv4/ip_forward’ and make sure setting is at 1 (enabled).

4. Make sure that the route back to the SSLVPN server is available. By default, the source IP of the incoming request will be in the 10.8.0.0/16 range. When the SSLVPN server routes the request to the destination application server, it will need to have a route back to this 10.8.0.0/16 subnet through the SSLVPN server. Here’s a typical request that reaches the destination application server but has no route back. It shows the following:

The request is being forwarded to 11.0.0.11 with the source IP address of 10.8.0.6. The host 11.0.0.11 has no route back to the SSLVPN server for 10.8.0.6, so it sends it to the default router, and the datagram eventually gets dropped. Failure to route the response back through the SSLVPN server will result in a communication problem and the application will fail.

There are two options around this:

a) Add a route entry for the 10.8.0.0/16 subnet via the SSLVPN server on all remote application servers (a lot of work), or

b) Use iptables to rewrite the source IP address from the 10.8.0.0/16 subnet address to that of the private interface of the SSLVPN server. A sample command to do this would be

iptables -t nat -A POSTROUTING -s 10.8.0.0/16 -j SNAT –to 10.0.0.1

where 10.0.0.1 is the IP address of the private interface in the SSLVPN server. Looking at the same tcpdump output as above, with iptables remapped source IP addresses, you would see this:

Disclaimer: As with everything else at NetIQ Cool Solutions, this content is definitely not supported by NetIQ, so Customer Support will not be able to help you if it has any adverse effect on your environment. It just worked for at least one person, and perhaps it will be useful for you too. Be sure to test in a non-production environment.