WAAS Express is WAAS functionality built into IOS running on a device such as a router. The WAAS Central Manager can manage a WAAS Express device along with other WAAS devices in the WAAS network. This article describes how to troubleshoot WAAS Express device operation.

Note: WAAS Express Central Manager support was introduced in WAAS version 4.3.1. This section is not applicable to earlier WAAS versions.

Verifying WAAS Express Image Version

To verify the WAAS Express image version use the show waas status command on the WAAS Express router. To view the WAAS Express image version from the WAAS Central Manager, choose My WAN > Manage Devices.

Verifying WAAS Express License

The WAAS Express license comes in two varieties: evaluation license (valid for 12 years) and permanent license. Use the show waas status command on the WAAS Express device to display the license information.

Verifying WAAS Enabled Interfaces

Use the show waas status command on the WAAS Express device to list the set of interfaces on which WAAS is enabled. This command also displays the kind of optimization supported by the device. Some of the WAAS Express router models do not support DRE.

To view similar information from the Central Manager, choose the WAAS Express device, then choose Monitor > Optimization > Connections Statistics to see the Connections Summary Table.

Figure 1. Connections Summary Table

Verifying WAAS Optimized Data

On the WAAS Express device, use the show waas statistics application command to list the optimized data classified into each application. The WAAS Express device does not show pass-through data. This data is used to generate the TCP related charts in the WAAS Central Manager.

Verifying WAAS Express Alarms

On the WAAS Express device, use the show waas alarms command to list the alarms that are present in the device and their status.

waas-express# show waas alarms
WAAS status: enabled
Alarms
Connection limit exceeded: on <----- on indicates this alarm is active. off indicates inactive
Too many peers discovered: off
WAAS license expired: off
WAAS license revoked: off
WAAS license deleted: off
High CPU: off

To view alarms for all devices from the Central Manager, choose My WAN > Alerts. In addition to the alarms listed above, an alarm is raised if the clocks of the WAAS Express and WAAS Central Manager devices are not synchronized.

Verifying WAAS Express Peers

On the WAAS Express device, use the show waas statistics peer command to list the peer devices of the WAAS Express device.

To view similar information from the Central Manager, choose Monitor > Topology.

Offline Alarms

The WAAS Express device may go to an offline state in the Central Manager because of the following issues:

Central Manager does not have WAAS Express device credentials.

Credentials are not configured for this WAAS Express device in the Central Manager. The WAAS Central Manager needs the WAAS Express username and password to communicate with the WAAS Express device. You can configure credentials in the Central Manager by choosing My WAN (or a WAAS Express device or device group) > Admin > WAAS Express Credentials.

Authentication failed while communicating with WAAS Express device.

The Central Manager is not able to communicate with the WAAS Express because wrong credentials are configured. You can configure credentials in the Central Manager by choosing My WAN (or a WAAS Express device or device group) > Admin > WAAS Express Credentials.

SSL Handshake failed while communicating with WAAS Express devcie.

The WAAS Express device certificate is changed and the same certificate is not imported for this device in the Central Manager. To reimport the WAAS Express device certificate, choose the WAAS Express device, then choose Admin > Certificate.

No route to WAAS Express device.

The Central Manager is not able to reach the WAAS Express Device. Configure the correct WAAS Express management IP address by choosing the WAAS Express device, then choosing DeviceName> Activation.

Connection is refused by WAAS Express device.

The HTTPS server port configured on the WAAS Express device is not the same as the port shown in the Central Manager DeviceName> Activation page. Configure the correct WAAS Express HTTPS server port in this page.

WAAS support is not available on WAAS Express device.

The WAAS Express device is downgraded to an IOS image version with no WAAS support. Install an IOS image with WAAS support.

Connection timed out while communicating with WAAS Express device.

The WAAS Express device is taking more than 30 seconds to respond to the Central Manager. It could be because the WAAS Express device is overloaded or the network is slow.

License is expired on WAAS Express device.

The Evaluation license on the WAAS Express device is expired. Install a Permanent license by using the WAAS Express license install command.

The WAAS Express device and Central Manager are using the cipher rc4-128-md5 for SSL communication. Sometimes the Central Manager fails to decrypt the SSL data sent by the WAAS Express. Configure the ciphers 3des-ede-cbc-sha, des-cbc-sha, and rc4-128 by using the WAAS Express command ip http secure-ciphersuite 3des-ede-cbc-sha des-cbc-sha rc4-128-sha.

Failed to check the status of WAAS Express device.

The Central Manager is not receiving configuration status from the WAAS Express device. Contact Cisco TAC for assistance troubleshooting.

Management Status is offline.

If you see this error message, contact Cisco TAC for assistance troubleshooting.

Verifying WAAS Express HTTPS Configuration

To verify the HTTPS server configuration on the WAAS Express device, use the show ip http server secure status command.

WAAS-Express - WAE - WAAS CM Compatibility

WAAS-Express Version 1.0,1.5

This version of WAAS-Express supports the transport optimization which includes TFO, LZ, and DRE.

WAAS-Express version 1.0 is introduced in IOS software release 15.1(3)T1WAAS-Express version 1.5 is introduced in IOS software release 15.1(4)M. In addition to optimization, this release adds support for embedded monitoring capability called Performance Agent (PA). For more information on PA, please see PA page on CCO

Unexpected WAAS-Express License Expiration

The WAAS-Express license is active in show license. However, WAAS-Express license is expired in show waas status. This is potentially a known bug, CSCtw86624. Verify this by issuing following show commands. WAAS CM thinks that license is expired and shows the device as offline. However, the connections should be optimized, since based on the license, the feature is active.

Solution: Upgrade to a recommended WAAS-Express Version 2 image - 15.2(4)M1 or install a permanent license.

Verify this by comparing the WAAS-Express router certificate expiration date stored on the WAAS CM. Navigate to this page from the WAAS-Express device page, Admin->Certificate. Compare the certificate information with the output of show crypto pki certificate output on the WAAS-Express router. If there is any mismatch, it is very likely the certificate ia re-generated.

Possible Cause #1: WAAS-Express device certificate changes

Verify this by comparing the WAAS-Express router certificate expiration date stored on the WAAS CM. Navigate to this page from the WAAS-Express device page, Admin->Certificate. Compare the certificate information with the output of show crypto pki certificate output on the WAAS-Express router. If there is any mismatch, it is very likely the certificate ia re-generated.

Issue show run | include crypto pki trustpoint. Non-persistent trustpoint naming is in the format of TP-self-signed-xxxxxxxxxx.

There are serveral instances where the certificate could be re-generated but the main reason is trustpoing is created as non-persistent. If you enable SSL Express AO with 15.2(3)T, you could also potentially hit CSCtz85134.

Solution: Upgrade to 15.2(4)M1 and re-create persistent trustpoint. Delete the certificate from WAAS CM and re-register.

Was this an upgrade from 15.1(3)T to 15.2(3)T?

In 15.2(3)T, there is a mandatory config within the crypto pki trustpoint, which requires rsa-keypair to be configured. If this config does not present before upgrade, this could potentially cause the router not be able to detect the trustpoint. This will cause HTTPS connectivity to fail. This problem is documented in CSCty04359.

Solution: Remove the trustpoint and re-create. Delete the certificate from WAAS CM and re-register.

Possible Cause #2: Incorrect certificates or trustpoints are used

Does the router have multiple trustpoints configured?

During WAAS CM registration, WAAS-Express router selects the trustpoint which it uses for sending certificate to WAAS CM. This may be different trustpoint from what the local HTTPS server on the WAAS-Express router uses.

Solution: Verify that the same trustpoing is configured in ip http secure-trustpoint <trustpoint_name> and ip http-client secure-trustpoint <trustpoint_name>

Possible Cause #3: Device authentication problem

Is authentication failing?

Verify that you can login to the WAAS-Express router, by directing your browser to WAAS-Express router using HTTPS and attempt the authentication manually.

Solution: Verify that manual authentication is successful.

Debug Information

If you believe you are running into certificate related issuses, please provide below information to support team.

If the counter for Interface Application Configincrements, it is likely your policy is configured to pass-through this particulate connection. Check your WAAS policy on both WAAS-Express and its peer.

Solution: Check and validate your optimization policy. Use below debug to discover if traffic is marked as pass-through in the policy.

show policy-map type waas interface
debug waas infra events

If the counter for Interface Global Config increments, this could be caused by asymetrical routing in your network. This is the case where WAAS-Express or its peer does not see both directions of the TCP traffic. This could be caused by true asymetrical routing in the network, or could be caused by some packets are getting dropped by devices in the traffic path (ACL, firewall, etc.)

Solution: Check for asymetric routing of dropped packets in the network.See what could cause asymetric routing or dropped packets in the network below.

Connections could also be pass-through if the peers are not compatible with each other. This may happen if you run the non-compatible version between WAAS-Express and WAE. Check the compatibility table above for recommedned software releases.

Solution #1: Check if the peer is incompatible using show waas statistics aoim

Solution #2: If you believe you have asymetrical routing scenario in your network, check the following.

What could cause asymetric routing or dropped packets in the network

Multiple WAN links in either the WAAS-Express router or the peer. Note that WAAS-Express it not supported on active/active or active/standby routers because both traffic leaving and entering the WAN need to be on the same WAAS-Express router. If there are multiple WAN links, make sure all the WAN links have config waas enable. Make sure that all the WAN links and routers on the peer routers have config to redirect traffic to WAAS.

Control packets (SYN, SYN-ACK, ACK) are not tagged with WAAS option.This could happen if the traffic is not redirected to WAAS on the peer side. Check your WCCP ACL.

Note: Pass-through connections are not counted in the per-platform connection limit. WAAS-Express does not track pass-through connections, hence there are no statistics related to pass-through flows. There, however, are counters that indicate how many flows were put into pass-through and why.

Connections are not getting the desired optimization level

This is usually caused by misconfiguration. HTTP-Express Accelerator and CIFS-Express Accelerator are disabled by default in WAAS-Express Version 2 image.Check that the Express Accelerator is enabled globally.

Symtom: Established connections do not get the desired or configured policy to use CIFS, SSL, or HTTP-Express AO

Symtom: Expected connection optimization is THDL, but established connection has TDL

This typically is caused by mis-configuration of the policy.

Note: HTTP-Express AO is not enabled by default.

Solution #1: Check if the core WAAS device is compatible. This check can be done using show waas statistics aoim

Solution #2: Check if HTTP-Express Accelerator is getting negotiated during auto-discovery using auto-discovery debugs. This may be because the accelerator is disabled globally (note that HTTP accelerator is not enabled by default), or HTTP class is missing “accelerate http” in the action.

Symtom: Expected connection optimization is TSDL, but established connection has TDL

The connection may also be getting pipe’ed. This can checked using show waas statistics accelerator ssl

Router#show waas statistics accelerator ssl
SSL-Express:
Global Statistics
-----------------
Time Accelerator was started: 16:31:37 UTC Jul 26 2012
...
Pipe through due to C2S cipher mismatch: 0
Pipe through due to C2S version mismatch: 0
Pipe through due to W2W cipher mismatch: 0
Pipe through due to W2W version mismatch: 0
Pipe through due to detection of non-SSL traffic: 0
Pipe through due to unknown reasons: 0
Total pipe through connections: 0
...

Expected connection optimization is TSHDL, but established connection has only TSDL or THDL

SSL-Express Accelerator introduces HTTP-Express Accelerator in the path. Make sure both SSL-Express and HTTP-Express Accelerator are enabled globally.

The connection got pipe-through’ed and shows up as TG. As shown above, check reason in show waas statistics accelerator ssl

If the connection shows up as TSDL could be due to one of the following

HTTP-Express Accelerator is disabled.

HTTP-Express Accelerator is not compatible with the HTTP AO on core WAAS device.

At least 3 optimization features of HTTP-Express Accelerator are not enabled.

The first data packet does not contain HTTP content.

If the connection shows up as THDLcould be due to one of the following

SSL-Express Accelerator is not up and running on edge device.

SSL AO is not up and running on core device.

SSL-AO was not negotiated in AOIM.

For proxy, HTTP CONNECT request is to a port other than 443.

The 3-way DATA-INSPECT handshake where both edge and core devices notify each other regarding addition of SSL-AO to the optimization for this connection fails.

Post DATA-INSPECT handshake, the 3-way TFO handshake where both edge and core devices agree to add SSL-AO to the optimization for this connection fails.

Router crash/tracebacks

Router crashes and tracebacks may have been seen during testing. Search of previous cases and DDTSs for similar known issues. In addition we also need to isolate what feature is resulting in the crash. If an IOS feature other than ios-waas or layer4-forwarding is resulting in a crash/traceback, then that particular feature development team/ router TAC should be contacted accordingly.

Do a topic search at topic.cisco.com

Check previous customer cases for similar/known issues.

Information to be provided to the development team:

show tech or if not possibleshow running-config output

Exact IOS version.

Exact steps to reproduce the problem.

Decodes of traceback, or crashinfo in the case of crash.

Topology of the network

Any relevant information that will help with the reproduction of the problem internally.

Slow connection/degraded performance

Degraded performance may be caused by various reasons: the nature of the traffic, the load on the router, network topology or packet drops in the network. For dealing with slow connections, we need to determine relative degradation with respect to pass-through or non-optimized connections.

Step to troubleshoot

What is the optimization action for the connection?

Check Accel field in show waas connection. Is it TDL, THDL, TSDL, etc?

If a particular Accelerator is being used, does turning it off recover from the poor performance?

If there is upload traffic, try disable uplink DRE in the WAAS-Express parameter-map.

If the connection is put in TFO-only mode, is there a degradation seen with respect to pass-through mode?

What is the load on the router, check cpu utilization using: show proc cpu history

Check whether CPU throttling messages are seen in the log. When the CPU is too high, WAAS-Express slows down the optimization in order to protect the CPU from being too overloaded

Check output of interface statistics to determine if there are packet drops.

Check if there are any ACLs that are dropping packets. A good debug to find which feature drops any packets is debug ip cef drop.

Check if any device in the middle is dropping packets.

WAEs turn on ECN by default, and send packets with ECT bit set. Old devices may not like packets with ECT bit set and hence can drop these packets leading to retransmissions and hence degraded performance. In a particular customer case, a device (with an old IOS image) in the middle was dropping packets that had ECT bit set in the TCP header.

ECN can be turned off on core WAE by using the following command in config mode: no tcp ecn enable

Does the setup have WAAS-Express enabled on multiple WAN links? If so, is the load-sharing being used a supported option?

Per-packet load-sharing is not a supported option.

Per-destination load-sharing is a supported option. There should be no performance impact seen with this load-sharing.

Asymmetric routing in the network, causing packet drops and retransmissions.

If the router does not see all packets of a particular flow, this may lead to slow/hung connections.

Moving WAAS-Express device between Device-Groups on CM

If a WAAS-Express device is moved between device-groups on the WCM, it is sometimes seen that the policy definitions under the new device-group do not take effect. When a device is unassigned from a device-group, it gets the policies from the backup policy set of what the device last owned.

Use the following steps when moving the device between device-groups:

* Go to the Policy Definitions page of that device and select the new device-group and click on Submit.
OR
* Go to device-group-1 -> Assign Devices page and unassign the device from this DG.
* Go to device-group-2 -> Assign Devices page and assign the device to this DG.
* Go to device-group-2 -> Policy Definitions page and click on 'Force DG settings' button.

Other useful information

Statistics mismatch on WAAS-Express and WCM/WAE:

There are no known issues in this area. Please collect the logs using following procedure and provide them to the development team.