Introduction

This document describes what debugs to collect for most of the common Group Encrypted Transport VPN (GETVPN) issues.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

GETVPN

Syslog Server Use

Components Used

This document is not restricted to specific software and hardware versions.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Background Information - GETVPN Troubleshooting Tools

GETVPN provides an extensive set of troubleshooting tools in order to ease the troubleshoot process. It is important to understand which of these tools are available, and when they are appropriate for each troubleshooting task. When troubleshooting, it is always a good idea to start with the least intrusive methods, so that the production environment is not negatively impacted. In order to assist that process, this section describes some of the commonly used tools available:

Control Plane Debugging Tools

Show Commands

Show commands are commonly used in order to show runtime operations in a GETVPN environment.

Syslogs

GETVPN has an enhanced set of syslog messages for significant protocol events and error conditions. This should always be the first place to look before you run any debugs.

Group Domain of Interpretation (GDOI) Event Trace

This feature was added in Version 15.1(3)T. Event tracing offers lightweight, always-on tracing for significant GDOI events and errors. There is also exit-path tracing with traceback enabled for exception conditions.

GDOI Conditional Debugs

This feature was added in Version 15.1(3)T. It allows filtered debugs for a given device based on the peer address, and should always be used when possible, especially on the Key Server.

Global Crypto and GDOI Debugs

These are the all of the various GETVPM debugs. Admins must use caution when debugging in large-scale environments. With GDOI debugs, five debug levels are provided for further debugging granularity:

Troubleshoot

Logging Facility Preparation and Other Best Practices

Before you begin to troubleshoot, ensure that you have prepared the logging facility as described here. Some best practices are also listed here:

Check the router amount of free memory, and configure logging buffered debugging to a large value (10 MB or more if possible).

Disable logging to the console, monitor, and syslog servers.

Retrieve the logging buffer content with the show log command at regular intervals, every 20 mins to an hour, in order to prevent log loss due to buffer reuse.

Whatever happens, enter the show tech command from affected Group Members (GMs) and Key Servers (KSs), and examine the output of the show ip route command in global and each Virtual Routing and Forwarding (VRF) involved, if any are required.

Use Network Time Protocol (NTP) in order to sync the clock between all devices that are debugged. Enable millisecond (msec) timestamps for both debug and log messages:

Policy Issue Occurs POST Registration, and Pertains to the Merge of Global Policy and Local Overrides

This issue usually manifests itself in the form of messages that indicate that an encrypted packet was recieved for which the local policies indicate that it is not supposed to be encrypted and vice versa. All of the data requested in the previous section and the show tech command output is required in this case.

Troubleshoot Time-based Anti-replay (TBAR)

The TBAR feature requires time-keeping across groups, and therefore requires the GMs pseudo-time clocks to be constantly resynced. This is performed during rekey or every two hours, whichever comes first.

Note: All output and debugs must be collected at the same time from both GMs and KS so that they can be correlated appropriately.

In order to investigate issues that occur at this level, collect this output.

On the GMs:

gm1# show crypto gdoigm1# show crypto gdoi replay

On the KS:

ks1# show crypto gdoi ks members ks1# show crypto gdoi ks replay

In order to investigate TBAR time-keeping in a more dynamic way, enable these debugs:

As of Cisoc IOS Version 15.2(3)T, the ability to record TBAR errors has been added, which makes it easier to spot these errors. On the GM, use this command in order to check if there are any TBAR errors:

Troubleshoot KS Redundancy

Cooperative (COOP) establishes an IKE session in order to protect interKSs communication, so the troubleshooting technique previously described for IKE establishment is applicable here as well.

COOP-specific troubleshooting comprises output checks of this command on all KSs involved:

ks# show crypto gdoi ks coop

Note: The most common mistake made with deployment of COOP KSs is to forget to import the same RSA key (both private and public) for the group on all KSs. This causes problems during rekeys. In order to check and compare public keys amongst KSs, compare the output of the show crypto key mypubkey rsa command from each KS.

If protocol-level troubleshooting is required, enable this debug on all KSs involved:

ks# debug crypto gdoi ks coop packet

FAQ

Why do you see this error message "% Setting rekey authentication rejected"?

You see this error message when you configure the KS after this line is added:

The reason for this error message is usually because the key labelled GETVPN_KEYS does not exist. In order to fix this, create a key with the correct label using the command:

crypto key generate rsa mod <modulus> label <label_name>

Note: Add the exportable keyword at the end if this is a COOP deployment and then import the same key in the other KS

Can a router configured as KS for one GETVPN group also function as a GM for the same group?

No. All GETVPN deployments require a dedicated KS which cannot participate as a GM for the same groups. This feature is not supported, because adding GM functionality to KS with all possible interactions like encryption, routing, QoS, etc., is not optimal for the health of this crucial network device. It must be available at all times for the entire GETVPN deployment to work.