Command-Line Interface Troubleshooting Commands

The command-line interface (CLI) allows you to configure and monitor Cisco NX-OS using a local console or remotely using a Telnet or SSH session. The CLI provides a command structure similar to Cisco IOSsoftware, with context-sensitive help, show commands, multiuser support, and roles-based access control.

Each feature has show commands that provide information about the feature configuration, status, and performance. Additionally, you can use the following commands for more information:

show system--Provides information about system-level components, including cores, errors, and exceptions. Use the show system error-id command to find details on error codes.

Error Facility: sysmgr
Error Description: request was aborted, standby disk may be full

show platform--Provides platform-specific information, including route forwarding, Quality of Service (QoS), and access control list (ACL) information.

Configuration Files

Configuration files contain the Cisco NX-OS commands used to configure the features on a Cisco NX-OS device. Cisco NX-OS has two types of configuration files, running configuration and startup configuration. The device uses the startup configuration (startup-config) during the device startup to configure the software features. The running configuration (running-config) contains the current changes that you make to the startup-configuration file. You should create a backup version of your configuration files before modifying that configuration. You can back up the configuration files to a remote server. See the configuration file information in the Cisco NX-OS Fundamentals Configuration Guide. You can also create a checkpoint copy of the configuration file that you can roll back to if problems occur. See the Rollback feature in the Cisco NX-OS System Management Configuration Guide.

Cisco NX-OS features can create internal locks on the startup configuration file. In rare cases, these locks may not be removed by the features. Use the show system internal sysmgr startup-config locks command to determine if any locks remain on the startup configuration file. Use the system startup-config unlock command to remove these locks.

CLI Debug

Cisco NX-OS supports an extensive debugging feature set for actively troubleshooting a network. Using the CLI, you can enable debugging modes for each feature and view a real-time updated activity log of the control protocol exchanges. Each log entry has a timestamp and is listed chronologically. You can limit access to the debug feature through the CLI roles mechanism to partition access on a per-role basis. While the debug commands show real-time information, you can use the show commands to list historical and real-time information.

Caution:

Use the debug commands only under the guidance of your Cisco technical support representative because some debug commands can impact your network performance.

Note:

You can log debug messages to a special log file, which is more secure and easier to process than sending the debug output to the console.

By using the ? option, you can see the options that are available for any feature. A log entry is created for each entered command in addition to the actual debug output. The debug output shows a time-stamped account of the activity that occurred between the local device and other adjacent devices.

You can use the debug facility to track events, internal messages, and protocol errors. However, you should be careful when using the debug utility in a production environment, because some options may prevent access to the device by generating too many messages to the console or creating CPU-intensive events that could seriously affect network performance.

Note:

We recommend that you open a second Telnet or SSH session before you enter any debug commands. If the debug session overwhelms the current output window, you can use the second session to enter the undebug all command to stop the debug message output.

Debug Filters

You can filter out unwanted debug information by using the debug-filter command. The debug-filter command allows you to limit the debug information produced by related debug commands.

Ping and Traceroute

Note:

Use the ping and traceroute features to troubleshoot problems with connectivity and path choices. Do not use these features to identify or resolve network performance issues.

The ping and traceroute commands are two of the most useful tools for troubleshooting TCP/IP networking problems. The ping utility generates a series of echo packets to a destination across a TCP/IP internetwork. When the echo packets arrive at the destination, they are rerouted and sent back to the source.

The traceroute utility operates in a similar fashion but can also determine the specific path that a frame takes to its destination on a hop-by-hop basis.

Using Ping

Use the ping command to verify connectivity and latency to a particular destination across an IPv4 routed network.

Use the ping6 command to verify connectivity and latency to a particular destination across an IPv6 routed network.

The ping utility allows you to send a short message to a port or end device. By specifying the IPv4 or IPv6 address, you can send a series of frames to a target destination. Once these frames reach the target, they are looped back to the source and a time stamp is taken.

Using Traceroute

Use traceroute to do the following:

Trace the route followed by the data traffic.

Compute the interswitch (hop-to-hop) latency.

The traceroute utility identifies the path taken on a hop-by-hop basis and includes a time stamp at each hop in both directions. You can use traceroute to test the connectivity of ports along the path between the generating device and the device closest to the destination.

Use the traceroute command for IPv4 networks. Use the traceroute6 command for IPv6 networks. If the destination cannot be reached, the path discovery traces the path up to the point of failure.

Using the show system resource Command

Use the show system resources command to display system-related CPU and memory statistics. The output includes the following:

Load is defined as the number of running processes. The average reflects the system load over the past 1, 5, and 15 minutes.

Processes displays the number of processes in the system, and how many are actually running when the command is issued.

CPU states shows the CPU usage percentage in user mode, kernel mode, and idle time in the last one second.

Memory usage provides the total memory, used memory, free memory, memory used for buffers, and memory used for cache in kilobytes. Buffers and the cache are also included in the used memory statistics.

Using Onboard Failure Logging

Cisco NX-OS provides the facility to log failure data to the persistent storage, which can be retrieved and displayed for analysis. This onboard failure logging (OBFL) feature stores failure and environmental information in nonvolatile memory on the module. This information will help you analyze failed modules.

Using GOLD Diagnostics

Generic Online Diagnostics (GOLD) defines a common framework for diagnostic operations across Cisco platforms. The GOLD implementation checks the health of hardware components and verifies proper operation of the system data and control planes. Some tests take effect when the system is booting up; other tests take effect when the system is operational.

A booting module goes through a series of checks before coming online to allow the system to detect faults in the hardware components at bootup and to ensure that a failing module is not introduced in a live network.

Defects are also diagnosed during system operation or runtime. You can configure a series of diagnostic checks to determine the condition of an online system. You must distinguish between disruptive and nondisruptive diagnostic tests. Although nondisruptive tests occur in the background and do not affect the system data or control planes, disruptive tests do affect live packet flows. You should schedule disruptive tests during special maintenance windows. The show diagnostic content module output displays test attributes such as disruptive or nondisruptive tests.

You can configure runtime diagnostic checks to run at a specific time or to run continually in the background.

Health-monitoring diagnostic tests are nondisruptive, and they run in the background while the system is in operation. The role of online diagnostic health monitoring is to proactively detect hardware failures in the live network environment and inform you of a failure.

GOLD collects diagnostic results and detailed statistics for all tests including the last execution time, the first and last test pass time, the first and last test failure time, the total run count, the total failure count, the consecutive failure count, and the error code. These test results help administrators determine the condition of a system and understand the reason for a system failure. Use the show diagnostic result command to view diagnostic results.

Using Embedded Event Manager

Embedded Event Manager (EEM) is a policy-based framework that allows you to monitor key system events and then act on those events through a set policy. The policy is a preprogrammed script that you can load that defines actions that the device should invoke based on set events occurring. The script can generate actions, including, but not limited to, generating custom syslog or SNMP traps, invoking CLI commands, forcing a failover, and much more.

Using Ethanalyzer

Ethanalyzer is a Cisco NX-OS protocol analyzer tool based on the Wireshark (formerly Ethereal) open source code. Ethanalyzer is a command-line version of Wireshark that captures and decodes packets. You can use Ethanalyzer to troubleshoot your network and analyze the control-plane traffic.

To configure Ethanalyzer, use the following commands:

Command

Purpose

ethanalyzer local interface

Captures packets sent or received by the supervisor and provides detailed protocol information.

ethanalyzer local interface inband

Captures packets sent or received by the supervisor and provides detailed protocol information in the inband and outband interfaces.

ethanalyzer local interface mgmt

Captures packets sent or received by the supervisor and provides detailed protocol information in the management interfaces.

ethanalyzer local interface {inband | mgmt} brief

Captures packets sent or received by the supervisor and provides a summary of protocol information.

ethanalyzer local interface {inband | mgmt} limit-captured-frames

Limits the number of frames to capture.

ethanalyzer local interface {inband | mgmt} limit-frame-size

Limits the length of the frame to capture.

ethanalyzer local interface {inband | mgmt} capture-filter

Filters the types of packets to capture.

ethanalyzer local interface {inband | mgmt} display-filter

Filters the types of captured packets to display.

ethanalyzer local interface {inband | mgmt} decode-internal

Decodes the internal frame header for Cisco NX-OS.

Note Do not use this option if you plan to analyze the data using Wireshark instead of Ethanalyzer.

ethanalyzer local interface {inband | mgmt} write

Saves the captured data to a file.

ethanalyzer local read

Opens the captured data file and analyzes it.

Ethanalyzer does not capture data traffic that Cisco NX-OS forwards in the hardware.

SNMP and RMON Support

The SNMP standard allows any third-party applications that support the different MIBs to manage and monitor Cisco NX-OS.

SNMPv3 provides extended security. Each device can be selectively enabled or disabled for SNMP service. In addition, each device can be configured with a method of handling SNMPv1 and v2 requests.

Cisco NX-OS also supports Remote Monitoring (RMON) alarms and events. RMON alarms and events provide a mechanism for setting thresholds and sending notifications based on changes in network behavior.

The AlarmGroup allows you to set alarms. Alarms can be set on one or multiple parameters within a device. For example, you can set an RMON alarm for a specific level of CPU utilization on a device. The EventGroup allows you to configure events that are actions to be taken based on an alarm condition. The types of events that are supported include logging, SNMP traps, and log-and-trap.

Using RADIUS

The RADIUS protocol is used to exchange attributes or credentials between a head-end RADIUS server and a client device. These attributes relate to three classes of services:

Authentication

Authorization

Accounting

Authentication refers to the authentication of users for access to a specific device. You can use RADIUS to manage user accounts for access to a Cisco NX-OS device. When you try to log into a device, Cisco NX-OS validates you with information from a central RADIUS server.

Authorization refers to the scope of access that you have once you have been authenticated. Assigned roles for users can be stored in a RADIUS server with a list of actual devices that the user should have access to. Once the user has been authenticated, the device can then refer to the RADIUS server to determine the access that the user will have.

Accounting refers to the log information that is kept for each management session in a device. You can use this information to generate reports for troubleshooting purposes and user accountability. You can implement accounting locally or remotely (using RADIUS).

The accounting log shows only the beginning and end (start and stop) for each session.

Using syslog

The system message logging software saves messages in a log file or directs the messages to other devices. This feature provides the following capabilities:

Logging information for monitoring and troubleshooting

Selection of the types of logging information to be captured

Selection of the destination of the captured logging information

You can use syslog to store a chronological log of system messages locally or to send this information to a central syslog server. The syslog messages can also be sent to the console for immediate use. These messages can vary in detail depending on the configuration that you choose.

The syslog messages are categorized into seven severity levels from debug to critical events. You can limit the severity levels that are reported for specific services within the device. For example, you may wish only to report debug events for the OSPF service but record all severity level events for the BGP service.

Log messages are not saved across system reboots. However, a maximum of 100 log messages with a severity level of critical and below (levels 0, 1, and 2) are saved in NVRAM. You can view this log at any time with the show logging nvram command.

Logging Levels

Cisco NX-OS supports the following logging levels:

0-emergency

1-alert

2-critical

3-error

4-warning

5-notification

6-informational

7-debugging

By default, the device logs normal but significant system messages to a log file and sends these messages to the system console. Users can specify which system messages should be saved based on the type of facility and the severity level. Messages have a time stamp to enhance real-time debugging and management.

Enabling Logging for Telnet or SSH

System logging messages are sent to the console based on the default or configured logging facility and severity values.

Users can disable logging to the console or enable logging to a given Telnet or Secure Shell (SSH) session.

To disable console logging, use the no logging console command in configuration mode.

To enable logging for Telnet or SSH, use the terminal monitor command in EXEC mode.

When logging to a console session is disabled or enabled, that state is applied to all future console sessions. If a user exits and logs in again to a new session, the state is preserved. However, when logging to a Telnet or SSH session is enabled or disabled, that state is applied only to that session. The state is not preserved after the user exits the session.

The no logging console command disables console logging and is enabled by default.

switch(config)# no logging console

The terminal monitor command enables logging for Telnet or SSH and is disabled by default.

Using SPAN

You can use the Switched Port Analyzer (SPAN) utility to perform detailed troubleshooting or to take a sample of traffic from a particular application host for proactive monitoring and analysis.

When you have a problem in your network that you cannot solve by fixing the device configuration, you typically need to take a look at the protocol level. You can use debug commands to look at the control traffic between an end node and a device. However, when you need to focus on all the traffic that originates from or is destined to a particular end node, you can use a protocol analyzer to capture protocol traces.

To use a protocol analyzer, you must insert the analyzer inline with the device under analysis, which disrupts input and output (I/O) to and from the device.

In Ethernet networks, you can solve this problem by using the SPAN utility. SPAN allows you to take a copy of all traffic and direct it to another port within the device. The process is nondisruptive to any connected devices and is facilitated in the hardware, which prevents any unnecessary CPU load.

SPAN allows you to create up to 16 independent SPAN sessions within the device. Each session can have up to four unique sources and one destination port. In addition, you can apply a filter to capture only the traffic received or the traffic transmitted. You can even capturetraffic from a particular VLAN.

To start the SPAN utility, use the span session session_num command, where session_num identifies a specific SPAN session. When you enter this command, the system displays a submenu, which allows you to configure the destination interface and the source VLAN or interfaces.

Using the Blue Beacon Feature

On some platforms, you can cause the platform LEDs to blink. This feature is a useful way to mark a piece of hardware so that a local administrator can quickly identify the hardware for troubleshooting or replacement.

To flash the LEDs on a hardware entity, use the following commands:

Command

Purpose

blink chassis

Flashes the chassis LED.

blink fan number

Flashes one of the fan LEDs.

blink module slot

Flashes the selected module LED.

blink powersupply number

Flashes one of the power supply LEDs.

blink xbar number

Flashes one of the crossbar module LEDs.

To flash a single port LED on a module, use the following command in interface configuration mode: