Troubleshooting Ports

This chapter describes how to identify and resolve problems that can occur with ports in the Cisco MDS 9000 Family of multilayer directors and fabric switches. It includes the following sections:

Overview

Initial Troubleshooting Checklist

Overview of the FC-MAC Driver and the Port Manager

Common Problems with Port Interfaces

Overview

Before a switch can relay frames from one data link to another, the characteristics of the interfaces through which the frames are received and sent must be defined. The configured interfaces can be Fibre Channel interfaces, Gigabit Ethernet interfaces, the management interface (mgmt0), or VSAN interfaces (IPFC).

Each physical Fibre Channel interface in a switch can operate in one of several port modes: E port, F port, FL port, TL port, TE port, SD port, and B port. In addition to these modes, each interface can be configured in auto or Fx port modes. These modes determine the port type during interface initialization.

Each interface has an associated administrative configuration and operational status:

The administrative configuration does not change unless you modify it. This configuration has various attributes that you can configure in administrative mode.

The operational status represents the current status of a specified attribute like the interface speed. This status cannot be changed and is read-only. Some values may not be valid when the interface is down (such as the operation speed).

For a complete description of port modes, administrative states, and operational states, refer to the Cisco MDS 9000 Family Configuration Guide and the Cisco MDS 9000 Fabric Manager Configuration Guide.

Initial Troubleshooting Checklist

Troubleshooting a SAN problem involves gathering information about the configuration and connectivity of individual devices and the entire SAN fabric. In the case of port interfaces, begin your troubleshooting activity as follows:

ChecklistÂ

Check off

Check the physical media to ensure there are no damaged parts.

Verify that the SFP (small form-factor pluggable) devices in use are those authorized by Cisco and that they are not faulty.

Verify that you have enabled the port by right-clicking the port in Device Manager and selecting enable or by using the no shut CLI command.

Right-click the port in Device Manager or use the show interface CLI command to verify the state of the interface. Refer to [#wp113794 TableÂ 8-1] for reasons why a port may be in a down operational state.

Verify that you if you have one host-optimized port configured as an ISL, you have not connected to the other three ports in the port group.

Verify that no ports on a Generation 2 module are out of service.

Note:

Use the show running interface CLI command to view the interface configuration in Cisco SAN-OS Release 3.0(1) or later. The interface configuration as seen in the show running-config CLI command is no longer consolidated.

Table 8-1 Reason Codes for Nonoperational States

Reason Code

Description

ApplicableMode

Link failure or not connected

The physical layer link is not operational.

All

SFP not present

The small form-factor pluggable (SFP) hardware is not plugged in.

Initializing

The physical layer link is operational and the protocol initialization is in progress.

Reconfigure fabric in progress

The fabric is currently being reconfigured.

Offline

The Cisco SAN-OS software waits for the specified R_A_TOV time before retrying initialization.

Inactive

The interface VSAN is deleted or is in a suspended state.

To make the interface operational, assign that port to a configured and active VSAN.

Hardware failure

A hardware failure is detected.

Error disabled

Error conditions require administrative attention. Interfaces may be error-disabled for various reasons. For example:

Configuration failure.

Incompatible buffer-to-buffer credit configuration.

To make the interface operational, you must first fix the error conditions causing this state; then, administratively shut down and reenable the interface.

Isolation due to ELP failure

The port negotiation failed.

Only E ports and TE ports

Isolation due to ESC failure

The port negotiation failed.

Isolation due to domain overlap

The Fibre Channel domains (fcdomain) overlap.

Isolation due to domain ID assignment failure

The assigned domain ID is not valid.

Isolation due to other side E port isolated

The E port at the other end of the link is isolated.

Isolation due to invalid fabric reconfiguration

The port is isolated due to fabric reconfiguration.

Isolation due to domain manager disabled

The fcdomain feature is disabled.

Isolation due to zone merge failure

The zone merge operation failed.

Isolation due to VSAN mismatch

The VSANs at both ends of an ISL are different.

Nonparticipating

FL ports cannot participate in loop operations. It may happen if more than one FL port exists in the same loop, in which case all but one FL port in that loop automatically enters nonparticipating mode.

Only FL ports and RL ports

PortChannel administratively down

The interfaces belonging to the PortChannel are down.

Only PortChannel interfaces

Suspended due to incompatible speed

The interfaces belonging to the PortChannel have incompatible speeds.

Suspended due to incompatible mode

The interfaces belonging to the PortChannel have incompatible modes.

Suspended due to incompatible remote switch WWN

An improper connection is detected. All interfaces in a PortChannel must be connected to the same pair of switches.

Note:

We recommend that you do not disable and then enable a T or TE port. This would affect all the VSANs crossing the EISL instead of just the VSAN experiencing the isolation problem.

Limitations and Restrictions

You must administratively enable a port with the no shut command. When the interface is enabled, the administrative state of the port is up. If you administratively disable an interface with the shut command, the administrative state of the port is down, and the physical link layer state change is ignored.

For a port to be in an up operational state where it can transmit or receive traffic, the interface must be administratively up, the interface link layer state must be up, and the interface initialization must be complete.

The interface cannot transmit or receive data when a port's operational state is down.

The interface is operating in TE mode when a port's operational state is trunking.

Overview of the FC-MAC Driver and the Port Manager

This section describes the internal details of port related components in Cisco SAN-OS. Use this section to understand the underlying functions that may be causing port related problems.

The FC-MAC driver resides in the module component of the Cisco MDS 9000 Family SAN-OS software. It performs the following functions:

Initialization of FC-MAC ASIC.

Speed negotiation.

Link/loop port initialization and credit recovery.

Statistics collection.

Error handling (mainly by acting on error interrupts).

SFP detection and housekeeping.

Statistics collection.

Debug command support under the show hardware internal fc-mac command on the module.

The FC-MAC driver does not handle FLOGI, RSCN, or configuration management.

This section includes the following topics:

Port Manager Overview

Troubleshooting Port States with the Device Manager

Isolating Port Issues Using Device Manager

Using Port Debug Commands

Useful Commands at the FC-MAC Level

Port Manager Overview

The Port Manager is management software running on the supervisor module. The Port Manager handles the following tasks:

Port configuration management.

Link events, including notifying the registered application on the supervisor module.

E or TE port initialization.

SFP validation.

The FC-MAC detects the port is in one of the following states:

Disable--The port is administratively disabled.

Enable--The port is administratively enabled. In this state, the port may be in speed initialization, loop-initialization, link (point-to-point connection) initialization, or the link-up state.

HW Failure--The port has been declared bad due to a hardware failure.

Pause--An intermediate state after the link is down and subsequent enabling of the port to start the port initialization.

You can check the state of the port by attaching to the module using the command:

show hardware internal fc-macportportport-info

Note:

You must use the attach module CLI command to access these FC-MAC show commands.

The FLOGI server is a separate application that handles the FLOGI processing for Nx ports.

Red X—A small form-factor pluggable transceiver (SFP) is present but there is no connection. This could indicate a disconnected or faulty cable, or no active device connection.

Red box—An FSP is present but fabric login (FLOGI) has failed. Typically a mismatch in port or fabric parameters with the neighboring device. For example, a port parameter mismatch would occur if a node device were connected to a port configured as an E port. An example of a fabric parameter mismatch would be differing timeout values.

Yellow box—In Device Manager, a port was selected.

Gray box—The port is administratively disabled.

Black box—FSP is not present.

Figure 8-1 Device Manager: Device View

Device Manager: Summary View

In Device Manager, selecting the Summary View ([#wp96602 FigureÂ 8-2]) expands on the information available for port monitoring. The display includes:

VSAN assignment

For N ports, the port world-wide name (pWWN) and Fibre Channel ID (FC ID) of the connected device

For ISLs, the IP address of the connected switch

Speed

Frames transmitted and received

Percent utilization for the CPU, dynamic memory, and Flash memory

Figure 8-2 Device Manager: Summary View

Device Manager: Port Selection

To drill down for additional port information, use either the Device View or Summary View, and double-click any port. The initial display (Figure 8-3) shows administrative settings for Mode, Speed, and Status, plus current operational status, failure cause, and date of the last configuration change.

Device Manager: Port Monitoring

To display additional details about port traffic, use the Device View or Summary View. In Device View, choose one or more ports, right-click and choose Monitoring from the pop-up menu. In Summary View, choose one or more interfaces, and click the Monitor tool. The initial display (Figure 8-4) shows traffic information for the selected interval, including the number of bytes and frames received and transmitted.

Frame Errors—View frame error statistics, including the number of frames with invalid CRC, Class 3 frames that were discarded upon reception, FBSY returns for selected situations, and FRJT returns resulting from frame rejection by fabric.

Class 2 Traffic—View the amount of Class 2 traffic for the selected interval.

The port monitoring view also provide access to graphing tools. Select the cells that you are interested in, and then click one of the graphing tools to display the related line Figure 8-5), bar, area, or pie chart.

Figure 8-5 Device Manager: Port Monitoring Line Chart

Isolating Port Issues Using Device Manager

To isolate port issues using Device Manager, follow these steps:

1. Choose Interfaces > FC ALL and verify that the Status Oper field is up to determine if the host HBA and the storage port can provide link level connectivity to their respective switches.

See Table 8-1 for details on nonoperational interface reasons.

2. If the port is down and offline, set Admin Status to up and click Apply to bring the port online.

3. Repeat Step 1 to determine if the port is online.

If either of the ports fails to remain in the online state, then you may have a faulty GBIC, cabling or HBA/subsystem port.

4. If both ports are online, select the FLOGI tab to verify that the Fibre Channel ports for the host and storage have performed a fabric login (FLOGI) and are communicating with their respective switches.

5. Choose FC > Name Server to verify that the assigned FC ID during FLOGI exists in the name server database.

At this point the HBA and subsystem ports have successfully established link level connectivity and each one can communicate with its locally attached switch in the fabric. The next step is to verify zone membership. For a more detailed discussion and description of VSANs and zones see "Chapter 14, "Troubleshooting Zones and Zone Sets."

Troubleshooting Port States from the CLI

To display complete information for an interface, use the 'show' interface command. In addition to the state of the port, this command displays:

Using Port Debug Commands

To issue commands with the internal keyword, you must have an account that is a member of the network-admin group.

Examples of when to use these commands include:

An Fibre Channel port fails to move to the up state after such events as link failures, admin-up operations, or new connections.

Unexpected link flaps.

The port moves to "error disabled" state.

Maintain a set of information for the module before these problems occur (if possible) and then gather another set of information after these problems occur.

Useful Commands at the FC-MAC Level

Troubleshooting a port problem involves analysis of the debug facilities provided by the FC-MAC driver, or the FC-MAC2 driver in the case of the MDS 9120, MDS 9140, MDS 9216i, and the MPS-14/2 module. Table 8-2 lists several CLI debugging commands at the FC-MAC level.

Note:

You must use the attach module CLI command to access these FC-MAC show commands.

Note:

Use the fcmac2 keyword for the MDS 9120, MDS 9140, MDS 9216i, and the MPS-14/2 module.

Represents an aggregation of a number of debug commands from all ASICs. The information includes interrupt-statistics, error-statistics, exception-log, link-events, and all debug information that is provided by the FC-MAC driver.

Note:

To issue CLI commands with the internal keyword, you must have an account that is a member of the network-admin group.

Isolating Port Issues Using the CLI

To isolate port issues using the CLI, follow these steps:

1. Use the show interface command to determine if the host HBA and the storage port can provide link level connectivity to their respective switches.

NPI1# show interface fc2/5 status
fc2/5 is down (Offline)
NPI2# show interface fc2/5 status
fc1/5 is up Port mode is F

See Table 8-1 for details on nonoperational interface reasons.

2. If the port is down and offline, use the no shutdown command to bring the port online.

If either of the ports fails to remain in the online state, then you may have a faulty GBIC, cabling or HBA/subsystem port.

4. If both ports are online, use the show flogi command to verify that the Fibre Channel ports for the host and storage have performed a fabric login (FLOGI) and are communicating with their respective switches.

At this point the HBA and subsystem ports have successfully established link level connectivity and each one can communicate with its locally attached switch in the fabric. The next step is to verify zone membership. For a more detailed discussion and description of vsans and zones see "Chapter 14, "Troubleshooting Zones and Zone Sets."

Common Problems with Port Interfaces

The following issues are commonly seen with port interfaces:

Port Remains in a Link Failure or Not Connected State

Port Remains in Initializing State

Unexpected Link Flapping Occurs

Port Bounces Between Initializing and Offline States

E Port Bounces Remains Isolated After a Zone Merge

Port Cycles Through Up and Down States

Port Is in ErrDisabled State

Troubleshooting Fx Port Failure

Port Remains in a Link Failure or Not Connected State

If a link does not come up, then the switch was unable to achieve bit or word synchronization with the node device. This situation may occur if nothing is connected to the interface, as in the case of a broken fibre, or if there is no bit synchronization between the switch interface and the directly connected Nx port. This problem may be the result of one or more of the possible causes listed in Table 8-3.

Symptom Port remains in a link-failure state.

Table 8-3 Port Remains in a Link-Failure State

Symptom

Possible Cause

Solution

Port remains in a link-failure state.

Port connection is bad.

Use the show port internal info CLI command to verify the port status is in link-failure. Use the 'show hardware internal fc-mac port port'gbic-info CLI command to determine if there is a signal present.

Note:

You must use the attach module CLI command to access the FC-MAC show commands.

Verify the type of media in use. Is it copper or optical, single-mode (SM) or multimode (MM)?

Verify that the media is not broken or damaged. Is the LED on the switch green? Is the active LED on the host bus adapter (HBA) for the connected device on?

Right-click on the port in Device Manager and select disable and then enable, or use the shut CLI command followed by the no shut command to disable and enable the port. If this does not clear the problem, try moving the connection to a different port on the same or another module.

There is no signal because of a transit fault in the SFP or the SFP may be faulty.

When this occurs, the port stays in a transit port state and you see no signal. There is no synchronization at the MAC level. The problem may be related to the port speed setting or autonegotiation. See the "Troubleshooting Port Problems" section. Verify that the SFP on the interface is seated properly. If reseating the SFP does not resolve the issue, replace the SFP or try another port on the switch.

Link is stuck in initialization state or the link is in a point-to-point state.

Choose Logs > Switch Resident > Syslog on Device Manager or use the show logging CLI command to check for a Link Failure, Not Connected systemmessage.

Right-click on the port in Device Manager and select disable and then enable, or use the shut CLI command followed by the no shut command to disable and enable the port. If this does not clear the problem, try moving the connection to a different port on the same or another module.

Note:

We recommend that you do not disable and then enable a T or TE port. This would affect all the VSANs crossing the EISL instead of just the VSAN experiencing the problem.

Troubleshooting Port Problems

Start the debugging with the command 'show hardware internal fc-mac port 'portlink-status. See the "Useful Commands at the FC-MAC Level" section to understand how to use the FC-MAC information.

Note:

You must use the attach module CLI command to access the FC-MAC show commands.

When this command executes, it performs the following checks in the order shown here and displays the appropriate information:

1. Checks whether the port was declared a failure because of an exception. For additional information, use the 'show process exceptionlog 'CLIcommand.

2. Checks whether the port is administratively enabled.

3. Checks whether the physical link state is up. If the state is up, then it does the following:

Checks for possible completion of the FLOGI process.

Note:

FLOGI is transparent to the MAC driver and is based on some expected configuration. The MAC driver assumes that the FLOGI process is completed.

Checks for error counters.

4. Checks whether the port is in the offline state. The port goes to the offline state if the FLOGI or ELP (in case of auto mode) on the port does not succeed.

5. Checks for pause state. A pause state is in an intermediate state (as maintained by the FC-MAC driver) after the link goes down and before the port is enabled by the Port Manager.

Note:

The link reinitializes after a link down event is initiated only if enable is issued by the Port Manager.

6. Checks for the presence of SFP/GBIC. If present, FC-MAC checks for loss of signal. The loss of signal state indicates either the physical connectivity between two end ports is bad or there is a transmit fault in the SFP. Use the 'show hardware internal fc-mac port 'port gbic-info command to check for the transmit fault.

Note:

You must use the attach module CLI command to access the FC-MAC show commands.

7. Checks for the speed and sync state of the port. If the port is in the speed initialization state, then:

Auto speed is in progress is displayed if the port is in automode.

Waiting for stable sync is displayed if the port is configured for a fixed speed.

Sync not acquired is displayed if the MAC state indicates a loss of synchronization. In auto mode, this state is not necessarily an error. In any case, check the speed capabilities and configuration at both ends.

Port Remains in Initializing State

Symptom Port remains in the initializing state.

A port goes into the initialization state after a successful completion of link level initialization. For Fx and FL types of ports, the next step is to complete the FLOGI process. The port remains in the initialization state until the FLOGI (fabric login) process completes.

For E or TE port types, the next step is to complete the ELP process. If the ELP fails the port is moved to the offline state after a timeout and the entire process repeats until the port comes online.

Table 8-4 lists possible causes for FLOGI to fail for a given port and possible solutions.

Table 8-4 Port Remains in the Initializing State

Symptom

Possible Cause

Solution

Port remains in the initializing state.

The port is up because the link partner has put itself in a bypass mode.

Use the show hardware internal fc-mac portportstatistics command to check whether the Class-3 input counter is increasing after the successful completion of link initialization.

Note:

You must use the attach module CLI command to access the FC-MAC show commands.

The FLOGI packet was dropped somewhere in the data path, starting from FC-MAC to the FLOGI server.

Use the show hardware internal fc-mac portportstatistics command to check for Class-3 packet counters.

Note:

You must use the attach module CLI command to access the FC-MAC show commands

Analyze the output of the show hardware internal error command for a possible drop of FLOGI packets somewhere in the path. See the "We recommend that you do not disable and then enable a T or TE port. This would affect all the VSANs crossing the EISL instead of just the VSAN experiencing the problem." section.

Right-click on the port in Device Manager and select disable and then enable, or use the shut CLI command followed by the no shut command to disable and enable the port. If this does not clear the problem, try moving the connection to a different port on the same or another module.

A software bug resulted in an error while handling the FLOGI packet.

Note:

We recommend that you do not disable and then enable a T or TE port. This would affect all the VSANs crossing the EISL instead of just the VSAN experiencing the problem.

Troubleshooting Port Registration Issues Using the CLI

To troubleshoot Nx port registration in the CLI, follow these steps:

1. Use the show interface 'fc 'slot/port command and verify that the fibre channel interface connected to the device in question is up and free of any errors. (See Example 8-3.)

We recommend that you do not disable and then enable a T or TE port. This would affect all the VSANs crossing the EISL instead of just the VSAN experiencing the problem.

4. Use the no shutdown command on the Fibre Channel interface.

switch(config-if)# no shutdown

By shutting down the interface and bringing it back up, you can determine what happens when the connected device tries to log in to the interface.

Use the show flogi internal event-history interface command to view the events that occurred on the interface after you enabled it again. The comments that follow each section of output explain the meaning of the output.

Note:

To issue commands with the internal keyword, you must have an account that is a member of the network-admin group.

Other attribute objects of the Nx port are registered one per register operation after the FLOGI process is complete. The Nx port performs PLOGI to the well-known WWN of the Name Server, 0xFFFFFC. The FC_CT Common Transport protocol uses Request and Accept messages to conduct transactions. To verify that additional attributes are correctly registered and recorded in the database, you can use the SAN-OS debug facility.

Note:

The command show fcns database detail vsan X displays a detailed list of all devices registered in the fabric.

Unexpected Link Flapping Occurs

Symptom Unexpected link flapping occurs.

When a port is flapping, it cycles through the following states, in this order, and then starts over again:
1. Initializing - The link is initializing.
2. Offline - The port is offline.
3. Link failure or not connected - The physical layer is not operational and there is no active device connection.

When troubleshooting unexpected link flapping, it is important to know the following information:

Who initiated the link flap.

The actual link down reason.

Be sure to check the HBA, because a faulty HBA can manifest symptoms on the attached switch port. For example, if an Nx port is self-diagnosed as faulty by the HBA driver or firmware, the driver can place the port in optical bypass mode. This results in the receive and transmit paths being internally connected through the port. If this happens, the switch port connected to the faulty device will reach bit and word synchronization with itself. If the port is configured in auto mode, this will cause the port to issue an ELP and to try to initialize as an xE port, even if an end device is physically connected to that interface. In this case, a port reason code of isolation because of ELP failure can be displayed even if an ISL is not present.

Table 8-5 lists possible causes and solutions for link flapping.

Table 8-5 Unexpected Link Flapping Occurs

Symptom

Possible Cause

Solution

Unexpected link flapping occurs.

The bit rate exceeds the threshold and puts the port into an error disabled state.

Right-click the port in Device Manager and select disable and then enable, or use the shut CLI command followed by the no shut command to return the port to the normal state.

The switch cannot complete the link reset. The link reset protocol failure results in a link flap that may be the result of:

The input buffer did not become empty within the link reset timeout period.

The link partner did not respond to a link reset initiated by the switch.

The switch initiates the link reset when all credits are lost for more than four seconds or when there is a temporary signal or sync loss condition that lasts for less than 100msec. See the "Troubleshooting Port Problems" section to verify this condition.

Right-click the port in Device Manager and select disable and then enable, or use the shut CLI command followed by the no shut command to disable and enable the port. If this does not clear the problem, try moving the connection to a different port on the same or another module.

There is a credit loss condition on an FL port.

When credit loss or a transmit stuck condition is detected in the FL port, the FC-MAC drive flaps the link as a recovery process. See the "Troubleshooting Port Problems" section.

Some problem in the switch triggers the link flap action by the end device. Some of the causes are:

Packet drop in the switch, because of either a hardware failure or an intermittent hardware error such as X-bar sync loss.

Packet drop resulting from a software error.

A control frame is erroneously sent to the device.

Determine link flap reason as indicated by the MAC driver. Use the debug facilities on the end device to troubleshoot the problem. An external device may choose to reinitialize the link upon encountering the error. In such cases, the exact method of reinitializing the link varies by device. See the [#wp89810 "Troubleshooting Port Problems" section] for more information on externally triggered link flaps.

Note:

We recommend that you do not disable and then enable a T or TE port. This would affect all the VSANs crossing the EISL instead of just the VSAN experiencing the problem.

Link Initialization Flow

Fibre Channel primitive sequences are used to establish and maintain a link and they continue to be transmitted until a response has been received. Four primitive sequences are used in the link initialization process:

Not operational sequence (NOS)

Offline sequence (OLS)

Link reset sequence (LRS)

Link reset response sequence (LRR)

Figure 8-6 uses the ordered sets of 8b/10 encoding in the primary operational states. They include:

AC = Active state

LR = Link recovery state

LF = Link failure state

OF = Offline state

Figure 8-6 Link Initialization Flow

Figure 8-6 shows the link initialization flow. It displays the ordered sets transmitted between the ports and the primary operational states of the port during the process. They include:

1. Active state.
2. Link recovery state (LR):

a. LR transmit substate (LR1)

b. LR receive substate (LR2)

c. LRR receive substate (LR3)

3. Offline state (OLS):

a. OLS transmit substate (OL1)

b. OLS receive substate (OL2)

c. Wait for OLS substate (OL3)

4. Link failure state:

a. NOS receive substate (LF1)

b. NOS transmit substate (LF2)

The Cisco MDS 9000 Family switch maintains port counters for link initialization ordered sets, including OLS, LRR, and NOS for fabric connections, as well as primitives for arbitrated loop connections on FL ports and TL ports. Understanding the link initialization flow and viewing the port counters using show interface can be useful when you troubleshoot port initialization problems. Table 8-6 displays the reasons for a link flap.

Table 8-6 Link Flap Reasons Initiated by a Device Connected to the Switch

Reason

Description

Sync Loss

A synchronization loss condition persisted for more than 100 milliseconds. Look at the Invalid Transmission Word Count to check whether the physical link is really bad and if that caused the loss of synchronization. Sometimes this is not necessarily a problem with the physical link, but with the way some devices initialize the link. Use attach module to connect to the module and then use the show hardware internal debug-info interface CLI command. See Table 8-2.

Loss of signal

A signal loss condition persisted for more than 100 milliseconds. Look at the Invalid Transmission Word Count to check whether the physical link is really bad and if that caused the loss of synchronization. Sometimes this is not necessarily a problem with the physical link, but with the way some devices initialize the link. If the link does not come up after a flap, then probably the other end is in a shutdown state or the cable is broken. You can check for the broken or disconnected optical link by using the show hardware internal fc-mac portportgbic-info CLI command.

Note:

You must use the attach moduleCLI command to access the FC-MAC show commands

NOS received

A NOS received condition is detected. If the other end is an MDS port, then the NOS is transmitted by the other end in one of the following conditions:

A signal loss or sync loss condition is detected.

The port is administratively shut down.

The port is operationally down.

OLS received

An OLS received condition is detected.

LR received B2B

Link reset (LR) failed because of the receive queue (in the queue engine) not being empty.

Cr loss

Too many credit loss events occurred.

Rx queue overflow

The receive queue overflowed in the queue engine occurred. This can happen under the following conditions:

Improper credit configuration at one or both ends of the link.

A bad link can sometimes result in extra R_RDYs. Check for invalid transmission words at both ends.

LIP F* received

An loop initialization procedure (LIP) was received.

LC port shutdown

The port shutdown was invoked. Use the show process exception CLI command to check for any other errors.

LIP received B2B

An LIP was received while the Rx queue was not empty.

OPNy tmo B2B

An open circuit on a loop (OPNy) timeout occurred while the Rx queue was not empty.

OPNy Ret B2B

An OPNy was returned while the Rx queue was not empty.

Cr Loss B2B

Credit loss occurred while the Rx queue was not empty.

Viewing Port Counters

You can use the show interface counters command to view port counters. Typically, you only observe counters while actively troubleshooting, in which case you should first clear the counters to create a baseline. The values, even if they are high for certain counters, can be meaningless for a port that has been active for an extended period. Clearing the counters provides a better idea of the link behavior as you begin to troubleshoot.

Use one of the following commands in EXEC mode to clear all port counters or counters for specified interfaces:

clear counters interface all

clear counters interface <range>

The counters can identify synchronization problems by displaying a significant disparity between received and transmitted frames. For example, in the case of a broken fiber, if only the Tx path from the F port to the N port is broken, then the switch interface will still have an operational Rx path and will still obtain bit synchronization from the bit stream received from the N port. The switch port will also be able to recognize an incoming NOS from the N port and reply with an OLS. However, because the transmitted OLS never reaches the N port, the R_T_TOV timer expires. In this scenario, the status of the port will also show Link failure or not connected.

The key difference between this case and the no bit synchronization case is that the input and output counts for OLS and NOS increment (as there is bit synchronization but no word synchronization). In such a state, you can check that the Tx path from the switch to the Rx input on the N port interface is properly connected. A faulty transmitter on the switch's SFP or a faulty receiver on the N port's SFP could also cause the issue.

The output in Example 8-4 also displays evidence of corrupt data on the wire if there are a high number of CRCs and errors. Discards may or may not indicate a problem. For example, a frame can be discarded because of an ACL violation.

Port Bounces Between Initializing and Offline States

An ELP failure may result in a port bouncing between the initializing and offline states. Table 8-7 lists possible causes and solutions to this problem.

Table 8-7 Port Bounces Between the Initializing and Offline States

Symptom

Possible Cause

Solution

Port bounces between the initializing and offline states.

An ELP packet was dropped in one of the two switches.

Use the show hardware internal fc-mac portportstatistics CLI command and the show hardware internal error command. Analyze the output of the two commands for possible packet drops. See the "Troubleshooting ELP Issues Using the CLI" section. See also the "E Port Is Isolated in a VSAN" section.

Note:

You must use the attach module CLI command to access the FC-MAC show commands

There is a software bug or incompatibility in handling the ELP process.

Analyze the event history provided by the Port Manager after using the show port internal event-history CLI command. See the "Troubleshooting ELP Issues Using the CLI" section.

This sample output shows the default settings for these timeout values.

3. Optionally, use the fctimer command in config mode to globally set these timeout values across all VSANs or use the fctimerD_S_TOV<timeout>vsan<vsan-id> command for example, to set the D_S_TOV timeout for a particular VSAN to override the global values.

4. Use the show port internal info interface fc command to verify that Rx buffer size matches on both ends of the ISL.

Note:

To issue commands with the internal keyword, you must have an account that is a member of the network-admin group.

StepÂ 3 File:Blank.gifUse one of the following two approaches to resolve a zone merge failure:

â¢File:Blank.gifChoose File > Restore from the Edit Local Full Zone Database dialog box to overwrite the zoning configuration of one switch with the other switch's configuration.

The Restore option overwrites the local switch's active zone set with that of the remote switch.

â¢File:Blank.gifIf the zoning databases between the two switches are overwritten, you cannot use the Restore option. To work around this, you can manually change the content of the zone database on either of the switches using the Edit Local Full Zone Database, and then choose Switches > Interfaces > FC Physical andselect down and then up on the Admin Status drop-down menu for the isolated port.

StepÂ 4 File:Blank.gifIf the isolation is specific to one VSAN and not on an E port, the correct way to issue the cycle up or down is to remove the VSAN from the list of allowed VSANs on that trunk port, and reinsert it.

Note File:Blank.gifWe recommend that you do not disable and then enable a T or TE port. This would affect all the VSANs crossing the EISL instead of just the VSAN experiencing the isolation problem.

Using the Zone Merge Analysis tool in Fabric Manager, the compatibility of two active zone sets in two switches can be checked before actually merging the two zone sets. Refer to the Cisco MDS 9000 Fabric Manager Configuration Guide for more information.

Troubleshooting E port Isolation Using the CLI

To troubleshoot E port isolation due to zoning using the CLI, follow these steps:

StepÂ 1 File:Blank.gifUse the show interface command output to verify that the E port did not come up because of a zone merge failure.

The import option of the command of overwrites the local switch's active zoneset with that of the remote switch. The export option overwrites the remote switch's active zoneset with the local switch's active zone set.

â¢File:Blank.gifIf the zoning databases between the two switches are overwritten, you cannot use the import option. To work around this, you can manually change the content of the zone database on either of the switches, and then issue a shutdown/no shutdown command sequence on the isolated port.

StepÂ 4 File:Blank.gifIf the isolation is specific to one VSAN and not on an E port, the correct way to issue the cycle u p or down is to remove the VSAN from the list of allowed VSANs on that trunk port, and reinsert it.

Note File:Blank.gifWe recommend that you do not disable and then enable a T or TE port. This would affect all the VSANs crossing the EISL instead of just the VSAN experiencing the isolation problem.

Port Cycles Through Up and Down States

SymptomÂ Â Â Port cycles through the up and down states.

This problem may be attributable to an error experienced by the connected device. [#wp81652 TableÂ 8-9] lists the possible causes and solutions for this problem.

TableÂ 8-9 Port Cycles Through the Up and Down States

Symptom

Possible Causes

Solutions

Port cycles through the up and down states.

One or more packets were dropped in the switch.

Analyze the debug log provided by the Nx port. Select Tools > Traceroute using Fabric Manager or use the fctrace CLI command to analyze the link.

Look for FLOGI messages in the logs for this port. See the [#wp98215 "Troubleshooting Port Registration Issues Using the CLI" section]

There is a problem in FLOGI processing.

The device received unexpected packets.

There was a higher layer software error.

Port Is in ErrDisabled State

The ErrDisabled state indicates that the switch detected a problem with the port and disabled the port. This state could be caused by a flapping port or a high amount of bad frames (CRC errors), potentially indicating something wrong with the media.

SymptomÂ Â Â Port is in ErrDisabled state.

An E port may be isolated because of a zone merge failure. [#wp102875 TableÂ 8-10] lists possible causes and solutions to this problem.

TableÂ 8-10 Port is in ErrDisabled State

Symptom

Possible Cause

Solution

Port is in ErrDisabled state.

Flapping port.

See the [#wp38672 "Verifying the ErrDisable State Using the CLI" section]. Verify the SFP, cable, and connections.

Verifying the ErrDisable State Using the CLI

StepÂ 1 File:Blank.gifUse the show interface command to verify that the switch detected a problem and disabled the port. Check cables, SFPs, and optics.

mds# show interface fc1/14

fc1/14 is down (errDisabled)

StepÂ 2 File:Blank.gifUse the show port internal event-history interface command to view information about the internal state transitions of the port. In this example, port fc1/7 entered the ErrDisabled state because of a capability mismatch, or "CAP MISMATCH." You might not know how to interpret this event, but you can look for more information with other commands.

mds# show port internal event-history interface fc1/7

>>>>FSM: <fc1/7> has 86 logged transitions<<<<<

1) FSM:<fc1/7> Transition at 647054 usecs after Tue Jan 1 22:44..

Previous state: [PI_FSM_ST_IF_NOT_INIT]

Triggered event: [PI_FSM_EV_MODULE_INIT_DONE]

Next state: [PI_FSM_ST_IF_INIT_EVAL]

2) FSM:<fc1/7> Transition at 647114 usecs after Tue Jan 1 22:43..

Previous state: [PI_FSM_ST_IF_INIT_EVAL]

Triggered event: [PI_FSM_EV_IE_ERR_DISABLED_CAP_MISMATCH]

Next state: [PI_FSM_ST_IF_DOWN_STATE]

StepÂ 3 File:Blank.gifUse the show logging logfile command to display the switch log file and view a list of port state changes. In this example, an error was recorded when someone attempted to add port fc1/7 to PortChannel 3. The port was not configured identically to PortChannel 3, so the attempt failed.

Troubleshooting Fx Port Failure

Fx port problems can be caused by a variety of configuration issues. While most issues can be solved by simply ensuring that the ports are configured properly, some issues require the use of more in-depth troubleshooting techniques.

Overview of Symptoms

An F port may be connected to a single N port, which is the mode used by peripheral devices (hosts or storage). In all the possible cases an administrator can encounter in troubleshooting an Fx port, two different scenarios can be recognized:

â¢File:Blank.gifThe port does not come up (check the interface configuration, cabling, and the port connected to the switch).

â¢File:Blank.gifThe port comes up, but the host cannot communicate with the storage subsystem (check the VSAN and zone configurations).

Typical end-user questions that lead to Fx port troubleshooting include:

â¢File:Blank.gifWhy is no storage visible on my newly installed server?

â¢File:Blank.gifWhy is previously assigned storage not visible to my server after reboot?

Typical administrator questions to investigate:

â¢File:Blank.gifWhy does the server fail to complete FLOGI to the switch?

â¢File:Blank.gifWhy does the storage device fail to complete FLOGI to the switch?