Invalid Cells on the LightStream 1010 and 8510

Cisco ATM switch routers, which include the LightStrem 1010 and the Catalyst 8500 series switch routers, use a switching fabric with a shared-memory architecture. In some cases, the switch drops cells and increments the invalid cells counter, as shown in the output of one of the following commands and their respective platforms:

show switch fabric - Catalyst 8540

show controller atm 2/0/0 - LightStrem 1010

show controller controller0 - Catalyst 8510

These
switches increment the invalid cells counter when they discard an ATM cell that has a valid header error checksum (HEC) value, but arrives on a non-existent virtual circuit (VC). Possible reasons include the following:

Corrupted cell header

Incomplete or no configuration of that VC in the switch fabric. For example, if you configure a virtual path identifier/virtual channel identifier (VPI/VCI) pair only on a router and not on the attached ATM switch, cells transmitted on this VC from the router are considered invalid by the switch.

This document explains the invalid cells counter on Cisco ATM campus switches and provides tips on how to troubleshoot incrementing invalid c
ells.

The output of show controllers atm 2/0/0 (or 13/0/0) or atm 0 (depending on software version and chassis) on the LightStream 1010 or Catalyst 8510 prints a table of the most recently received invalid cells. The show controllers atm command is cleared on read, meaning that the invalid cells counter is cleared when the show command is executed. Thus, if you are not receiving invalid cells continuously on an interface, the invalid cells counter will show as zero when you read a subsequent time.

Invalid Cells on the Catalyst 8540

The
show switch fabric command on the Catalyst 8540 does not print a log of the most recent invalid cells. However, you can use the following commands to determine on what VPI and VCI the invalid cell arrived.

Step 1 Use the show switch fabric command to determine the MSC# with incrementing invalid cells. Each of the two required switch processors in the Catalyst 8540 contains four MSC ASICs. Each ASIC forms the switch fabric for several ports.

Step 6 Use the
show atm vc interface atm command to confirm the connection exists in the system.

Note: The invalid cells log may refer to port numbers and VPI/VCI values that do not correspond to actual port numbers and VCs. The reason is that the port interface (PIF) chip on some modules considers idle cells on some ports as invalid cells. The interface chip actually changes the default VPI of 0 for idle cells to a different value. For example, VPI/VCI 0/16 on port #1 will be changed to VPI/VCI 4/16 on the PIF chip. The interface chip drops idle cells from these ports and increments the invalid cells counter. On ports 0 and 6, the interface chip does not change the default idle cell value of VPI/VCI 0/0 since the physical VPI is the same as VPI on the interface chip.

Invalid Cells on Ethernet Interfaces

Ethernet interfaces can also experience cell drops due to invalid cells in an ATM switch router. Ethernet interfaces derive much of their local intelligence from a PIF ASIC, which segments Ethernet frames for transmission across the switch router's internal ATM switch fabric. A PIF checks if a received frame or packet is received for a protocol for which the interface is configured. It then searches the content-addressable memory (CAM) table and determines the egress VPI/VCI value for the destination port. Finally, the PIF segments the frame into cells, applies a five-byte header with the appropriate egress VPI/VCI information, and sends the cells out the switch fabric. If the PIF needs to drop a frame, it tags all cells in that frame with a VPI/VCI =0/0, and the switch fabric then drops these cells.

An Ethernet PIF drops cells and increments the invalid cells counter due to one of the following reasons:

Layer-2 filtering of MAC addresses reachable out the same interface as the received Ethernet frame. The Catalyst 8500 filters such "local" MAC addresses by sending received frames into the switch fabric on the 0/0 VPI/VCI. These drops are equivalent to the in-discards counter on the Catalyst 5000.

Packet discard on a multiplexing-based (MUX-based) Gigabit Ethernet line card. On such cards, when a received packet has a non-forwardable or un-recognizable protocol and must be discarded, the switch puts the packet on the 0/0 VPI/VCI.

Microcode put a packet on a VPI/VCI that is not actually established in the switch core.

Rejected and Discarded Cells

In some cases, the switch discards cells and reports these drops in the output of one of the following commands, depending on the platform:

The purpose of these sections is to review each of the above reasons and provide tips on how to troubleshoot why you are seeing non-zero values for rejected cells. But before you start, look at
Understanding Switch Architecture.

Understanding Switch Architecture

The LightStream 1010 and Catalyst 8510 use an architecture that differs from the Catalyst 8540.

On the LightStream 1010 and 8510, use the show controller atm 2/0/0 (or 13/0/0 if used in the Catalyst 5500) or show controller atm0 to view statistics for the CPU and ATM switch processor:

The
discarded cellsfield indicates the total count of discarded cells across all ports.

On the Catalyst 8540, use the show switch fabric command to display the number of rejected cells. Note that this output differs from the LightStream 1010 output in that it shows rejected cells per Modular Switching Component (MSC) number. MSC application-specific integrated circuits (ASICs) form the switch fabric for a defined set of modules and ports.

Step 2 Check the
MSC#, Rej. Cells,andInv. Cellsfields. They indicate the rejected cells per MSC# or matching set of physical ports.

Step 3 Check the Switch Fabric Statistics section for theRejected CellsandInvalid Cellsfields. They indicate the total number of rejected cells.

Each of the two required switch processors in the 8540 contain four MSC ASICs, which build the internal switch fabric for half of the ports in a system. Use the show mmc ports command to determine which physical ports use a particular MSC#.

With each slot, the first half of the ports use an even MSC#, and the second half use an odd MSC#. However, when using the original LightStream 1010 port adapter modules (PAMs) with a super carrier module (SuperCAM) carrier access module, all ports on a single SuperCAM map to an even SP and MSC ASIC. For example, the first half of the ports in slot 0 normally connect to MSC0 of SP0, while the second half of the ports in slot 0 connect to MSC1 of SP1. However, with a SuperCAM, both sets of ports connect to MSC0 of SP0.

Inadequate Buffer Space

Cisco campus ATM switches use a shared-memory architecture that stores up to 65,536 cells. Using a shared-memory design provides the following benefits:

Supports more connections or higher loads.

Supports an increased amount of multicast traffic since only a single copy of any multicast cell is stored in the common cell memory.

Since all ports can use the full memory, it is important that the buffer management process ensures fairness among ports by ensuring that one or a small subset of ports cannot occupy all of the buffers.

Note: The opposite of a shared-memory architecture is a per-port output buffer architecture, in which each port has dedicated memory buffers that cannot be accessed by other ports. The Catalyst 6000 and Catalyst 5000 are output-buffered switches.

On an LightStream 1010, use the sh controller atm 2/0/0 command to view the number of memory buffers currently in use.

Check the
memory bufferfield. It should display
a non-zero value on a busy production switch.

You can use the following managed objects of the CISCO-RHINO-MIB to poll your ATM switch router for the number of free buffers and discarded cells:

Managed Object

Description

ciscoAtmSwitchTotalBuffer

Total cell buffer count in the switch's shared memory.

ciscoAtmSwitchFreeBuffer

Free cell buffer count in switch's shared memory.

ciscoAtmSwitchDiscardCells

Total discarded cells of the switch.

Exceeding Maximum Queue Limits

ATM switch routers use configurable queue limits and thresholds to control queuing in the system. The queuing processes and configurable values vary with the feature card installed on the ATM switch processor (ASP) or multiservice switch processor (MSP):

Feature card per-class queuing (FC-PCQ)

Feature card per-flow queuing (FC-PFQ) and 8540

Service category limit

Yes

No

Maximum queue size per interface

Yes

No

Threshold groups

No

Yes

The Catalyst 8510 and the LightStream 1010 with a FC-PCQ support service category limits, which restrict the number of cells admitted into the switch as determined by the type of output queues. Use the show atm resource command to display these limits. Use the atm service-category-limit command to configure a non-default value.

Note: In the previous example
all ATM service classes have access to most of the shared memory by default.

The Catalyst 8510 and LightStream 1010 with a FC-PCQ also support maximum queues sizes, which determine the number of cells that can be scheduled for transmission per ATM service class per interface. Use the atm output-queue command to configure a non-default value.

Because not all queue size values are supported by the switch fabric, the value installed is displayed, as well as the configuration value requested. The value installed is always greater than or equal to that requested. Use the show atm interface resource atm command to display both values.

Note: Systems with a FC-PFQ queue cells on input, not output, so the
atm output-queue commands do not apply.

Catalyst 8510 and LightStream 1010 systems with a FC-PFQ and Catalyst 8540s support the threshold groups feature. Each group consists of virtual paths (VPs) and virtual circuits (VCs) that belong to the same ATM service category, such as VBR-nrt or UBR. By default, one threshold group holds cells for one ATM service class. Use the atm threshold-group command to assign more than one service category to a threshold group and to assign a service category to a non-default group number. Use the show atm resource command to confirm your changes.

In the following sample output, the ATM switch router is using the default settings. One ATM service class is assigned to one threshold group.

Each threshold group consists of eight regions, with each region having a set of thresholds. A threshold group congests when its member VCs have a large number of cells stored in the shared cell memory. As the cumulative number of queued cells for the member VCs approaches the "Max cells install" value, the maximum number of cells in each per-VC and per-VP queue shrinks from the max-queue-limit to min-queue-limit. Refer to the "Max Q limit instal" and "Min Q limit instal" columns in show atm resource output for the queue size values.

When congestion is in the range of 0 cells (uncongested) to 1/8th full, the connection queues are limited to max-queue-size. In general, as you move from one region to another, you make the new threshold max (previous-threshold/2, min-queue-threshold). When congestion is in the range of 7/8ths full to completely full, the connection queues are limited to min-queue-size. Note that the operation of the switch for threshold groups in upper regions only occurs if the group congests by going above 1/8th full. However, the maximum size and threshold position commands are effective even for threshold groups in the lowest region.

The following commands adjust the threshold group values.

Command

Description

atm threshold-group group max-cells number

Configures the maximum number of cells queued for all VCs in the group. See the "Max cells instal" value in show atm resource.

atm threshold-groupgroup max-queue-limit number

Configures the largest per-VC queue limit applied to all VCs in the group. See the "Max Q limit install" value in show atm resource.

atm threshold-group group min-queue-limit number

Configures the smallest per-VC queue-limit applied to all VCs in the group. See the "Min Q limit install" value in show atm resource.

atm threshold-groupgroup marking-threshold percent

Determines the point at which a per-VC queue is considered "full", and the switch begins to set the explicit forward congestion indication (EFCI) bit or implements available bit rate (ABR) relative-rate marking. See the "Q thresholds Mark" value in show atm resource.

atm threshold-groupgroup discard-threshold percent

Determines the point at which a per-VC queue is considered "full", and the switch begins to discard cells with the cell loss priority (CLP) bit to one and implements early packet discard (EPD). See the "Q thresholds Discard" value in show atm resource.

The show atm vc command displays the following two counters related to rejected or discarded cells caused by queue threshold values being exceeded:

Number of cells queued per threshold group

Number of drops due to full queue via the counters "Rx Clp0 q full drops" and "Rx Clp1 qthresh drops"

Note: The output of the show atm vc command changes with respect to drop counters depending on whether packet discard is enabled on the VC.

Check the numbers following the Clp0 q full dropsandClp1 qthresh dropsfields.

You also can obtain these counts via SNMP polling.

Managed Object

Description

ciscoAtmVclClp0VcqFullCellDrops

Total number of cells received on this virtual channel link (VCL) with the CLP bit clear, discarded because the per-VC queue limit is exceeded. This counter is valid only if EPD is disabled on the VCL. On LightStream 1010s, this counter is valid only when the switch processor is equipped with an FC-PFQ.

ciscoAtmVclVcqClpThreshCellDrops

Total number of cells received on this VCL, discarded because the discard threshold (as opposed to the queue-limit) is exceeded on the per-VC queue, and the CLP bit is set. This counter is valid only if EPD is disabled on the VCL. On LightStream 1010s, this counter is valid only when the switch processor is equipped with an FC-PFQ.

ciscoAtmVclLsPerVcQThreshGrp

Threshold Group to which the cell-queue for cells received by this VC are queued. Note that this value is not valid until the VCL is in an active cross-connect. On LightStream 1010s, this counter is valid only when the switch processor is equipped with an FC-PFQ.

Usage Parameter Control (UPC) Violations

When configured, an ATM switch at the network side of a user-network interface (UNI) polices the flow of cells in the forward (into the network) direction of a virtual connection. These policing mechanisms are known as usage parameter control (UPC). They determine whether received cells are complying with the negotiated traffic management values, and then take one of the following actions on violating cells
, depending on the configuration:

Pass the cell without changing the cell loss priority (CLP) bit in the cell header.

Check the Packet-discard-optionand Usage-Parameter-Control setting. Also check the Upc Violationsfield for the number of violations.

You also can obtain these counts via SNMP polling. Use the ciscoAtmVclUpcViolations managed object in the CISCO-ATM-CONN-MIB.

Note: In evaluating the arriving cell rate, the ATM switch router counts both Operation, Administration, and Maintenance (OAM) cells as well as with data cells since the current signaling protocol does not allow a user to explicitly specify traffic parameters for OAM flows.

Cell Loss Priority (CLP) Discarding

The standard ATM cell header includes the cell loss priority (CLP) bit, which explicitly indicates that a cell experienced congestion during transmission to the destination end. A CLP value of one means that the cell has a lower priority and thus is more likely to be dropped in times of congestion. Thus, you can use the CLP bit to generate different priority cell flows.

ATM switch routers use a threshold-based CLP selective discard mechanism that imposes a threshold on the number of cell buffers to be shared by CLP = 0 and CLP = 1 cells. When the switch port queue occupancy reaches a user-configurable threshold level, only CLP = 0 cells are allowed to enter the system, and CLP = 1 cells are discarded.

The show atm resource command displays the queue threshold percentage, after which cells are eligible for CLP discard or early packet discard. This value is the column labeled "Discard."

Discard - Threshold at which cells are eligible for CLP discard or early packet discard (EPD).

Optionally, you can enable tail packet discard on each VC for use with CLP selective discard. With the TPD option, the system "goodput" (usable throughput) is enhanced. You enable tail packet discard (TPD) by specifying the "pd" or packet discard parameter in the
atm pvc command. The "pd" parameter enables both tail packet discard and early packet discard.

With UPC, you can implement either drop or tag as UPC policy. You cannot set up a tag-and-drop policy in which you tag above sustained cell rate (SCR) and drop above peak cell rate (PCR).

The following example output was generated on a permanent virtual circuit (PVC) with packet discard (PD) enabled, UPC set to pass, and traffic shaping parameters set to 10 MB SCR and 20 MB PCR. Sending 25 MB through the PVC produces UPC violations on approximately 60% of the cells.

With switched virtual circuits (SVCs), Cisco campus ATM switches use the AAL5 information element (IE) to indicate whether or not to enable packet discard; the presence of the AAL5 IE tells the switch to enable PD. With ATM interfaces using UNI 4.0 signaling, ATM switches may use the frame-discard bits in the traffic management options field of the ATM Traffic Descriptor IE.

Intelligent Tail Packet Discard/Early Packet Discard (ITPD/EPD)

Most data frames are segmented and transmitted across an ATM cloud as multiple cells. If one or more cells are dropped by the network, the resulting packet fails the CRC check at the receiving end and must be retransmitted. Such retransmissions lead to poor effective throughput or goodput, which is defined as the number of delivered cells that are not part of a retransmission or an incomplete packet.

To maximize the number of completely delivered packets, your ATM switch router implements a unique ITPD/EPD scheme that intelligently and selectively discards cells belonging to the same packets in order to minimize the effects of fragmentation. Working together, ITPD/EPD can prevent frequent buffer overflows by dumping corrupted or complete packets from rapidly filling buffers. By dropping a small number of packets instead of cells from a large number of packets, occasional buffer overflows do not have serious negative effects on the end-to-end system goodput.

ITPD works to minimize fragmentation as it is occurring. ITPD acts in response to cell drops due to one of the following reasons:

UPC enforcement action upon violation

Buffer overflow

Exceeding any of the buffer limits

CLP selective discard

When one cell of a packet has been discarded by the ATM switch router, ITPD discards all subsequent cells of the same packet. Depending on the feature card, the last cell (also known as the end of packet (EOP) cell) may be dropped as well.

ATM switches identify the EOP cell via a bit in the payload type identifier (PTI) field of a cell header. The FC-PCQ does not drop the last cell of frame when doing EPD, while the FC-PFQ does.

EPD works to prevent fragmentation before it occurs. With EPD, the ATM switch router begins to discard all cells except the EOP cell from newly arriving packets when the switch buffer queues reach a user-configurable threshold level. If the first cell of a packet has entered the buffer, all remaining cells of the packet are also allowed to enter if sufficient buffer space is available. Otherwise, TPD is enabled.

Use the atm threshold-group group discard-threshold percent command to configure the threshold at which point the queue is considered full and EPD starts dropping cells. See the "Q thresholds Discard" value in the output of show atm resource for the default discard percentage.

The placement of the EPD threshold determines how efficiently the buffer is used and how often cells are dropped. The EPD threshold essentially functions as the effective buffer size. The excess buffer capacity above the EPD threshold is used to accommodate cells from those packets that have already had cells in the buffer or in the transmission on the line.

Setting the threshold depends on many factors, including:

Distribution of packet sizes

Traffic distribution

Duration of the congestion period

Proportion of the incoming cells during the congestion period that belong to outstanding packets and as a result must be buffered.

The interaction with other ATM-level or transport-level flow and congestion control mechanisms.

In addition, the amount of excess buffer capacity required depends on how the buffer is shared with non-TPD/EPD traffic. You can enable TPD by specifying the "pd" or packet discard parameter in the
atm pvc command. The "pd" parameter enables both tail packet discard and early packet discard. Packet discard can only be enabled for AAL5 connections. Here is how the drop behavior changes with UPC and the PD option:

If UPC is configured to drop and PD is disabled, then the switch drops violating cells only.

If UPC is configured to drop and PD is enabled, then the switch performs ITPD and drops all the cells following the violating one (except the last cell).

If PD is enabled and cells are queued to a buffer which has already exceeded its EPD threshold, then the switch drops the complete (AAL5) packet.

In other words, PD is applied as EPD whenever possible (high queue sizes, for example) and as ITPD in all other cases, including UPC drops and buffer overflow.