This document explains how to troubleshoot why the output of the
show interfaces command on a Cisco 12000 Series
Internet Router displays an increasing number of ignored errors. It also
provides troubleshooting tips for an increasing number of no mem
drops in the output of the execute-on
slot<slot#> show controllers (frfab |
tofab) qm stat command. When troubleshooting any of these errors,
verify that the counter is incrementing and is not simply an historical value.

The information in this document is based on the software and hardware
versions below.

Any Cisco IOS® software release which
supports the Cisco 12000 Series Internet Router. Usually these are the 12.0S
and 12.0ST releases.

All the Cisco 12000 platforms are covered by this document. These
include the 12008, 12012, 12016, 12404, 12410, and 12416.

The information presented in this document was created from devices in
a specific lab environment. All of the devices used in this document started
with a cleared (default) configuration. If you are working in a live network,
ensure that you understand the potential impact of any command before using
it.

The Cisco 12000 Series Internet Router uses a distributed architecture
to ensure optimum forwarding performance. To support high forwarding rates, it
maintains packet buffers on both the inbound and outbound line cards. These
packet buffers vary in size and generally are designed to support Maximum
Transmission Unit (MTU)-sized frames.

After it determines the outbound interface for a packet, the forwarding
processor does the following:

The forwarding processor sends a pointer with information about the
packet (including its location in memory) to the virtual output queue of the
outbound interface.

The line card's scheduler issues a request to the scheduler. The
scheduler issues a grant, and the packet is sent from the buffer memory across
the switching fabric to the outbound line card.

The outbound line card buffers the packets.

The L3 processor and associated Application-Specific Integrated
Circuits (ASICs) on the outbound LC transmit the packet out the interface.

If the outbound interface is oversubscribed, it begins to buffer the
excess packets. During periods of sustained oversubscription, the outbound LC's
transmit queues fill. In this condition, the following happens depending on the
outbound LC:

Engine Type of outbound LC

Response to Outbound Congestion

Error Counter

Engine 0 and 1

Sends a backpressure signal. The inbound interface begins to
buffer the excess packets.

Ignored errors in the show
interfaces command output and/or no mem drops in
execute-on slot <slot#> show controllers tofab QM
stat command output of the inbound LC, depending on its L3
Forwarding Engine.�

�You will get ignored errors for L3 Engines 0, 1, and 2 ingress LCs.
However, for four, 16 and more ports on Engine 2 LCs, the ignored counter will
not increase.

On any intelligent networking device, when one or more high-speed
interfaces feed a relatively low-speed interface, a mismatch in interface rates
occurs. Since the slower-speed outbound interface cannot possibly return
buffers as fast as the faster inbound interface is sending them to the output
hold queue, a delay in buffer return leads to some type of drops. This packet
flow breaks the assumption that the outbound interface returns the buffer at
the rate of buffer management time.

In addition to a mismatch in interface rates, ignored errors can
increment when the rate of arriving packets is greater than the CPU can process
them. This condition is very rare on the Cisco 12000 and usually results from a
large number of very small packets, or when a CPU-intensive feature, such as
Access Control Lists (ACLs) or traffic policing, is enabled on an LC that
implements these features in software. This is the case for Engine 0 LCs where
lots of features are implemented in software. However, on later engines, almost
all the features are implemented in hardware. For instance, Engine 3 (IP
Services Engine - ISE) and Engine 4+ line cards are designed for Edge
applications and implement enhanced IP services (such as Quality of Service -
QoS) in hardware with no performance impact. Examples of this hardware include
1-Port CHOC-48 ISE, 4-Port CHOC-12 ISE, 16-Port OC-3 POS ISE, 4-Port OC-12 POS
ISE, 1-Port OC-48 POS ISE, and 1-Port OC-48 POS ISE.

The ignored counter can also be incremented whenever a packet arrives
on an ingress line card and an appropriate size packet buffer is not available
to handle this packet. However, this condition is very rare and it is not
covered in this document.

The solution to ignored errors and no mem drops caused by output
interface oversubscription is the same for any L3 Engine type -- prevent buffer
starvation. In other words, we need a mechanism that prevents the FrFab queues
from filling.

Simply put, the ignored counter is incremented when a packet arrives on
an ingress line card (LC) and an appropriate size packet buffer is not
available to handle this packet. Thus, ignored packets typically do not point
to a bug in Cisco IOS software.

Here's a sample output from the show
interfaces command with a non-null ignored counter on a Cisco
12000 Series Router:

When the outbound LC is an Engine 0 or 1, it sends a backpressure
message to the other LCs telling them to no longer send data to that particular
LC. The inbound interface then buffers the excess packets in its ToFab queues
corresponding to this destination slot.

To isolate the most likely cause of why the Ignored counter is
increasing, you need to look at the ToFab queues of the ingress LC. You can
either attach to the LC over the Maintenance BUS (MBUS) using the
attach command, or use the execute-on
slot <slot#> show controllers tofab
queue command to check the ToFab queues. Execute this command a
few times and look for the following symptoms:

A decreasing and low value or value of 0 in the #Qelem column of a
non-IPC free queue

Line cards using a more recent L3 Engine architecture do not use a
backpressure mechanism. Instead, when the interface is oversubscribed and a
FrFab queue becomes depleted, the packets are simply dropped as they arrive on
the egress line card.

Engine 2 LCs do not fall back to the next larger buffer pool when a
smaller pool becomes depleted. The fall back mechanism has only been
implemented for Engine 2 LCs on the ToFab side (Rx). If it happens, the "bump
count" counter will increase in the output of the execute-on slot
<slot> show controller tofab QM stat
command.

These drops are counted as no mem drops in the
output of the execute-on slot <slot#>
show controllers frfab QM stat command, as illustrated below:

You need to find a way to prevent the FrFab side from buffering to the
point where the LC either backs up to the inbound interface or simply drops the
packets.

A simple solution for all the line cards, except Engine 2 LCs, is to
reduce the number of buffers available to a particular outbound interface on a
multi-interface LC. By default, an interface can use all of the carved FrFab
buffers. Use the tx-queue-limit command to configure
a non-default value. This prevents the egress LC from buffering more than the
configured number of packets on the interface queue for that specific port.
Make sure that you configure this number low enough so that it does not contain
all the FrFab queues for this interface. Note that this method does not
differentiate between high and low priority packets and simply implements tail
drop more aggressively for a particular interface.

Engine 3 line cards require the use of Modular QoS CLI (MQC) instead of
the legacy Command Line Interface (CLI). This command is not supported on
Engine 2-based line cards.

Here's a configuration example using the legacy Class of Service (CoS)
configuration:

Another solution is to implement a faster output interface, which gives
us a larger pipe. But larger pipes can fill quickly. Thus, the recommended
solution is to implement Quality of Service (QoS) mechanisms on the outbound
LC.

Cisco's Weighted Random Early Detection (WRED) feature implements a
differentiated or intelligent drop mechanism. It is designed to work with
adaptive traffic, such as TCP flows. It monitors the queue size and works to
maintain a consistent average queue size by dropping packets randomly from
various flows as the calculated average queue rises above a configurable
minimum threshold.

When implemented on the Cisco 12000 Series, WRED can prevent the FrFab
queues from filling and importantly is selective about which packets it drops.
Engine 0 LCs support WRED in software, whereas Engine 1 LCs do not support WRED
at all. The other L3 Engine LCs support WRED in hardware.

Another supported QoS mechanism on the Cisco 12000 Series is traffic
policing using Committed Access Rate (CAR) on Engine 0 and Engine 1 LCs, and a
modified version of CAR known as Per Interface Rate Control (PIRC) on Engine 2
LCs. Configure traffic policing on the outbound interface.

You can check if the CPU is overloaded on the incoming LC using the
execute-on slot <slot#> show
controllers tofab queues command. If you see a very large number
in the #Qelem column of the "Raw Queue" row, it means that too many packets are
intended to be handled by the CPU (which is located on the LC itself). You will
start to get ignored packets because the CPU cannot keep up with the amount of
packets. These packets are directed to the CPU of the LC, not to the Gigabit
Route Processor (GRP)!

What you need to do at this time is shift a part of the traffic from
this inbound LC so that its CPU is less impacted.

You should also have a look at the LC configuration to check if there
are some features configured on it that impact the CPU. Some features (such as
CAR, ACL, and NetFlow) can degrade the performance of the LC when implemented
in software (only on Engine 0 LCs). If this is the case, you should act
accordingly by either removing the feature or upgrading the Cisco IOS software
to a later release where the same feature implementation is improved (such as
Turbo ACL). See
Release
Notes of the Cisco 12000 Series Routers to find out which features have
been implemented or improved for different LCs.

Finally, the only solution may be to swap the LC for a more recent one
where the requested feature is implemented in the hardware. This really depends
on the Engine type of the LC.

You can use the following shortcut command to determine the L3 Engine
type of an LC:

Use Turbo ACLs, which optimize performance by allowing the router to
compile the ACLs before downloading them to the LC processor.

Avoid using the "log" keyword on ACLs.

Avoid outgoing ACLs when possible. In a system with Engine 0, 1 and 2
LCs, all processing of ACLs is done on the inbound LC. Even outbound ACL
filtering is done on the inbound card once it knows to which outbound interface
the packet is destined. For this reason, configuring an outbound ACL on an
interface affects all LCs in the system. In addition, Engine 2 LCs can perform
either incoming or outgoing ACLs, but not both simultaneously in the ASIC that
performs hardware-forwarding. If you configure both inbound and outbound ACLs,
the LC falls back to CPU-based forwarding for outgoing access lists, impacting
the switching performance of the LC. However, newer Engines such as Engine 3
and Engine 4+ are highly optimized for enhanced IP services like ACLs and
process outbound ACLs on the outgoing LC.

Assign traffic requiring specific features to one set of
LCs.

Assign traffic not requiring features to another set of LCs to
maintain peak packet forwarding performance.

Use LCs with higher Engine types when high performance is
needed.

Design backbone- or core-facing LCs to run features supported in
hardware or microcode.

Match the oversubscribed interface indicated in the show
controllers frfab queue output with the show
interfaces output for the same interface. The following output
confirms that the output interface rate is at the line rate and is
oversubscribed:

See the solutions sections of this
document for the next steps to resolve incrementing ignored errors based on the
architecture of the particular outbound interface. For example, on an Engine 0
LC, try diverting some traffic to another interface or, as a temporary measure,
reduce the number of packet buffers that this particular interface can use from
the line card's free queues. Use the following command:

Sometimes counters increment due to a Cisco IOS software defect. Be
sure you are running the latest available Cisco IOS software release in your
train to get rid of all the bugs that have already been fixed. If you still see
ignored packets, and the information in this document does not solve your
issue, contact Cisco's Technical
Assistance Center (TAC) for assistance.