This document explains how to read the output of the show
controller frfab queue and show controller tofab
queue commands. It also gives a detailed overview of the
underlying architecture of the Cisco 12000 Series Internet Router related to
these special queues.

The information presented in this document was created from devices in
a specific lab environment. All of the devices used in this document started
with a cleared (default) configuration. If you are working in a live network,
ensure that you understand the potential impact of any command before using
it.

This document focuses exclusively on the Packet Memory which is divided
into two banks: ToFab and FrFab (towards the Fabric and from the Fabric). The
ToFab memory is used for packets that come in one of the interfaces on the LC
and make its way to the fabric, whereas the FrFab memory is used for packets
that are going out an interface on the LC from the fabric.

Note: "ToFab" (towards the Fabric) and "Rx" (received by the router) are
two different names for the same thing, as are "FrFab" (From the Fabric) and
"Tx" (transmitted by the router). For example, the ToFab Buffer Management ASIC
(BMA) is also referred to as the RxBMA. This document uses the ToFab/FrFab
convention, but you may see the Rx/TX nomenclature used elsewhere.

Access to packet memory is made through the Buffer Management ASIC
(BMA). The BMA provides packet buffering and buffer queue management functions
to the line card. All packets pass two times through the BMA - once coming in
and once going out. In other words, packets arrive on a physical layer
interface module (PLIM), spend a short amount of time in SDRAM buffers, and are
then read out of the buffers and delivered to the Fabric Interface ASIC (FIA)
module. Here, they are segmented into Cisco cells and transmitted to the switch
fabric. The packets are then received from the switch fabric by the Fabric
Interface ASIC on the egress line card. They are reassembled, go to SDRAM
buffers, then to the PLIM, and finally sent on the wire.

Cisco IOS software implements a buffer-carving algorithm that divides
SDRAM into various sized buffers. The GRP and other sources provide carving
instructions to the line card, which then executes the instructions. There are
different types of carves. For example, a simple carve creates a pool of
same-sized buffers, while a complex carve creates multiple pools of different
sizes, with each pool containing buffers of the same size.

All buffers of the same size are associated in one pool. One pool is
always allocated for Inter-Process Communication (IPC) usage. Each associated
Queue Static RAM (QSRAM) is updated with the queue head, tail, length, length
threshold, associated buffer addresses in SDRAM, and the next queue element.

The following sequential conditions trigger a buffer carving on a line
card:

Cisco IOS software image in place - LC simple carve call to enable
Inter-Process Communication (IPC) so that the GRP can use IPCs to give the LCs
the initial carve specification. All the SDRAM available for carving is
recarved.

Once IPC is up - Using IPCs, the GRP can call an LC complex carve
multiple times to dynamically recarve all SDRAM.

A manual configuration or change of the MTU (Maximum Transmission
Unit) on one interface causes the memory to be recarved. FrFab queues are
carved up to the maximum MTU of the whole system, whereas the ToFab queues are
carved up to the maximum MTU of the particular line card.

Note: We only recarve if we change the maximum MTU
for the line card (ToFab queues), or if we change the
maximum MTU for the whole system (FrFab queues). For
instance, changing the MTU from 1500 to 4470 doesn't change anything if there
is already an interface with MTU 4470 on that line card (ToFab queues) or on
the whole system (FrFab queues).

There are now four carves for the FrFab queues and the maximum buffer
size has changed to 10064 bytes.

Note: On Packet Over Sonet (POS) line cards configured with Point-to-Point
Protocol (PPP) encapsulation, Maximum Receive Unit (MRU) negotiation does
occur, but it does not adjust the MTU size. Moreover, the PPP connections are
not reset when the MTU is changed on the interface.

This memory is carved into different pools of packet buffers. To see
how the receive memory is carved, you can attach to
a Line Card and execute the show controller tofab
queue command, as demonstrated below:

40606/40606 buffers specified/carved - Buffers to be
carved specified by Cisco IOS software and the number of buffers actually
carved.

non-IPC free queues - The non-IPC buffer pools are
the packet buffer pools. Packets arriving into the line card are allocated a
buffer from one of these buffer pools depending on the size of the packet. It
is possible to have only three non-IPC free queues; if the board is Ethernet,
you will not have the 4k pool, but only a pool up to 1.5k. This is because the
ToFab queues are carved up to the maximum transmission unit (MTU) of that
particular line card. The example output shows five packet buffer pools of
sizes 80, 608, 1568, 4544, and 9248 bytes. For each pool, more details are
given below:

#Qelem - The number of buffers that are currently
assigned to that queue. If it is a free queue, then these buffers are available
to the system. If it is a ToFab queue or a transmit queue, these buffers are
not available to the system. This is the column to check to find out which
queue is backed up.

Head and Tail - A head and tail mechanism is used
to ensure the queues are moving properly.

IPC Queue - Reserved for Inter-Process Communication
messages from the LC to the GRP.

Raw Queue -When an incoming packet has been assigned
a buffer from a non-IPC free queue, it's enqueued on the raw queue. The raw
queue is a First In, First Out (FIFO) processed by the LC CPU during
interrupts. If you see a very large number in the #Qelem column of the "Raw
Queue" row, you have too many packets waiting on the CPU and they will start
getting ignored because the CPU can't keep up with the load. However, this is
very rare.

ToFab Queue - Virtual output queues; one per
destination slot plus one for multicast traffic. The last portion of the
previous example shows 15 virtual output queues. This is a 12012 router, which
was originally designed as a 15-slot chassis; queues 13 through 15 are not
used.

After the ingress line card CPU makes a packet switching decision, the
packet is enqueued on the virtual output queue corresponding to the slot where
the packet is destined. The number in the fourth column is the number of
packets currently enqueued on a virtual output queue.

Step 1 - A packet comes into the physical layer
interface module (PLIM). As the packet is received and processed, it is DMA'd
(Direct Memory Access) into a small (approximately 2 x Maximum Transmission
Unit (MTU) buffer) memory called the "First In, First Out (FIFO) burst memory".
The amount of this memory depends on the type of LC (from 128 KB to 1 MB).

Step 2 - When the packet is completely in FIFO memory,
an application-specific integrated circuit (ASIC) on the PLIM contacts the
Buffer Management ASIC (BMA) and asks for a buffer to put the packet in. The
BMA is told what size the packet is, and allocates a buffer accordingly. If the
BMA cannot get a buffer of the right size, the packet is dropped and the
"ignored" counter is incremented on the incoming interface. There is no
fallback mechanism as with some other platforms.

Step 3 - While this is going on, the PLIM may be
receiving another packet in the FIFO burst memory, which is why it is 2xMTU in
size. If there is a free buffer available in the right queue, the packet is
stored by the BMA in the free queue list of the appropriate size. This buffer
is placed on the Raw Queue, which is examined by the Salsa ASIC or the R5K CPU,
depending on the line card switching engine type.

Step 4 - On the engine 0 LC, the R5K CPU determines
the destination of the packet by consulting its local Distributed Cisco Express
Forwarding (dCEF) tables in DRAM. It then moves the buffer from the Raw Queue
to a ToFabric queue corresponding to the destination slot. If the destination
is not in the dCEF tables, the packet is dropped. If the packet is a control
packet (for example, routing updates), it is enqueued to the queue of the GRP
and is processed by the GRP. On a 12016 router, there are 17 ToFab queues (16
unicast, plus one Multicast).

Step 5 - The ToFab BMA enqueues the buffer into the
proper ToFab queue. At this point, the #Qelem counter in the pool the buffer
came from decreases by one, and the ToFab queue counter increases by one.

Note: There is one ToFab queue per line card (this includes the GRP). These
queues are known as Virtual Output Queues (VOQs). These are important for
avoiding head-of-line blocking.

Step 6 - The Fabric Interface ASIC (FIA) sees that an
output queue is non-empty. The FIA is set up to segment the packet into 48-byte
cells. An 8-byte header is added onto the packet and the 56-byte Cisco cell is
sent across the switch fabric.

Transmit packet memory stores packets coming from the switch fabric and
awaiting transmission to the physical interface. This memory is also carved
into pools of various sizes.

From the GRP, you can attach to a line card
and execute the show controller frfab queue command
to display the transmit packet memory. In addition to the fields in the ToFab
output, the FrFab output displays an "Interface Queues" section. The output
varies with the type and number of interfaces on the outgoing LC.

One such queue exists for each interface on the line card. Packets
destined out a specific interface are enqueued onto the corresponding interface
queue.

The following list describes some of the key fields that are found in
the previous example:

Non-IPC free queues: These queues are packet buffer
pools of different sizes. When a packet is received over the fabric, an
appropriate sized buffer is taken from one of these queues, the packet is
copied into it, and the buffer is placed on the appropriate output interface
queue.

Note: There are as many pools as needed for the whole router. As a
consequence, FrFab queues are carved up to the maximum MTU of the whole system.
This is different for the ToFab queues which are carved up to the maximum MTU
of the particular line card.

IPC queue: Reserved for Inter-Process Communication
messages from the GRP to the LC.

Interface queues: These queues are for the
interfaces, not for the slot numbers. The last number (65535) is the
TX-queue-limit. This number controls the maximum length of any queue and can be
tuned by the TX-queue limit command on the Engine 0
line card. If you experience some congestion, this command can be used to
prevent the egress LC from buffering more than the configured number of packets
on the interface queue for that specific port. Make sure that you configure
this number low enough so that it does not contain all the FrFab queues for
this interface. However, this tuning provides no control over which packets get
dropped on the outbound LC. See
Troubleshooting Ignored
Packets and No Memory Drops on the Cisco 12000 Series Internet Router
for details.

At this point, the Cisco cells have been transmitted over the switch
fabric by the FIA.

Step 1 - These Cisco cells are DMA'd into FIFOs on the
FrFab FIAs, and then into a buffer on the FrFab BMA. The FrFab BMA is the one
that actually does the reassembly of cells into a packet.

How does the FrFab BMA know in which buffer to put the cells before it
reassembles them? This is another decision made by the incoming line card
switching engine. Since all queues on the entire box are the same size and in
the same order, the switching engine tells the transmitting LC to put the
packet in the same number queue from which it entered the router.

The FrFab BMA SDRAM queues can be viewed with the show
controller frfab queue command on the LC.

Step 2 - This step is basically the same as the ToFab
BMA output. Packets come in and are placed in packets that are dequeued from
their respective free queues. These packets are placed into the FrFab queue,
and enqueued on either the interface queue (there is one queue per physical
port) or the rawQ for output processing. Not much happens in the rawQ: per-port
multicast replication, Modified Deficit Round Robin (MDRR) - same idea as
Distributed Weighted Fair Queuing (DWFQ), and output Committed Access Rate
(CAR). If the transmit queue is full, the packet is dropped and the output drop
counter is incremented.

Step 3 - The FrFab BMA waits until the TX portion of
the PLIM is ready to send a packet. The FrFab BMA does the actual Media Access
Control (MAC) rewrite (based, remember, on information contained in the Cisco
Cell header), and DMAs the packet over to a small (again, 2xMTU) buffer in the
PLIM circuitry. The PLIM does the Asynchronous Transfer Mode (ATM) segmentation
and reassembly (SAR) and Synchronous Optical Network (SONET) encapsulation,
where appropriate, and transmits the packet.