show fifo event-history

This command is primarily intended for internal development use. It displays a log of the most recent messages generated by the Packet First-In-First_Out (FIFO) driver. The FIFO driver is the 16-Bit communication path (or interface) between the ACE BCM1250 and CDE.

The output displays the raw hexadecimal (hex dump) of the packets sent through this driver.

This command is used in conjunction with the diagnostic command "debug fifo ...", which has the following command syntax:

You can see the MAC/IP/TCP headers in this packet starting at "00 00 0c 07....". The types of kernel Packet FIFO packet that will be displayed are:

PKT-FIFO TX PACKET (CTRL)

PKT-FIFO TX PACKET (DATA)

PKT-FIFO RX PACKET (CTRL)

PKT-FIFO RX PACKET (DATA)

Caution:

When the debug fifo command diagnostic logging is enabled it can generate huge amounts of data in the debug window. Please use care when enabling the debug fifo command. Generally, this command should be used only while in contact with TAC.

show fragment

Shows statistics related to IP fragmentation and reassembly activities by the ACE. An IP fragment results from datagrams that are broken into smaller pieces because they are larger than the maximum MTU permitted by a traversed link. ACE can generate fragments or reassemble datagrams that arrive as fragments, when needed.

show ft history cfg_cntlr

This command displays internal messages generated by the ACE when performing a bulk synchronization operation to an FT standby. It is helpful when troubleshooting config sync issues. Errors indicated in the output can be further investigated using the command "show ft config-error".

This log wraps so it will not use up too much disk space and more importantly the output in the list is preserved over a reboot of the ACE.

While primarily used for internal debugging, this command shows information that can help general troubleshooting as well. For instance, if you have an FT-related problem involving ACE peer synchronization, you would typically be asked to invoke the show tech-support details command. The output of the show tech-support details command would include this FT log and others.

The item: "ha_dp_write_to_table:298" refers to the actual ACE code and "ha_dp_write_to_table" is a routine in the FT process which is executing and printing out information for DE to debug with. For this reason except for obvious errors which are self-explanatory, this log is typically only meaningful to internal development.

show ft history ha_mgr

Displays the HA manager debug log. This information can be useful for checking for correct HA state transitions in the device.

When high availability is enabled, the HA managers on both peers start an internal HA state machine and exchange redundancy protocol packets. Based on the priority configured for both sides, HA managers will go through the states transition, including non-redundant, election, standby-config, standby-bulk and eventually move its state to active and standby-hot state.

Configuration synchronization plays a very important role in the redundancy model. For ACE and ACE appliance, it can be detailed in two parts: bulk config-sync and incremental config-sync.

This event is posted when the peer device is up but the FT VLAN is down.

FSM_EV_FT_STATE

This event is posted when an STATE_UPDATE message is received from the peer.

FSM_EV_TIMEOUT

This event will indicate a timer expiry.

FSM_EV_CFG_SYNC_DONE

This event is posted to the FSM when receiving the configuration data is completed.

FSM_EV_BULK_SYNC_DONE

This event is posted to the FSM when receiving the bulk sync data is completed.

FSM_EV_COUP

This event is posted when an COUP message is received from the peer.

FSM_EV_RELINQUISH

This event is posted when an RELINQUISH message is received from the peer.

FSM_EV_TRACK_STATUS

This event is posted when the status of a tracked object is changed

show ft memory

Shows the number of memory allocations as well as the number of bytes of memory used by those allocations for various HA components. This command shows information about tracked memory consumed by HA and the internal libraries it uses.

show ft memory detail

This command is a more verbose version of the show ft memory command. These commands track the memory usage of the Fault Tolerant (FT) application. This detailed form of the command lists the actual name of the FT application doing the memory allocation. The first are those FT application compiled with the MTRACK debugging feature while the second group are non-MTRACK users which are lower level library files.

This command can be used to look for memory leaks within the FT subsystem of the ACE.

FSM_PEER_STATE_ERROR – Error has occurred as part of State Machine progression such as Version Mismatch and failure in establishing TCP connection to peer.

FSM_PEER_STATE_TL_ERROR – TL Connection failure

Maintenance mode

May be one of the following:

MAINT_MODE_FULL – All contexts on the service blade will become non-redundant causing their peer contexts to switch over to ACTIVE. This mode will be seen right before re-setting/rebooting the service blade mainly for performing hitless upgrades

MAINT_MODE_PARTIAL – All STANDBY contexts will transition to STANDBY_COLD state. This mode is entered if configuration sync failure occurs

MAINT_MODE_OFF – Maintenance mode is off

Heartbeat Interval

Interval in seconds

Heartbeat Count

Consecutive misses before declaring a failure

TL_CLOSE count

Number of TL_CLOSE messages received from peer

FT_VLAN_DOWN count

Number of times FT vlan went down

PEER_DOWN count

Number of times peer was declared down

SRG Compatibility

Indicates whether the software versions of the local ACE and the peer ACE are compatible. Possible states are:

The counter of interest here is "Number of Unidirectional HB's Received". This is the number of heartbeats received by the local peer that indicate the remote peer is not receiving HB signals. In other words, the remote peer is sending heartbeats, but not receiving any. Note that both peer modules send heartbeat packets and each packet indicates whether the other peer has been receiving heartbeats.

Event messages sent. This is expected. (Note that a corresponding WrkThread's "Msg Received" counter will be incremented for each "Msg Sent" event.)

Msg Dropped

The incrementing of this counter may indicate a problem. Contact Cisco TAC for more information.

Msg Received

Messages received.

Good

Incrementing of this counter is expected upon completion of a probe. (Note: A corresponding WrkThread's “Msg Send” will be incremented as well.)

Bad Opcode

You should contact Cisco TAC if this field is incrementing.

Bad Event Instance

You should contact Cisco TAC if this field is incrementing.

Msg Allocated

Messages allocated.

Msg Allocate Failed

Number of times message allocation failed.

Msg Freed

Messages freed

Total Events

The total number of events.

Total Events Skipped

This event may occur when a probe is still being run and hence the internal event manager does not send a message to WrkThread (i.e., it skips the probe for that interval time). For example, this is possible if the probe interval is less than the open/receive timeout, such as: probe interval (2sec) < open/receive timeout (5sec).

If the server is unreachable, the probe will continue to wait for 5 seconds before it sends a response back to event manager. The event manager in the meantime will skip the probe twice (at 2sec and 4sec).

show hm-internal wrkthread-stats

This command shows information on activity of the health monitor process by thread. Threads 1 and 2 are for generic TCP and UDP probes. Thread 3 is for ICMP probe. Thread 4 is for SNMP probe and thread 5 is for scripted probes.

In addition to threads, an event scheduler exists which is mainly responsible for firing probes at configured intervals. The worker threads communicate to the event manager via a pipe.

Communication statistics between worker threads and the event Manager:

Msg sent

This value is the number of messages sent by the respective worker thread to the event manager.

Msg Dropped

This is value represents number of errors encountered during communication from a worker thread to the event manager.

Msg Dropped (%)

The sum of Msg Sent and Msg Dropped represents the total number of communication messages sent from the worker thread to the event manager. This value represents the Msg dropped counter as a percentage. Its dervied as follows (Msg Dropped * 100)/(Msg sent + Msg Dropped).

Msg Received

This value is the number of messages received from the event manager to the respective Worker thread. The format of the event manager message has OPCODE, probe related data as a part of the message. The following errors pertain to the parsing of the above fields.

Usage

This represents the number of messages received with "Usage OPCODE" in the OPCODE field. It is unused.

Bad Opcode

The number of messages with invalid OPCODE, which is not recognized by the worker thread.

Cancel Probe

The number of messages with "Cancellation OPCODE" from the event manager. Its not used.

Run Probe

The number of valid messages received from the event manager. The worker thread can process this message effectively.

Run Probe(Bad Data)

The number of message which does not contain valid probe related data.

Errors/counters related to memory (A message is sent back to the event manager after a worker thread processes the probe. This message is allocated by the worker thread and, under normal circumstances, freed in the event manager.)

Msg Allocated

This value is the number of messages that are successfully allocated.

Msg Allocate Failed

This value is the number of failures during message allocation.

Msg Freed

This value represents the number of messages that were freed by the event manager. A probe that is being run is represented in the form of a qnode. The qnodes are created after receiving a message from the event manager and are destroyed after a message is sent back to the event manager with the probe result.

Qnode Created

This value is the number of qnodes that are successfully allocated.

Qnode Destroyed

This value is the number of qnodes that are successfully freed.

Error Create Qnode

This value is the memory allocation failure encountered during the qnode creation. For optimization purposes, these qnodes are stored in an array for ICMP probes and as a queue for non-ICMP probes. Since ICMP qnodes are an array, it has a capacity to hold 8K probes at any given point of time.

Error Add ICMP qnode

This value is encountered when the number of simultaneous ICMP probes exceeds 8K.

Qnode in All Queues

This value is the number of qnodes that are in queue currently in that thread.

Qnode in queues < 0

This value is incremented, when a qnode is getting destroyed twice. In the normal conditions it should not be encountered.

Qnodes cant be deleted

This value is incremented, if a qnode we are trying to delete is not in the queue. Under normal conditions, this should not be encountered.

Error connect Probe

For TCP/UDP based probes, every probe run creates a socket. If there is any error in the creation of a socket or a subsequent connect with the real server, this error is incremented. Its usually seen in scalable configurations.

Miscellaneous statistics (During a probe run, after socket creation the corresponding qnodes are put in a queue for optimization. The value of the socket is checked for sanity. These counters are incremented using that. These values are for purely internal consumption and should not be considered during debugging.)'

Active Sockets

This number indicates the number of sockets that are currently opened.

show hyp stat

Available only from the admin context, this command shows contents of some of the more useful Hyperion registers. Hyperion is the ASIC that passes data between the "ten gig ethernet" (that is, the data path on the cat, which is either 8 GB or 20 GB wide) and the CDE (classification and distribution engine) FPGA which directs traffic to/from all the components of the ACE.

The registers are cleared when read. Thus, every time you read them by issuing this command, you will get the difference since the last time the command was issued.

The sum of all errors that prevented the receipt of a packet (or datagrams) and include CRC, Overrun, Underrun and Aborted Frames.

unknown

The number of packets dropped on input because of an unknown protocol.

ignored

Number of received packets ignored by the VLAN because the interface hardware ran low on internal buffers.

unicast RPF drops

The number of UNICAST packets which were dropped due to the "Unicast Reverse Path Forwarding (RPF)" feature being able to verify the IP source address. Related to certain type of Denial of Service (DOS) attacks on the network.

OUTPUT or Transmitted on the VLAN interface

unicast packets output

Packets transmitted to a UNICAST address on this VLAN.

bytes

The number of UNICAST bytes transmitted on this VLAN.

multicast

The number of MULTICAST packets transmitted on this VLAN.

broadcast

The number of BROADCAST packets received on this VLAN.

output errors

The sum of all errors that prevent a packet from being transmitted on this interface and includes CRC, Overrun, Underun and Aborted Frames.

ignored

Number of packets failed to be transmitted by the VLAN because the interface hardware ran low on internal buffers.

show interface internal iftable

This command displays information about the control plane interface table.

show interface internal vlantable

Displays VLAN-related system information for all or a specified VLAN. VLAN information is displayed one line for every possible VLAN ID 1-4094. If you specify a VLAN number, information for just that VLAN is displayed.

The output includes details for both control plane and data plane traffic. To view only the CP traffic, you can load the dplug and issue the netstat -s command.

The specific reasons for the "couldn't reassemble" counter being incremented can be shown using the show np 1/2 me-stats -sreass | inc Drop command. You can get a breakdown of the total fragment errors by issuing the "show frag" command.