The information in this document was created from the devices in a
specific lab environment. All of the devices used in this document started with
a cleared (default) configuration. If your network is live, make sure that you
understand the potential impact of any command.

Each WS-X6348 card is controlled by a single Application-Specific
Integrated Circuit (ASIC) that connects the module to both the 32 GB data bus
backplane of the switch and to a set of four other ASICs which controls groups
of 12 10/100 ports.

An understanding of this architecture is important as it can help in
troubleshooting interface problems. For example, if a group of 12 10/100
interfaces fails the online diagnostics (refer to Step 18 of this document to
learn more about the show diagnostic module
<mod#> command), this typically
indicates one of the ASICs mentioned above has failed.

In the command output above, check the status of the module. It
could be in one of the following states:

Ok - Everything is
fine.

power-deny - Not enough power is
available to power the module.

other - Most likely the Serial
Communication Protocol (SCP) communication is broken.

faulty/unknown - This indicates
most likely a bad module or slot.

err-disabled - View the output
from the show log command (shown in Step 4) to see
if there are any messages on why the module is in the
err-disabled
state.

Verify that the configuration for the specific interface and any
global configuration that might effect the interface is correct. Ensure that
options such as spanning-tree portfast, are configured when appropriate.

Check for any interface related messages in the log by issuing the
show log command. With Integrated Cisco IOS (Native
Mode), the log can display messages from both the Switch Processor (SP) (SP =
Supervisor/Policy Feature Card (PFC)) and the Route Processor (RP) (RP = MSFC).

The following command can be used to determine the status of the
interface as well as whether the interface is configured as a Layer 3 (L3)
routed interface (the default), a trunk, or a Layer 2 (L2) switchport.

If an interface is in the notconnect state, check the cabling as
well as the device connected to the other end. If an interface is in the faulty
state, it indicates a hardware problem; issue the show diagnostic
module <mod> command for module
diagnostic results. If the interface is an L2 interface and shows the inactive
state, ensure its VLAN still exists by issuing the show
vlan command and try to shut/no shut the interface. VLAN Trunk
Protocol (VTP) problems can sometimes cause a VLAN to be deleted, which results
in interfaces associated with that VLAN becoming inactive.

The Vlan field displays routed if the interface is configured as
an L3 routed interface. It displays trunk if the interface is configured as a
trunk interface, or if the VLAN number that the interface is a member of is
configured as an L2 access switchport.

The Duplex and Speed fields have an a in front of the value
displayed (such as a-full) if the value has been obtained through
auto-negotiation. If the interface is hardcoded, the a will not be present for
those fields. While not in a connected state, an auto-negotiation-enabled
interface displays auto in these fields. Make sure that the device attached to
this interface has the same settings as this interface regarding either
hard-setting the speed and duplex or auto-negotiating the speed and duplex.

The reason (found under the Reason field) for an interface to be
placed in an err-disabled state can be any of the following:

bpduguard

dtp-flap

link-flap

pagp-flap

root-guard

udld

An error-disabled state is an operational state similar to a link
down state. You must issue the shutdown and the
no shutdown commands to manually recover an
interface from err-disable after fixing the cause of the error. An interface
displaying Reason = none implies that the interface is not currently in an
err-disabled state.

If an interface is configured as a trunk, check to make sure it is
in the correct status and that the appropriate VLANs are spanning-tree
forwarding and not pruned by VTP. For a dot1q trunk, make sure that the native
VLAN matches that of the device on the other side of the trunk.

In the above output, you can see that Fast Ethernet interface 4/2
is in the Status state of trunking and is a dot1q trunk with Native vlan = 1.
The trunking mode has been hard-set to on.

Note: While VLAN 2 exists in the Vlans allowed and active in
management domain list, it does not exist in the Vlans in spanning tree
forwarding state and not pruned list, since Fast Ethernet interface 4/2 is
actually spanning-tree blocking for VLAN 2.

Verify that dynamic Content Addressable Memory (CAM) entries are
being created for any traffic entering the L2 switchport or trunk interface you
are troubleshooting. Make sure that the CAM entry is associated with the
correct VLAN.

For L3 routed interfaces, make sure that you are learning IP routes
and Address Resolution Protocol (ARP) entries. Ensure that routing protocol
neighbors are being formed correctly through the interface in question.

Make sure CDP is enabled on the interface by issuing the command
below. If CDP is disabled on the interface, the following command will not
provide any output. You can also issue the show running-config
interface fastethernet <mod/port>
command to ensure that the no cdp enable command is
not present on the interface.

In the following example, Fast Ethernet interface 4/1 on the
Catalyst 6509 switch directly connects to Fast Ethernet interface 5/1 on
another Catalyst 6509. The neighbor Catalyst 6500 is running hybrid CatOS
6.3(9), and is named "e-6509-b." It has an IP address of 192.168.2.3. This
information was learned through a CDP version 2 advertisement.

Most non-Cisco devices as well as Cisco devices with CDP disabled
allow CDP packets to pass through them. This can sometimes lead you to believe
that two Cisco CDP enabled devices are directly connected when, in fact, they
are not. CDP uses multicast destination address 01-00-0C-CC-CC-CC, which is
typically flooded throughout the VLAN of a switch that is not CDP enabled or
that does not support CDP.

Note: The clear cdp table and
clear cdp counters commands are available and can be
used to clear the CDP table and counters if needed.

Check the state and health of the interface that is encountering
problems, and whether traffic is passing through it.

FastEthernet4/1 is up - This
indicates that the interface hardware is currently active. It can also indicate
that the interface has been taken down by an administrator by issuing the
shut interface command, if the status reads
administratively down.

line protocol is up - This
indicates whether the software processes that handle the line protocol for the
interface consider the line usable.

MTU - The Maximum Transmission
Unit (MTU) is 1500 bytes for Ethernet by default (the maximum data portion size
of a standard Ethernet frame). For jumbo frame support, the MTU can be
increased to a maximum of 9216 bytes by issuing the MTU
<bytes> interface
command.

Full-duplex, 100Mb/s - The current
speed and duplex setting of the interface. Issue the show
interfaces FastEthernet <mod/port>
status (as shown in Step 5) to determine if this setting has been
hard-set in the configuration, or obtained through auto-negotiation with the
link partner. Also make sure that the device attached to this interface has the
same settings as the interface regarding either hard-setting the speed and
duplex or auto-negotiating the speed and duplex.

Last input, output - The number of
hours, minutes, and seconds since the last packet was successfully received or
transmitted by the interface. This is useful for knowing when a dead interface
failed.

Last clearing of "show interface"
counters - The last time the clear
counters command was issued since the last time the switch was
rebooted. The clear counters command is used to
reset all of the statistics displayed through issuing the show
interfaces FastEthernet <mod/port>
command.

Note: Variables that might affect routing (for example, load and
reliability) are not cleared when the counters are cleared.

Input queue - The number of
packets in the input queue. Size/max/drops means the current number of frames
in the queue/the max number of frames the queue can hold before it must start
dropping frames/the actual number of frames dropped because the max queue size
was exceeded. The input queue size can be modified by issuing the
hold-queue <queue size> in
interface command. Be careful when increasing the size of the
queue as this may result in traffic delays because the frames get stuck in the
queue for a longer period of time.

Total output drops - The number of
packets dropped because the output queue is full. A common cause of this might
be traffic from a high bandwidth link being switched to a lower bandwidth link
or traffic from multiple inbound links being switched to a single outbound
link. For example, if a large amount of bursty traffic comes in on a gigabit
interface and is switched out to a 100Mbps interface, this might cause output
drops to increment on the 100Mbps interface. This is because the output queue
on that interface is overwhelmed by the excess traffic due to the speed
mismatch between the incoming and outgoing bandwidths.

Output queue - The number of
packets in the output queue. Size/max means the current number of frames in the
queue/the max number of frames the queue can hold before it is full and must
start dropping frames. The output queue size can be modified by issuing the
hold-queue <queue size> out
interface command. Be careful when increasing the size of the
queue as this may result in traffic delays because the frames get stuck in the
queue for a longer period of time.

5 minute input/output rate - The
average input and output rate seen by the interface in the last five minutes.
To get a more accurate reading by specifying a shorter period of time (to
better detect traffic bursts for example), issue the load-interval
<seconds> interface
command.

packets input/output - The total
error free packets received and transmitted on the interface. Monitoring these
counters for increments is useful in determining whether traffic is flowing
properly through the interface. The bytes counter includes both the data and
MAC encapsulation in the error free packets received and transmitted by the
system.

no buffer - The number of received
packets discarded because there is no buffer space. Compare with ignored count.
Broadcast storms can often be responsible for these events.

Received broadcasts - The total
number of broadcasts and multicasts received on the
interface.

runts - The frames received that
are smaller than the minimum IEEE 802.3 frame size (64 bytes for Ethernet) and
with a bad Cyclic Redundancy Check (CRC). This can be caused by a duplex
mismatch and physical problems such as a bad cable, port, or Network Interface
Card (NIC) on the attached device.

giants - The frames received that
exceed the maximum IEEE 802.3 frame size (1518 bytes for non-jumbo Ethernet)
and have a bad Frame Check Sequence (FCS). Try to find the offending device and
remove it from the network. In many cases it is the result of a bad
NIC.

throttles - The number of times
the interface requested another interface within the switch to slow down in
sending information to the interface.

input errors - This includes
runts, giants, no buffer, CRC, frame, overrun, and ignored counts. Other
input-related errors can also cause the input errors count to be increased, and
some datagrams may have more than one error. Therefore, this sum may not
balance with the sum of enumerated input error counts.

CRC - This increments when the CRC
generated by the originating LAN station or far-end device does not match the
checksum calculated from the data received. This usually indicates noise or
transmission problems on the LAN interface or the LAN itself. A high number of
CRCs is usually the result of collisions but can also indicate a physical issue
(such as cabling, bad interface or NIC) or a duplex mismatch.

frame - The number of packets
received incorrectly having a CRC error and a non-integer number of octets
(alignment error). This is usually the result of collisions or a physical
problem (such as cabling, bad port or NIC) but can also indicate a duplex
mismatch.

overrun - The number of times the
receiver hardware was unable to hand received data to a hardware buffer because
the input rate exceeded the receiver's ability to handle the
data.

ignore - The number of received
packets ignored by the interface because the interface hardware ran low on
internal buffers. Broadcast storms and bursts of noise can cause the ignored
count to be increased.

Input packets with dribble - A
dribble bit error indicates that a frame is slightly too long. This frame error
counter is incremented for informational purposes, since the switch accepts the
frame.

underruns - The number of times
that the transmitter has been running faster than the switch can
handle.

output errors - The sum of all
errors that prevented the final transmission of datagrams out of the
interface.

Note: This may not equal the sum of the enumerated output errors, as
some datagrams may have more than one error, and others may have errors that do
not fall into any of the specifically tabulated categories.

collision - The number of times a
collision occurred before the interface transmitted a frame to the media
successfully. Collisions are normal for interfaces configured as half duplex,
but should not be seen on full duplex interfaces. If collisions are increasing
dramatically, this points to a highly utilized link or possibly a duplex
mismatch with the attached device.

interface resets - The number of
times an interface has been completely reset. This can happen if packets queued
for transmission are not sent within several seconds. Interface resets can also
occur when an interface is looped back or shut down.

babble - The transmit jabber timer
expired. A jabber is a frame longer than 1518 octets (excluding framing bits,
but including FCS octets), which does not end with an even number of octets
(alignment error) or has a bad FCS error.

late collision - The number of
times that a collision is detected on a particular interface late in the
transmission process. For a 10Mbit/s port this is later than 512 bit-times into
the transmission of a packet. Five hundred and twelve bit-times corresponds to
51.2 microseconds on a 10 Mbit/s system. This error can indicate a duplex
mismatch among other things. For the duplex mismatch scenario, the late
collision is seen on the half duplex side. As the half duplex side is
transmitting, the full duplex side does not wait its turn and transmits
simultaneously causing a late collision. Late collisions can also indicate an
Ethernet cable or segment that is too long. Collisions should not be seen on
interfaces configured as full duplex.

deferred - The number of frames
that have been transmitted successfully after waiting because the media was
busy. This is usually seen in half duplex environments where the carrier is
already in use when trying to transmit a frame.

lost carrier - The number of times
the carrier was lost during transmission.

No carrier - The number of times
the carrier was not present during the transmission.

Output buffer - The number of
failed buffers and the number of buffers swapped
out.

Check that traffic counters are incrementing both inbound and
outbound on the port.

Align-Err - The number of frames
with alignment errors (frames that do not end with an even number of octets and
have a bad CRC) received on the interface. These usually indicate a physical
problem (such as cabling, bad interface or NIC), but can also indicate a duplex
mismatch. When the cable is first connected to the interface, some of these
errors may occur. Also, if there is a hub connected to the interface,
collisions between other devices on the hub may cause these errors.

FCS-Err - The number of valid size
frames with FCS errors but no framing errors. This is typically a physical
issue (such as cabling, bad interface or NIC) but can also indicate a duplex
mismatch.

Xmit-Err and Rcv-Err - These
indicate that the internal interface send (Tx) and receive (Rx) buffers are
full. A common cause of Xmit-Err might be traffic from a high bandwidth link
being switched to a lower bandwidth link, or traffic from multiple inbound
links being switched to a single outbound link. For example, if a large amount
of bursty traffic comes in on a gigabit interface and is switched out to a
100Mbps interface, this might cause Xmit-Err to increment on the 100Mbps
interface. This is because the interface’s output buffer is overwhelmed by the
excess traffic due to the speed mismatch between the incoming and outgoing
bandwidths.

Undersize - The frames received
that are smaller than the minimum IEEE 802.3 frame size of 64 bytes (excluding
framing bits, but including FCS octets) that are otherwise well formed. Check
the device sending out these frames.

Out-Discard - The number of
outbound packets chosen to be discarded even though no errors have been
detected. One possible reason for discarding such a packet could be to free up
buffer space.

Single-coll (single collision) -
The number of times one collision occurred before the interface transmitted a
frame to the media successfully. Collisions are normal for interfaces
configured as half duplex but should not be seen on full duplex interfaces. If
collisions are increasing dramatically, this points to a highly utilized link
or possibly a duplex mismatch with the attached device.

Multi-coll (multiple collision) -
The number of times multiple collisions occurred before the interface
transmitted a frame to the media successfully. Collisions are normal for
interfaces configured as half duplex but should not be seen on full duplex
interfaces. If collisions are increasing dramatically, this points to a highly
utilized link or possibly a duplex mismatch with the attached
device.

Late-coll (late collisions) - The
number of times that a collision is detected on a particular interface late in
the transmission process. For a 10Mbit/s port, this is later than 512 bit-times
into the transmission of a packet. Five hundred and twelve bit-times correspond
to 51.2 microseconds on a 10 Mbit/s system. This error can indicate a duplex
mismatch among other things. For the duplex mismatch scenario, the late
collision is seen on the half duplex side. As the half duplex side is
transmitting, the full duplex side does not wait its turn and transmits
simultaneously causing a late collision. Late collisions can also indicate an
Ethernet cable or segment that is too long. Collisions should not be seen on
interfaces configured as full duplex.

Excess-coll (excessive collisions)
- A count of frames for which transmission on a particular interface fails due
to excessive collisions. An excessive collision happens when a packet has a
collision 16 times in a row. The packet is then dropped. Excessive collisions
are typically an indication that the load on the segment needs to be split
across multiple segments but can also point to a duplex mismatch with the
attached device. Collisions should not be seen on interfaces configured as full
duplex.

Carri-Sen (carrier sense) - This
occurs every time an Ethernet controller wants to send data on a half duplex
connection. The controller senses the wire and checks if it is not busy before
transmitting. This is normal on an half duplex Ethernet
segment.

Runts - The frames received that
are smaller than the minimum IEEE 802.3 frame size (64 bytes for Ethernet) and
with a bad CRC. This can be caused by a duplex mismatch and physical problems
such as a bad cable, port, or NIC on the attached device.

Giants - The frames received that
exceed the maximum IEEE 802.3 frame size (1518 bytes for non-jumbo Ethernet)
and have a bad FCS. Try to find the offending device and remove it from the
network. In many cases it is the result of a bad NIC.

IntMacRx-Err - The IntMacRx-Err
counts non-network related errors on the MAC level, meaning the packet might
have been fine, but the frame was dropped due to internal
problems.

The output of the show spanning-tree interface
FastEthernet <mod/port> or
show spanning-tree vlan
<vlan#> commands can be used to
verify that whether a particular port is forwarding or blocking with respect to
spanning-tree protocol. Blocking ports will not forward traffic.

The show diagnostic module
<module#> command can be used to
check the results of the online diagnostic test performed at switch boot time
or when a module is reset. The results of these tests can be used to determine
if a hardware component failure has been detected on the module. It is
important to set the diagnostic mode to complete, otherwise all or some of the
diagnostic tests will be skipped. If a hardware component failure has occurred
between now and the last switch or module reset, the diagnostics must be run
again through a switch or module reset in order to detect the failure.

In order to run the diagnostic tests for a module follow these
three steps.

View the diagnostic test result for the interfaces on the module
for any indication of a failure. Also, look for failures in groups of 12
interfaces which would suggest a Coil ASIC failure or Pinnacle interface
failure.

The following is a list of commands that was used in the above
troubleshooting the WS-X6348 module connectivity issues in this document.
Please log the troubleshooting output collected using these commands before
opening up a TAC case to provide to the TAC engineer for analysis.

show version

show module
<mod#>

show running-config

show log

show interfaces fastethernet
<mod/port> status

show interfaces fastethernet
<mod/port> trunk

show interfaces fastethernet
<mod/port> switchport

show mac-address-table dynamic interfaces fastethernet
<mod/port>

show spanning-tree interfaces fastethernet
<mod#/port>

show ip route

show ip arp

show ip [eigrp/ospf]
neighbors

show cdp neighbors fastethernet
<mod/port> detail

Repeat the following five commands three times to monitor counter
increments (Steps 12-16 only):

show interfaces fastethernet
<mod/port>

show interfaces fastethernet
<mod/port> counters

show interfaces fastethernet
<mod/port> counters
errors

show interfaces fastethernet
<mod/port> counters
trunk

show interfaces fastethernet
<mod/port> counters broadcast

diagnostic level complete (global
configuration command)

hw-module module <module#>
reset

show diagnostic module
<mod#>

Below is list of additional commands which can be collected before
opening up a TAC case for further troubleshooting by the TAC engineers or
development engineers. These commands are hidden commands and should be used
exactly as shown for troubleshooting the WS-X6348 module issues by the TAC
engineers. You can alternatively provide these commands at the request of the
TAC engineer handling the case.