Customers often contact
Cisco
Technical Support when they notice one or more of their switch ports
have become error-disabled; that is, the ports have a status of errDisable.
They want to know why this happened and how the ports can be restored to
normal. This document describes what the errDisable state is, how to recover
from it, and provides two examples of recovering from errDisable. Throughout
this document, the terms errDisable and error-disable are used interchangeably.
(errDisable is the status of a port as shown by the show
port command, error-disable or error-disabled are the English
language equivalents of errDisable.)

The information in this document is based on these software and
hardware versions. You need these in order to create the examples in this
document:

Two Catalyst 4000/5000/6000 family switches (or their equivalent) in
a lab environment with cleared configurations. Our primary machine was a
Catalyst 5500 running CatOS 5.4(2). This was connected to a Catalyst 6509
running 5.3(5a)CSX, but could be any CatOS machine and capable of EtherChannel
and portfast.

Two RJ-45 Ethernet crossover cables.

CatOS 5.4(x) on at least one switch.

Two FastEthernet ports in each switch capable of EtherChannel and
portfast.

A terminal connection to one or both of the switches.

The information in this document was produced from an isolated lab
environment. Ensure that you first understand the potential impact of any
command on your network before using it. The clear config
all command was entered on each switch to ensure a default
configuration. Should you want to replicate and experiment with these errors,
please only try to duplicate them in an isolated environment that will not
impact your live network. These examples are for instruction only. Output from
some commands has been truncated where it does not enhance the discussion.

The information in this document was created from the devices in a
specific lab environment. All of the devices used in this document started with
a cleared (default) configuration. If your network is live, make sure that you
understand the potential impact of any command.

The errDisable feature was first implemented in CatOS release 3.2(2).
If the configuration showed a port to be enabled, but software on the switch
detected an error situation on the port, the software would shut down that
port. In other words, the port was automatically disabled by the switch
operating system software because of an error condition encountered on the
port.

When a port is error-disabled, it is effectively shut down and no
traffic is being sent or received on that port. The port LED is set to the
color orange and when you enter the show port
command, the port status shows errdisable. Here is an example
of what an error-disabled port would look like from the command line interface
of the switch.

The error-disable function serves two purposes. First, it lets the
administrator know when and where there is a port problem. Second, it
eliminates the possibility that this port could cause other ports on the module
(or the entire module) to fail due to buffers being monopolized by the bad
port, port error messages monopolizing inter-process communications on the
card, even ultimately causing serious network issues. The error-disable feature
helps prevent these situations.

At first, this feature was implemented to handle special collision
situations where the switch detected excessive or
late collisions on a port. Excessive
collisions occur when a frame is dropped because of encountering 16
collisions in a row. Late collisions occur after every
device on the wire should have recognized that the wire was in use. These types
of errors could be caused by a cable that is out of specification
(too long, wrong type, defective), a bad network interface card (NIC)
card (with physical problems, or driver problems), or a
port duplex misconfiguration. This last cause is common
because of failures to negotiate the speed and duplex properly between two
directly connected devices (for example, a NIC card connected to a switch).
Only half-duplex connections should ever have collisions in a LAN; due to the
Carrier-Sense Multi-Access (CSMA) nature of Ethernet, collisions are normal for
half-duplex, as long as they do not exceed a small percentage of traffic.

As the capabilities of the CatOS grew, there were more ways that a port
could become error-disabled. For example on the catalyst 6500 running catOS,
the Errdisable feature is supported for these connectivity issues:

ARP inspection

Broadcast suppression

BPDU port-guard

Channel misconfiguration

Crossbar failure

Duplex mismatch

Layer 2 protocol tunnel misconfiguration

Layer 2 protocol tunnel threshold exceeded

UDLD

The error-disable function allows the switch to shut down a port when
it encounters any of these situations. Remember, a port being error-disabled is
not by itself a cause for alarm, as long as one determines and resolves its
root cause. An error-disabled port is a symptom of a deeper problem that must
be resolved.

Identify and fix whatever caused the ports to become error-disabled
(cable, NICs, EtherChannel, and so on).

If you do not identify and fix the underlying issue that caused the
ports to be error-disabled, then the ports will just become error-disabled
again when the problem reoccurs. Some errors can occur quite often (an example
is the error detected by BPDU portguard, which can occur every two seconds). If
you tried to reenable the ports without fixing the source of the problem they
would just become error-disabled again.

Reenable the port.

Just fixing the source of the problem will not cause the ports to
become enabled again. Once you fix the source of the problem, the ports are
still disabled (and the port LEDs are still orange); the ports must be
reenabled before they will become active. At first the only way to reenable the
port was to manually enter the set port enable
command for the ports in question. Over time there have been optional
extensions added to the error-disable feature to make it more flexible and
automatic.

Note: An error-disabled port is not the only reason a port LED could go
orange; it is only one of the reasons. That is why it is always good to check
the port status with the show port command.

Some customers wanted to have the ability to determine whether a port
should be shut down due to special collision errors discovered by the CatOS.
There were some situations, such as if the link was a backbone connection, for
example, where shutting down the ports would actually be worse than the errors
that were encountered on the ports; it would be more desirable to leave the
ports functioning as much as possible until the problem could be addressed,
rather than shutting them down. So in release 4.2(2), a new command was added
to the CatOS called set option errport that allows
the administrator to determine what action the switch took upon discovering a
port having these special collision errors. The original and default state is
set option errport disable,
where the switch will put a port in error-disabled state when encountering the
error-disable type of special collision errors. In contrast, if the command
set option errport enable is
used, then the switch will leave the ports enabled, even though it encounters
collision errors that would normally disable those ports.

This command affects the switch globally; it cannot be issued for an
individual port. It is not listed in the command reference, but is listed in
the release notes for 4.2(2) (Release
Notes for Catalyst 5000 Family Software Release 4.x). Please note that
this command appears to be counter-intuitive; one must enable the errport
option to disable the err-disable feature (enabled by default). Put more
clearly, simply use the set option errport
enable command to prevent a port from
becoming error-disabled.

The set option errport command is only
recommended if you realize that you incur some risk of other ports on the
module being affected if you allow these error conditions to continue. It is
only a stop-gap measure, not a "fix" to the problem; it merely prevents the
ports that are encountering these errors from being shut down until you can
address the real problem. Use with caution.

With CatOS release 5.4(1), a new command called set
errdisable-timeout is introduced. This command is a more
sophisticated version of the set option errport
command discussed earlier. This command will automatically
reenable an error-disabled port after a configurable amount of time (from 30
seconds to 24 hours, specified in seconds), eliminating the need to manually
reenable the error-disabled port.

This command will affect the ports that are enabled by the current
configuration on the switch but have been put into the error-disable state by
the CatOS software. Use the command show errdisable-timeout
to see the current status of the errdisable-timeout feature. It
is possible to specify five separate areas where this feature can be enabled:
bpdu-guard, channel-misconfig, duplex-mismatch, (which includes the special
collision errors mentioned above), udld,
other. This way it can still give you permanent
error-disable protection in areas where you want it, but allow you to
selectively pick areas where you would rather have the ports keep functioning
until you can fix the problem.

In software versions 5.2.1 and 5.2.2 for the Catalyst 6000 series,
there is a software defect that causes network outages when a port changes
state to error-disabled. When a ports goes errDisable, the switch will cause
all learned MAC address to be inadvertently learned on the error-disabled port.
This will cause the network outages on the associated VLAN. This software
defect has Cisco bug ID CSCdm48887 and the issue is resolved in software
versions 5.2.3 and later.

The short-term workaround for preventing this issue is as follows:

Issue the command set option errport
enable to disable the error-disabled feature.

Re-enable all error-disabled ports using the set port
enable mod_num/port_num command.

At this point in the document, we provide two examples of how you might
encounter an error-disabled port and how to fix them; a brief discussion of
three other reasons that a port could become error-disabled; and a summary of
the commands discussed relating to error-disabled ports. The specific examples
shown below for these issues are easy to duplicate in a lab environment.

Use these steps in order to recover a port from errDisable
state:

Version of Software Used in this Document

The show version command displays the
software version the switch is running for this document. This is here just to
show what version of CatOS we were using for this test and what modules were
involved.

Note: When a port is error-disabled, the LED associated with the port
on the front panel is solid orange.

How to Determine the Reason for the Error-Disabled State (console
messages, syslog, show errdisable-timeout)

When the switch puts a port in the error-disabled state, it sends a
message to the console and describes why the port was disabled. These are two
sample messages that show why a port is disabled: one from the portfast
BPDU-guard feature, and another from an EtherChannel configuration problem.

Note: The messages do not explicitly state errDisable or
error-disabled; however, they do indicate that the switch is disabling the
port. After the console messages are generated, they are not saved, unless you
utilize a syslog server in your network. If you configure the switch to send
these messages to a syslog server, then you will have a more permanent record
of when and why the port was disabled. For information on how to configure your
switch to send messages to a syslog server, see the document
Configuring
System Message Logging in the CatOS 5.4 Configuration Guide.

If you are running CatOS 5.4(1) or later, there is a feature called
errdisable-timeout which, if enabled, tells you why a port was disabled. This
is an example

How to Correct the Problem. After you discover why the ports were
disabled, you should first fix the root problem, then reenable the port.

Fix the Root Problem

This depends on what the triggering problem actually is. There
are numerous things that could trigger the shutdown. These are some of the most
noticeable and common causes.

EtherChannel Misconfiguration

For EtherChannel to work, the ports involved must have
consistent configurations; the same VLAN, same trunk mode, same speed, same
duplex, and so forth. Most of the configuration differences within a switch are
caught and reported when you create the channel. In some situations, usually
when you use the ON mode (as opposed to auto or desirable), everything can be
consistent on one switch so that switch starts channeling. But, the connected
neighboring switch cannot be set the same and can cause the first switch to
become error-disabled. If both of the switches support Port Aggregation
Protocol (PAgP), you can configure the channel modes on each switch to be
desirable instead of on in order to avoid this problem.

Duplex Mismatch

Duplex mismatches are common because of failures to
auto-negotiate speed and duplex properly. Unlike with half-duplex, which must
wait until no other devices are transmitting on the same LAN segment, a
full-duplex device will transmit whenever it has something to send, regardless
of other devices. If this transmission occurs while the half-duplex device is
transmitting, the half-duplex device will consider this either a collision
(during the slot time), or a late collision (after the slot time). Since the
full-duplex side never expects collisions, it will never realize that it must
retransmit that dropped packet. A low percentage rate of collisions are normal
with half-duplex, but not with full-duplex. If the switch port receives a lot
of late collisions, this usually indicates a duplex mismatch problem; make sure
ports on both sides of the cable are set to the same speed and duplex. The
show port command will tell you the speed and duplex
for Catalyst switch ports. Later versions of Cisco Discovery Protocol (CDP) can
warn you about a duplex mismatch before the port is actually put in
error-disable state. In addition, there may be settings on a NIC card that
cause the problem (things like auto polarity features - if in doubt, turn them
off). If you have multiple NIC cards from a vendor and they all appear to have
the same problem, check the manufacturer's web site for release notes and make
sure you have the latest drivers from the NIC manufacturer. Other causes for
late collisions include a bad NIC (with physical problems, not just
configuration problems), a bad cable, or a cable segment that is too long.

Some newer versions of switch software monitor if portfast is
enabled on a ports. A port using portfast should be connected to an
end-station, not to devices that generate STP packets called BPDUs. If the
switch notices a BPDU coming in a port that has portfast enabled, it will put
the port in errDisable mode.

UDLD

UDLD is a protocol on some new versions of software that
discovers if communication over a link is one-way only, and therefore partially
broken. A damaged fiber cable or other cabling/port issue could cause this
one-way only communication. Spanning tree loops can occur with this problem.
UDLD allows the port to detect a unidirectional link, and can be configured to
put a port in errDisable state when it detects this condition.

Other

Any process within the switch that recognizes a problem with
the port can place it in the error-disable state.
Look at the console messages or the message that were sent to a syslog server
that state why the port is being shut down. Also, if the errdisable-timeout
feature is enabled (minimum CatOS 5.4(1)), the show errdisable-timeout will
tell you the general reason that the port was disabled.

Reenable the Port

After you fix the root problem, the ports will still be disabled;
you must reenable the ports. This can be done manually using the
set port enable command.

Cat5500> (enable) set port enable 11/1-2
Ports 11/1-2 enabled.

If you have CatOS 4.2(2) or later, one can use the
set option errport command as described above to
prevent ports from becoming error-disabled. Since you are not actually fixing
the source of the problem this can be risky. If you have CatOS 5.4(1) or later,
you can use the errdisable-timeout command to
automatically reenable the ports as described in the next section.

How to Reenable the Port Automatically Using errdisable-timeout -
CatOS 5.4(1)

The errdisable-timeout command allows
you to selectively pick which type of errors will automatically reenable the
ports after a specified amount of time. The output shows the default state
which is errdisable-timeout disabled (not active) for all five possible
conditions. If any condition was enabled, the ports with this condition would
be reenabled after 30 seconds.

A nice feature of this command is that if you enable
errdisable-timeout, it will list generally why the ports have been put into
error-disable state. For more detailed descriptions, you must refer to the
messages displayed at the time of occurrence. Remember that the first step in
fixing the error-disable condition is to fix the original error that brought
about the shutdown. Notice below that the reason port 11/1 was shut down was
because of the bpdu-guard feature.

If you reenable the port without fixing the problem, the ports will
just become error-disabled again. This will continue over and over again until
you solve the real problem. Notice the three messages below. In the first one,
the switch describes disabling port 11/1 because it received a BPDU on a port
that is enabled for portfast (this is an error causing situation if bpdu-guard
is on). After 25 seconds, the port is automatically reenabled by the
errdisable-timeout feature. Then, four seconds later, the port is
error-disabled again because the real problem was never fixed.

The benefit of having to manually reenable the ports is that it
reminds you and prompts you to deal with the real problem.

Can I Eliminate Ports From Becoming Error-Disabled Due to
Collisions

Here is an example of how to keep the switch from error-disabling a
port due to excessive or late collisions. The set option
errport command became available in CatOS release 4.2(2). Again,
please remember that this should be used only as a "stop-gap" type of measure.
It keeps the ports from being error-disabled due to collisions but can leave
you vulnerable to collisions that would normally cause the switch to shut the
port down. When you execute this command, it will stop the switch from
disabling the port due to collisions.

The command show option errport will
show the current mode the error-disable feature is in. Also, the
set option errport enable command does not fix the
cause of the errors; it only keeps the port from being shut down because of the
errors. There still exists the possibility that errDisable ports could affect
other ports on the module if the errors persist or become drastic. So, you
should use this command only if you understand that these errors could
potentially cause larger problems within the switch module and you are willing
to take those risks.

A new feature starting in CatOS 5.4(1) allows the switch to monitor
ports that have portfast enabled. A port using portfast must only be connected
to an end station (such as a workstation or server), not
to devices that generate spanning tree BPDUs, like switches, or bridges and
routers doing bridging. If the switch receives a spanning tree BPDU on a port
that has portfast enabled, it will put the port in errDisable mode in order to
guard against potential loops. Portfast assumes that a port on a switch has no
possibility of generating a physical loop, and thus skips the initial spanning
tree checks for that port, avoiding end stations from timing out on boot up.
Portfast must be implemented carefully by the network administrator; on ports
where portfast has been enabled, BPDU guard helps ensure that the LAN stays
loop-free.

Here is how you turn this feature on. This example was picked because
it is easy to create an error-disable situation.

Our Catalyst 5500 switch is connected to another switch (a 6509) that
we made to be the root of the spanning tree. The 6509 will be sending us BPDUs
every 2 seconds (using default spanning tree settings). When we enable portfast
on the 5500 switch port, the bpdu-guard feature will watch for BPDUs coming in
on this port. When a BPDU comes into the port, meaning that a non-end device
has been detected off of that port, the bpdu-guard feature will shut the port
down to avoid possible Spanning tree loop.

To fix these situations, we must address the underlying problem, and
then reenable the port. Since this is a port with an improper connection
(portfast enabled and connected to another switch), we will turn off the
portfast feature. Again, portfast is only supposed to be used on ports
connected to end stations.

Even though we fixed the root of the problem, notice that the port is
still in error-disable state. If you looked at the port LED, it would still be
orange. We must reenable the port before it will become active again.

Here is another common error-disable situation that can occur on ports
capable of EtherChannel. If one switch is configured for EtherChannel and the
other is not, it can cause the spanning tree process to shut down the channeled
ports on the side configured for EtherChannel. In this scenario we connected
two crossover cables from the 5500 switch to another switch. We turned on
EtherChannel on the 5500 switch using the command set port channel
11/1-2 on. The ON mode of EtherChannel does not send PAgP packets
to negotiate with the other side before channeling; it just assumes the other
side is channeling. In addition, we did not turn EtherChannel on for the other
switch; we left these ports as individual unchanneled ports. If left in this
state for a minute or so, STP on the 5500 will think there is a loop. This will
cause the channeling ports to be put in error-disable state. Notice below that
a loop was detected and the ports were disabled. The show port
channel command shows that the ports are no longer channeling;
and, when we look at one of the ports involved, we see its status is
errdisable.

In order to determine what the problem was, we need to look at the
error message. The message said that the EtherChannel encountered a spanning
tree loop. As we know from the paragraph above, this can occur when one device
(our switch in this case) has EtherChannel turned on manually by using the ON
mode (as opposed to desirable) and the other connected device (the other switch
in this case) does not have EtherChannel turned on at all. One way to fix the
situation is to set the channel mode to desirable on both sides of the
connection, and then reenable the ports. This will cause each side to form a
channel only if they both agree to channel. If they do not agree to channel,
they will continue to function as normal ports.

Note: For a list of things that can cause EtherChannel misconfiguration
errors, look in the Configuration Guide on EtherChannel for the CatOS version
you are using. The newer releases have specific sections of the Configuration
Guide titled
Configuring Fast EtherChannel and Gigabit EtherChannel that list the
dependencies for a channel to form correctly, including the channel modes to
configure.

Notice that even though we turned off the EtherChannel feature and set
the EtherChannel mode to desirable, the ports are still disabled. We have
corrected the cause of the problem, but now we must reenable the ports before
we can use them.

show option errport—to view the status of
the set option errport command

set option errport disable—to allow the
switch to disable any ports that have errors which the operating system deems
worthy of being disabled. This is the default state and would only be different
if someone had previously issued the set option errport
enable command

show errdisable-timeout—to display the
current settings of the errdisable-timeout feature and the reason why any ports
are currently error-disabled

set errdisable-timeout—can be used to help
determine why a port was error-disabled (used in conjunction with the
show errdisable-timeout command)