During this step, the packet is received into the Nexus 7000 port. When troubleshooting this step, we want to look to ensure there is transceiver interoperability, and validate whether we are seeing any errors on the interface. We do this via using the following commands

During this step, the packet is received into the Nexus 7000 port. When troubleshooting this step, we want to look to ensure there is transceiver interoperability, and validate whether we are seeing any errors on the interface. We do this via using the following commands

Line 81:

Line 103:

Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning

Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning

-

==Step 2: Linksec Decryption Occurs, 1st stage Port QoS==

+

==Linksec Decryption Occurs, 1st stage Port QoS==

In step 2, Linksec decryption occurs as well as receive side stage 1 QoS.

In step 2, Linksec decryption occurs as well as receive side stage 1 QoS.

Line 271:

Line 293:

''' queue dropped pkts : 0'''

''' queue dropped pkts : 0'''

-

+

==Second Stage Port QoS Occurs==

-

==Step 3: Second Stage Port QoS Occurs==

+

For the ingress QoS, we are concerned about the Receive side QoS parameters in the show queuing command.

For the ingress QoS, we are concerned about the Receive side QoS parameters in the show queuing command.

Line 283:

Line 304:

* '''show policy-map interface'''

* '''show policy-map interface'''

-

+

==Layer 2 Source/Destination MAC Processing==

-

==Step 4: Layer 2 Source/Destination MAC Processing===

+

In this step, the ASIC submits the packet headers to theLayer 2 engine for lookup, and the Layer 2 engine performs source/destination MAC processing.

In this step, the ASIC submits the packet headers to theLayer 2 engine for lookup, and the Layer 2 engine performs source/destination MAC processing.

{note|When evaluating the Hardware mac table, if the Index is set to 0x00400, or the GM bit is set to “1”, that traffic will be routed. For example, you will see the index set to 0x00400 and GM bit set to 1 for traffic destined to the mac address local to the device}

+

{{note|When evaluating the Hardware mac table, if the Index is set to 0x00400, or the GM bit is set to “1”, that traffic will be routed. For example, you will see the index set to 0x00400 and GM bit set to 1 for traffic destined to the mac address local to the device}}

PHX2-N7K-1# '''show mac address-table'''

PHX2-N7K-1# '''show mac address-table'''

Line 330:

Line 350:

-

PHX2-N7K-1# sho hardware mac address-table 1 int eth1/1

+

PHX2-N7K-1# '''show hardware mac address-table 1 int eth1/1'''

Valid| PI | BD | MAC | Index | Stat| SW | Modi| Age | Tmr | GM | Sec|

Valid| PI | BD | MAC | Index | Stat| SW | Modi| Age | Tmr | GM | Sec|

TR | NT | RM | RMA | Cap | Fld | Always | | | |

TR | NT | RM | RMA | Cap | Fld | Always | | | |

Line 361:

Line 381:

1 1 2 '''000c.294b.c5ca 0x00422''' 0 3 0 67 1 0

1 1 2 '''000c.294b.c5ca 0x00422''' 0 3 0 67 1 0

-

0 0 0 0 0 0 0 0

+

0 0 0 0 0 0 0 0

+

+

==Layer 3 Engine Processing==

+

+

After the Layer 2 engine is finished, it sends the header to the Layer 3 engine. The layer 3 engine applies layer 3 intelligent features, to all packets, and layer 3 forwarding, to routed packets. As such, this section will divide the troubleshooting into two components, the layer 3 features applied to all packets, and the layer 3 forwarding.

+

+

The layer 3 features which are applied to all packets include the below features :

+

+

1) ACL

+

2) QoS

+

3) Netflow

+

4) Hardware IPS

+

+

Following the evaluation of the features, we will evaluate the layer 3 forwarding troubleshooting.

+

+

===Layer 3 Engine Processes Layer 3 Features===

+

+

We’ll drill through each of the layer 3 features below, looking at one feature at a time.

+

+

The first feature we will look at is ACL. To troubleshoot ACL, we want to evaluate the configuration, and any relevant hit counters. We then can identify if the hardware on the linecard is programming the ACL.

+

+

It is important to note, that if you wish to see per ACL counters, you must enable “statistics per-entry” in the ACL.

The next feature we will look at is the QoS troubleshooting for the Nexus. Note, we will have QoS applied, potentially, on both ingress and egress. So we should interrogate both the ingress and egress QoS.

+

+

The commands to troubleshoot QoS are

+

+

* '''show policy-map interface''' ''interface''

+

+

PHX2-N7K-1# '''show policy-map interface eth 1/2'''

+

+

Global statistics status : enabled

+

Ethernet1/2

+

+

Service-policy (queuing) '''input:''' default-in-policy

+

policy statistics status: enabled

+

+

Class-map (queuing): in-q1 (match-any)

+

queue-limit percent 50

+

bandwidth percent 80

+

'''queue dropped pkts : 0 '''

+

+

Class-map (queuing): in-q-default (match-any)

+

queue-limit percent 50

+

bandwidth percent 20

+

''' queue dropped pkts : 0 '''

+

+

'''Service-policy (qos) output: test-police-86 '''

+

'''policy statistics status: enabled'''

+

+

''' Class-map (qos): test-police-86 '''(match-all)

+

''' Match: dscp 18'''

+

''' police cir 100 mbps bc 200 ms '''

+

+

Service-policy (queuing) '''output: ''' default-out-policy

+

policy statistics status: enabled

+

+

Class-map (queuing): out-pq1 (match-any)

+

priority level 1

+

queue-limit percent 16

+

'''queue dropped pkts : 0 '''

+

+

Class-map (queuing): out-q2 (match-any)

+

queue-limit percent 1

+

''' queue dropped pkts : 0 '''

+

+

Class-map (queuing): out-q3 (match-any)

+

queue-limit percent 1

+

''' queue dropped pkts : 0 '''

+

+

Class-map (queuing): out-q-default (match-any)

+

queue-limit percent 82

+

bandwidth remaining percent 25

+

queue dropped pkts : 0

+

+

Netflow processing also has portions which occur in hardware. For netflow, we collect statistics in hardware on the linecards. We then, can export them via software.

Cisco NX-OS supports a hardware based intrusion detection system that checks for ip packet verification. These checks handle well known, and unusable traffic types which can be witnessed during malicious activity, such as if the source is a broadcast address, or if the destination is the 0.0.0.0 address. You can validate if any of these checks are dropping packets.

+

+

{{note|It has been shown in the field, that frequently it is advantageous to disable IP fragment verification. This is done via the below command

+

* '''no hardware ip verify fragment'''

+

}}

+

+

To validate if you are seeing any drops because of ip packet verification, you can use the below command.

+

+

* '''show hardware forwarding ip verify'''

+

* '''show hardware rate-limit module''' ''module''

+

+

+

PHX2-N7K-1# '''show hardware forwarding ip verify'''

+

+

IPv4 and v6 IDS Checks Status Packets Failed

+

-----------------------------+---------+------------------

+

address source broadcast Enabled 0

+

address source multicast Enabled 0

+

address destination zero Enabled 0

+

address identical Enabled 0

+

address source reserved Enabled 8

+

address class-e Disabled --

+

checksum Enabled 0

+

protocol Enabled 0

+

fragment Enabled 0

+

length minimum Enabled 0

+

length consistent Enabled 0

+

length maximum max-frag Enabled 0

+

length maximum udp Disabled --

+

length maximum max-tcp Enabled 0

+

tcp flags Disabled --

+

tcp tiny-frag Enabled 0

+

version Enabled 0

+

-----------------------------+---------+------------------

+

IPv6 IDS Checks Status Packets Failed

+

-----------------------------+---------+------------------

+

length consistent Enabled 0

+

length maximum max-frag Enabled 0

+

length maximum udp Disabled --

+

length maximum max-tcp Enabled 0

+

tcp tiny-frag Enabled 0

+

version Enabled 0

+

+

PHX2-N7K-1# '''show hardware rate-limit module 1'''

+

+

Units for Config: packets per second

+

Allowed, Dropped & Total: aggregated since last clear counters

+

+

Rate Limiter Class Parameters

+

------------------------------------------------------------

+

layer-3 mtu Config : 500

+

Allowed : 0

+

'''Dropped : 0 '''

+

Total : 0

+

+

layer-3 ttl Config : 500

+

Allowed : 0

+

'''Dropped : 0 '''

+

Total : 0

+

+

layer-3 control Config : 10000

+

Allowed : 0

+

''' Dropped : 0 '''

+

Total : 0

+

+

layer-3 glean Config : 100

+

Allowed : 0

+

'''Dropped : 0'''

+

Total : 0

+

+

layer-3 multicast directly-connected Config : 3000

+

Allowed : 0

+

''' Dropped : 0'''

+

Total : 0

+

+

layer-3 multicast local-groups Config : 3000

+

Allowed : 0

+

''' Dropped : 0'''

+

Total : 0

+

+

layer-3 multicast rpf-leak Config : 500

+

Allowed : 0

+

''' Dropped : 0 '''

+

Total : 0

+

+

layer-2 storm-control Config : Disabled

+

+

access-list-log Config : 100

+

Allowed : 0

+

''' Dropped : 0 '''

+

Total : 0

+

+

copy Config : 30000

+

Allowed : 0

+

'''Dropped : 0'''

+

Total : 0

+

+

receive Config : 30000

+

Allowed : 20450903

+

'''Dropped : 0 '''

+

Total : 20450903

+

+

layer-2 port-security Config : Disabled

+

+

layer-2 mcast-snooping Config : 10000

+

Allowed : 906

+

'''Dropped : 0 '''

+

Total : 906

+

+

===Layer 3 forwarding for Routed Traffic===

+

+

The Layer 3 engine will only perform Layer 3 forwarding for traffic that is routed through the router. This is traffic which has been sent to the MAC address of a valid routed interface, local to the router.

+

+

To troubleshoot the routed traffic, we need to perform the following tasks:

+

1. Ensure that the control plane routing is correct.

+

2. Ensure that the hardware forwarding entries on the ingress module have the corresponding information.

+

+

{{note|All routing of traffic is performed on the forwarding engine of the ingress module.}}

+

+

For our example, we will troubleshoot a route 86.86.87.0/24, which is set to a next hop of 86.86.86.1, and set to route out of VLAN 86 (an SVI).

+

+

We first will look at the route, ensure it is set to the correct next hop (86.86.86.1), and set to route out of VLAN 86. We will then want to ensure that we have a corresponding ARP entry associated with this next hop, and validate that the adjacency is in the adjacency table.

+

+

As we can see below, 86.86.87.0/24 is set to route to 86.86.86.1, out VLAN 86. This next hop is associated with MAC address 0011.aabb.ccdd. We will use this information to investigate the hardware, next.

+

+

The commands used to accomplish this are as follows:

+

+

* '''show ip route''' (prefix)

+

* '''show ip arp''' (nexthop)

+

* '''show ip adjacency'''

+

+

PHX2-N7K-1# '''show ip route 86.86.87.0/24'''

+

IP Route Table for VRF "default"

+

86.86.87.0/24, 1 ucast next-hops, 0 mcast next-hops

+

'''*via 86.86.86.1, Vlan86''', [1/0], 01:19:24, static

+

+

PHX2-N7K-1# '''show ip arp 86.86.86.1'''

+

+

IP ARP Table

+

Total number of entries: 1

+

Address Age MAC Address Interface

+

'''86.86.86.1 - 0011.aabb.ccdd Vlan86 '''

+

+

+

PHX2-N7K-1# '''show ip adjacency '''

+

+

IP Adjacency Table for VRF default

+

Total number of entries: 1

+

Address Age MAC Address Pref Source Interface

+

'''86.86.86.1''' 00:00:37 '''0011.aabb.ccdd''' 1 Static ''' Vlan86'''

+

+

The above example shows the control plane. Now that we know how things are supposed to work, we can interrogate the hardware to ensure the hardware entries have propagated properly to the Layer 3 hardware engine. We can see that the IP FIB has properly associated 86.86.87.0/24 to the next hop of 86.86.86.1. We can also see, in the hardware entry, that this is routed out VLAN 86, that the RPF is valid if we have enabled RPF Checking), and that the route entry is correctly associated with the MAC address of 0011.aabb.ccdd.

+

+

This demonstrates that the routing in the forwarding plane is programmed correctly and that the forwarding will follow the information contained in the routing protocols.

To validate forwarding of the Layer 2 engine, we should first look at the centralized MAC table aggregated on the supervisor to validate whether the MAC addresses are correlated as we expect them, and assigned to the ports where we expect the MAC’s to reside.

+

+

Based on this, we can then validate the hardware programming on the egress module to validate that our MAC address table is properly programmed into the hardware based Layer 2 engine on the module.

As the packet nears entry from the Cisco Nexus 7000, one of the final steps is the application of Port QoS. The port level QoS will be able to buffer the traffic in times of congestion. To interrogate the egress QoS, we look at following QoS commands, paying attention to the Transmit QoS:

The final step in the process is the transmission of the frame out of the physical egress port. Troubleshooting of the physical port, is the same as in step 1, and includes the following commands:

+

+

* '''show interface''' ''interface''

+

* '''show interface''' ''interface'' '''transceiver'''

+

+

+

The output from these commands are documented in step 1 above.

Revision as of 21:23, 30 November 2011

This article below provides only basic information on how to troubleshoot unicast packet flow traffic issues for the M1 Series modules.

Troubleshooting L2/L3 unicast is covered in detail in Cisco-Live presentation. Sections of this presentation covers,
both platform independent, and platform specific step by step troubleshooting for unicast, among other things. Access to
this presentation is available FREE. Follow the below instructions to access the presentation

Packet is Received into Interface from Wire

During this step, the packet is received into the Nexus 7000 port. When troubleshooting this step, we want to look to ensure there is transceiver interoperability, and validate whether we are seeing any errors on the interface. We do this via using the following commands

Ethernet1/1
sfp is present
name is CISCO-AVAGO <<< If this says type is (unknown), it is not supported.
part number is SFBR-7700SDZ
revision is B4
serial number is AGD12434116
nominal bitrate is 10300 MBits/sec
Link length supported for 50/125um fiber is 82 m(s)
Link length supported for 62.5/125um fiber is 26 m(s)
cisco id is --
cisco extended id number is 4

Linksec Decryption Occurs, 1st stage Port QoS

In step 2, Linksec decryption occurs as well as receive side stage 1 QoS.

It is important to step back and evaluate the difference between stage 1 and stage 2 QoS. The difference is that some ports can be configured in shared mode, whereas some can be configured in dedicated mode, on the 10G modules. What this means, is that there is 10g of bandwidth that can be dedicated to a port or shared amongst ports (4 ports, on the m132 module).

When running in shared mode, there exists a chance for contention accessing the 10g bandwidth through the 4:1 Mux. To alleviate this, some QoS intelligence was passed down to the 4:1 Mux which aggregates the ports.

In dedicated mode, there is no QoS applied at the Mux, instead, all traffic is processed in phase 2 QoS. To summarize, in shared mode, 1st stage QoS ensures fair access to the 10g of port bandwidth. In both shared and dedicated mode, 2nd stage QoS occurs to provide ingress queuing to the system.

For the ingress QoS, we are concerned about the Receive side QoS parameters in the show queuing command.

Second Stage Port QoS Occurs

For the ingress QoS, we are concerned about the Receive side QoS parameters in the show queuing command.

Use the show policy-map command to view queue drops .

The commands to troubleshoot Port QoS are

show queuing interfaceinterface

show policy-map interface

Layer 2 Source/Destination MAC Processing

In this step, the ASIC submits the packet headers to theLayer 2 engine for lookup, and the Layer 2 engine performs source/destination MAC processing.

To validate forwarding of the Layer 2 engine, we should first look at the centralized mac table aggregated on the supervisor to validate whether the mac addresses are correlated as we expect them, and assigned to the ports where we expect the Mac’s to reside.

Based off of this, we can then validate the hardware programming on the ingress linecard to validate that our mac address table is properly programmed into the hardware based Layer 2 engine on the linecard.

We first will look at the mac address table, then we can ensure programming is properly occurring in the hardware table.

The commands used to accomplish this are as follows:

show mac address-table

show hardware mac address-tablemoduleinterfaceinterface

To drill down on a specific MAC address, we can use the grep function with these commands to validate the mac is associated with a particular port, and that the hardware programming reflects that.

When evaluating the Hardware mac table, if the Index is set to 0x00400, or the GM bit is set to “1”, that traffic will be routed. For example, you will see the index set to 0x00400 and GM bit set to 1 for traffic destined to the mac address local to the device

Layer 3 Engine Processing

After the Layer 2 engine is finished, it sends the header to the Layer 3 engine. The layer 3 engine applies layer 3 intelligent features, to all packets, and layer 3 forwarding, to routed packets. As such, this section will divide the troubleshooting into two components, the layer 3 features applied to all packets, and the layer 3 forwarding.

The layer 3 features which are applied to all packets include the below features :

1) ACL
2) QoS
3) Netflow
4) Hardware IPS

Following the evaluation of the features, we will evaluate the layer 3 forwarding troubleshooting.

Layer 3 Engine Processes Layer 3 Features

We’ll drill through each of the layer 3 features below, looking at one feature at a time.

The first feature we will look at is ACL. To troubleshoot ACL, we want to evaluate the configuration, and any relevant hit counters. We then can identify if the hardware on the linecard is programming the ACL.

It is important to note, that if you wish to see per ACL counters, you must enable “statistics per-entry” in the ACL.

The next feature we will look at is the QoS troubleshooting for the Nexus. Note, we will have QoS applied, potentially, on both ingress and egress. So we should interrogate both the ingress and egress QoS.

Cisco NX-OS supports a hardware based intrusion detection system that checks for ip packet verification. These checks handle well known, and unusable traffic types which can be witnessed during malicious activity, such as if the source is a broadcast address, or if the destination is the 0.0.0.0 address. You can validate if any of these checks are dropping packets.

Note:

It has been shown in the field, that frequently it is advantageous to disable IP fragment verification. This is done via the below command

no hardware ip verify fragment

To validate if you are seeing any drops because of ip packet verification, you can use the below command.

Layer 3 forwarding for Routed Traffic

The Layer 3 engine will only perform Layer 3 forwarding for traffic that is routed through the router. This is traffic which has been sent to the MAC address of a valid routed interface, local to the router.

To troubleshoot the routed traffic, we need to perform the following tasks:
1. Ensure that the control plane routing is correct.
2. Ensure that the hardware forwarding entries on the ingress module have the corresponding information.

Note:

All routing of traffic is performed on the forwarding engine of the ingress module.

For our example, we will troubleshoot a route 86.86.87.0/24, which is set to a next hop of 86.86.86.1, and set to route out of VLAN 86 (an SVI).

We first will look at the route, ensure it is set to the correct next hop (86.86.86.1), and set to route out of VLAN 86. We will then want to ensure that we have a corresponding ARP entry associated with this next hop, and validate that the adjacency is in the adjacency table.

As we can see below, 86.86.87.0/24 is set to route to 86.86.86.1, out VLAN 86. This next hop is associated with MAC address 0011.aabb.ccdd. We will use this information to investigate the hardware, next.

The above example shows the control plane. Now that we know how things are supposed to work, we can interrogate the hardware to ensure the hardware entries have propagated properly to the Layer 3 hardware engine. We can see that the IP FIB has properly associated 86.86.87.0/24 to the next hop of 86.86.86.1. We can also see, in the hardware entry, that this is routed out VLAN 86, that the RPF is valid if we have enabled RPF Checking), and that the route entry is correctly associated with the MAC address of 0011.aabb.ccdd.

This demonstrates that the routing in the forwarding plane is programmed correctly and that the forwarding will follow the information contained in the routing protocols.

SFabric Processing Occurs (optional)

This step occurs if the packet needs to traverse the fabric.

In this step, we need to interrogate if the fabrics are functioning properly, and if their utilization is at an acceptable level.
We can view the fabric status and utilization using the following commands:

Layer 2 Engine Performs Source/Destination MAC Processing

To validate forwarding of the Layer 2 engine, we should first look at the centralized MAC table aggregated on the supervisor to validate whether the MAC addresses are correlated as we expect them, and assigned to the ports where we expect the MAC’s to reside.

Based on this, we can then validate the hardware programming on the egress module to validate that our MAC address table is properly programmed into the hardware based Layer 2 engine on the module.

The commands used to accomplish this are as follows:

show mac address-table

show hardware mac address-tablemoduleinterfaceinterface

The output from these commands are documented in steps 3-4 above.

Egress Port QoS is Performed

As the packet nears entry from the Cisco Nexus 7000, one of the final steps is the application of Port QoS. The port level QoS will be able to buffer the traffic in times of congestion. To interrogate the egress QoS, we look at following QoS commands, paying attention to the Transmit QoS: