A difference of opinion on the 6500

Ran into something interesting the other day and the response I got from TAC was even more interesting. We had a 6500 that had multiple equal cost paths out of a few different interfaces. The switch was basically being used as a distribution switch with multiple WAN routers plugged into it. Many of the prefixes were equal cost from the switches perspective so I expected CEF to do the default of per destination (really source/destination) load balancing. The switch appeared to do that correctly, but not in the manner that I expected it would…

Here’s what the lab topology looks like that I was using…

Let’s assume that each of the 4 routers has a equal cost path to get to 4.2.2.2. The 6500 sees equal cost routes for that /32 through each of the four routers…

At this point, let’s say that I want to determine the particular path that a packet would take towards 4.2.2.2. How would I do that? Easy, all CEF enabled devices (all that I can think of anyways) have the command…

show ip cef exact-route <source IP> <destination IP>

If you are doing per destination CEF load sharing, than this command should tell you the exact next hop and interface that the switch will use to forward the packet. Let’s do an example to see…

Here you can see that I’m checking the path that a packet will take from the client (10.0.0.21) to the destination of 4.2.2.2. CEF says that the packet will take the path out of interface Gig2/3 to the next hop of 10.0.0.9.

Pretty slick huh? Let’s send some packets from the client and see what happens…

After pinging 4.2.2.2 for 30 second the statistics on the Gig2/3 interface only show a single packet being output while the client is receiving valid echo-replies to the pings. How can this be?

I opened a TAC case and while I didn’t get a documented answer on what I was seeing, it appears that I was using the wrong command. I should have been using the…

show mls cef exact-route <source IP> <destination IP>

When that command is run, I get this output…

And sure enough, if I look at the Gig2/4 interface I see my pings…

The answer I got from Cisco was that the ‘show ip’ version of the command looks at the control plane or MSFC for the lookup. The ‘show cef’ version of the command looks at the hardware forwarding table or PFC for the lookup.

This makes it pretty clear that the control plane of the switch has different forwarding logic than the data plane which I find sort of scary. Apparently one of the forwarding tables uses an additional piece of information to determine the path. This extra piece of information is called the UID. Here’s what Cisco has to say about the UID…

Cisco IOS introduced a concept called unique-ID/universal-ID which helps avoid CEF polarization. This algorithm, called the universal algorithm (the default in current Cisco IOS versions), adds a 32-bit router-specific value to the hash function (called the universal ID – this is a randomly generated value at the time of the switch boot up that can can be manually controlled). This seeds the hash function on each router with a unique ID, which ensures that the same source/destination pair hash into a different value on different routers along the path. This process provides a better network-wide load-sharing and circumvents the polarization issue. This unique -ID concept does not work for an even number of equal-cost paths due to a hardware limitation, but it works perfectly for an odd number of equal-cost paths. In order to overcome this problem, Cisco IOS adds one link to the hardware adjacency table when there is an even number of equal-cost paths in order to make the system believe that there is an odd number of equal-cost links.

So apparently that UID is what’s causing the what I’m seeing but it’s unclear which forwarding table is using it. I’m betting that it’s being used on the hardware forwarding side of things but that’s only a guess. Since the control plane should be programming the data plane the whole concept of them being different seems odd to me.

Just something to keep in mind. If you want to see how the packet would be forwarded by the SUP (in software) use the ‘show ip’ version of the command. If you want to see how it would be forwarded in hardware use the ‘show mls’ version of the command.

Post navigation

3 thoughts on “A difference of opinion on the 6500”

How did you get someone from TAC to answer that? That guy must have been good. Can you utilize two ISP providers like that? I thought one ISP would be backup only because you can only have one default route?