Tuesday, September 28, 2010

Cisco switches have a nifty but little used diagnostic feature: The 'traceroute mac' command.

It does pretty much what you'd expect. It traces the L2 path between two endpoints. Exactly how it accomplishes this feat is much less obvious. Normal (Layer 3) traceroute makes use of progressively larger TTL marks, and uses the ICMP "time to live exceeded" errors from routers along the path in order to print the path between two nodes. These mechanisms don't exist in a bridged environment. So how does it work?

Consider the following topology:

Running traceroute mac on the rightmost switch produces the following result:

The procedure for doing this work manually is straightforward. We look for each MAC address in the VLAN 11 forwarding table, then check to see whether the egress port has a CDP neighbor. If so, log into the next switch (using the management address reported by CDP). Lather, rinse, repeat.

The manual procedure works on each MAC address independently. The 'traceroute mac' command, however requires you to specify both source and destination stations. It's curious: L3 traceroute doesn't do that, it assumes you want to trace the path from here to somewhere else. 'traceroute mac' on the other hand can trace from somewhere else to somewhere else. Here we run a trace from a switch that isn't in the transit path between the two end stations.

In that case, we were logged into the 2950, but traced the L2 path between two stations that were each connected directly to the 2960. Two L2 hops away. The operation of this tool is somewhat mysterious. Here's what I can tell about it so far:

Both MAC addresses must appear in the forwarding table on each switch in the path.

Each switche uses CDP to figure out what next hop lies beyond its local egress port (and the IP address on which it can be reached).

It's an L3 process. The right switch in the figure above is in a different management subnet than his neighbors. These switches all forward traffic directly at L2, but they talk amongst themselves via the router-on-a-stick. This is different than the operation of CDP (a link layer protocol).

If more than one CDP neighbor appears on an interface in the path (several switches hanging from a hub), the process blows up because it's impossible to discern the next hop. You can replicate this easily by changing the hostname of a switch. The neighbors will see two entries until the old name times out.

The switch running the trace communicates directly with each device in the path. The first example involved:

An exchange between the 2960 and the 3550

An exchange between the 2960 and the 2950

The process requires a service running on UDP 2228 on each switch (see it with 'show ip sockets')

If there's no CDP neighbor on a port (like when I switched CDP off on the 2950), then that's the end of the trace.

The wire format is undocumented as far as I can tell. The wireshark wiki page for CDP mentions the protocol, but doesn't have any information on it. It's not CDP, but it's similar. Each packet seems to have some fixed fields, and some TLV sets.

It's very surprising to me that mapping out an L2 environment can be done using L3 (off subnet) tools with seemingly no security. I'm not much of a believer in security by obscurity (I generally run CDP on edge ports), but this level of network mapping without even requiring an SNMP read-only string seems like it could be a problem. The only hint of a complicating factor here is that the name of the target switch is embedded in the request packet. If that name is checked before a reply is sent, there's some small measure of security. But all an attacker needs is the name of a single switch, since that first switch will give up the names of all of his neighbors.

It will be difficult to strike the balance between security and usability when writing ACLs for this service, since you need to protect every IP interface on an L3 switch, while still providing service to clients on every IP interface of every L2/L3 switch.

Have you had any further luck figuring out how l2 traceroute actually works? So far this is the most detailed account I have found. I'm attempting to write a script to create the same functionality the cisco devices use.

Sorry, no. Everything I know about it is spelled out here. It still seems crazy to me that the switches will give up this information so readily. If you write that script, please follow up, I'd like to try it!