Friday, December 17, 2010

It's nice to have entries in your internal DNS for router interface names; it makes stuff like traceroute a whole lot easier to read. Many small-to-mid-size companies use Solarwinds products for network management.

This perl script takes the output that you get from using Solarwinds Network Configuration Manager to run the "show ip interface brief | exclude unassigned" command on a group of Cisco IOS devices and massages it into a format that looks like this:

hostname-interfaceName-interfaceNumber a.b.c.d

where a.b.c.d is the IP address assigned to the interface. This format is suitable for import into many DNS servers.

This code doesn't attempt to abbreviate interface names at all, but it could easily be modified to do so.

Monday, December 6, 2010

I have a requirement to connect two internal routers over a high-speed, 3rd party, non-Internet network. The two routers need to run EIGRP with each other. We want to encrypt the link, but the routers don't support IPSec in their current configuration. Purchasing IPSec acceleration hardware for them is expensive, but we happen to have two ASAs in inventory that aren't currently in production.

The simplest solution for this is to connect the two routers with a GRE tunnel, then use the ASAs to encrypt the GRE traffic.

While I've worked with ASAs quite a bit as stateful packet filters and as remote access VPN headends, it's been a really long time since I've used one in a point-to-point VPN. I thought I'd blog about the lab proof-of-concept so that I don't forget everything about the configuration.

My lab topology looks like this:

R4---ASA1---ASA2---R5

R4 and R5 have a standard GRE tunnel configured between them, running EIGRP to advertise their loopbacks. The tunnel destination is statically routed.

[Note: the difference in the EIGRP configs isn't a mistake. The lab routers are running two different images, one of which has auto-summary disabled by default. The other has it enabled by default, so I had to explicitly turn it off.]

This is a pretty standard GRE configuration that is used all the time to make a virtual point-to-point circuit across any other network. I'm intentionally leaving out the MTU complications for now.

Thursday, December 2, 2010

Recently I got pulled into a debate between two colleagues who were troubleshooting a problem where some users could access a website over SSL, and others couldn't. One person was arguing that the problem was caused by client misconfiguration, and the other was arguing that it wasn't. Following my mantra "when in doubt, capture packets", we captured some traffic and had a look. I'm not going to go through the entire troubleshooting process; rather I'm going to focus on what was ultimately causing the problem. Here's the packet sequence, output from Tshark with some of the TCP details removed to make it fit:

The key thing to note here is the delta between the SYN and SYN/ACK: about 42ms. When I was viewing this in Wireshark I set a packet time reference on packet #921, using the "Ctrl T" keyboard shortcut; this makes it easier to see the delta values.

I then set another time reference on packet 931, the TLSv1 Client Hello. Immediately following this, less than 1 millisecond later, we see a RST come back from the server. Red flag! Since we already established the probable latency between the hosts as ~42ms using the SYN - SYN/ACK pair, this is extremely suspicious.

A few packets later, we see a TLSv1 Server Hello message inbound, AFTER the RST. The delta? Approximately 42ms, exactly what we'd expect.

I immediately inferred from this that a transparent proxy content filter was spoofing the RST due to something that it deemed to be objectionable content, and for whatever reason it wasn't notifying the user.

I talked to the administrator for the content filter, and indeed, a policy had been incorrectly applied to some users that blocked the content in question.

Thursday, November 18, 2010

Will the "mls qos vlan-based" command mark packets if the packets aren't routed by the SVI where the service policy is applied?

I had never even considered this before, so I guessed "no" before reading the docs. Then I set it up in the lab, and the answer (superficially) still seemed to be "no". Then, however, I executed the old RTFM maneuver: the docs made it sound like the answer should be "yes". So I thought I'd have a closer look. Here are the details.

The test topology looks like this:

R5---Switch1---Switch2---Cat3750---R6

R5 & R6 are 2800 series routers.

Switch1 and Switch2 are 3560s.

Cat3750, unsurprisingly, is a Catalyst 3750, and the subject of the test.

All of the inter-switch links are 802.1q trunks, and the two routers are connected to access ports in VLAN 6.

R5's interface IP address is 10.6.6.5, with a loopback of 5.5.5.5.

R6's interface IP address is 10.6.6.6, with a loopback of 6.6.6.6.

R5 and R6 are running OSPF on all interfaces and are neighbors when the test begins.

First, I set up a QoS policy on the 3750 that would make packets distinguishable from each other:

Since all traffic should be marked as CS2, the OSPF hello packets being exchanged between R5 and R6 should get marked immediately if the policy is working, but I also generated some ICMP and telnet traffic just for good measure.

After my RTFM episode, I started wondering if the packets were getting marked anyway, but not counted by the switch. Idecided to try applying a service policy on R5 that would have rather dramatic effects if the packets were in fact marked:class-map match-any CM_CS2 match ip dscp cs2!policy-map PM_DROP_CS2 class CM_CS2 drop

So... in conclusion: the "mls qos vlan-based" command does indeed work for service policies applied to interfaces that don't route packets for the VLAN... essentially, the command causes the access port to inherit the QoS policy applied to the SVI, and the SVI doesn't count them. This does actually make sense, but it took working through the lab to make my brain process it correctly.

Monday, September 27, 2010

As an old-time Unix/Linux guy, I’ve been wanting the IOS equivalent of the “wc -l” command for oh, about a million years now. Juniper has had this in JunOS for quite some time, via their “count” command. In Unix/Linux shell environments, “wc -l” counts the number of lines in the input. This is very useful for things like counting the number of unused ports on a switch. Until recently, I was stuck with using SSH one-liners to pipe IOS commands through a shell filter, like this:

$ ssh 10.1.1.2 'sh int status | i not|disa' | wc -lPassword:19

In English: this executes the command “show interface status | include not|disa” via SSH on the remote device, which returns all the lines that are in either the “notconnected” or “disabled” status. Then it feeds those lines to the “wc -l” command, which counts the lines.

A couple of months ago, however, I noticed that recent versions of IOS for the Catalyst access switches (3560/3750, and probably the 2960--I haven’t tested the latter) have a “count” filter:

Friday, July 2, 2010

This was my 6th consecutive year at CiscoLive (formerly known as Networkers), and as always I had a great time. I thought I'd write down a few brief thoughts on my sessions and other experiences this year.

802.1x 8-Hour TechtorialThis was the first time I've done a "techtorial" (Cisco's term for a 4 or 8 hour expanded seminar that costs extra) since 2006, and it was probably the best one I've done. All of the presenters were excellent, and they did a great job of keeping the class engaged by switching frequently between lecture and live demonstrations by three different instructors. They also included a real-world case study, presented by the actual customer involved (a large Canadian university). Cisco has a tendency to make up artificial case studies (or anonymize them to the point of making them pointless), so it was great to see a live customer on stage, presenting the entire implementation process, warts and all.

LISP - A Next Generation Networking ArchitectureI have read a fair bit about LISP over the last couple of years, but this was the first time I've gone to a session on it. Basically, Dino and company are trying to solve three or four of the biggest problems in networking in one fell swoop: 1) global routing table size, 2) certain types of IPv4/IPv6 transport issues, 3) virtual machine mobility, and possibly 4) other mobility problems. I really don't have the background to evaluate a protocol that's designed to solve extremely difficult problems at a global scale, but it was fascinating to see the thought process and design issues involved.

Routed Fast Convergence and High AvailabilityI have been hearing about this session for years, and it didn't disappoint. I was already familiar with most of the tools discussed, but the devil is in the details, and I came away with a much better understanding of many techniques used to achieve fast convergence in routed networks.

Smart Grid: Developing a Communications Architecture for the Utility of the FutureI didn't intend to visit this session initially, but the speaker was late for my scheduled session, and I had no interest in sitting around waiting for him to show up. This session was happening nearby, and I knew that Bill Parkhurst is something of a network architecture guru, so it was an easy pick. Even though I have absolutely no background in the electrical utility world, this was a very useful session from a general professional development perspective.

Unified HA Network Design: The Evolution of the Next Generation NetworkIf I could recommend only one session on large-scale network design, this would be it. These guys are working on the largest, most failure-sensitive networks on the planet, and they're giving away what they've learned. What more needs to be said?

Advanced Security Management & Incident ResponseThis was my second time attending this session (the last time was in 2007), and it was definitely worth attending a second time. I really like operationally-focused sessions (as opposed to product-focused ones), and that's what this one is all about. The presenters are front-line senior incident responders in Cisco's internal security organization, and it's great to see how they deploy Cisco tools and even (gasp) non-Cisco tools to respond to actual security incidents. I really hope they can convince the powers-that-be to let them run this as an expanded, 8-hour techtorial on security operations.

Those are the highlights of my sessions. I didn't attend a single session this year that was actually bad, but those were the ones that stood out the most.

Other thoughts: the meals were above average compared to previous years, except for breakfast. I hate getting to a breakfast and seeing nothing but bread products and some random, lonely looking fruit. The CCIE party was great; probably the best one I've attended. The CCIE NetVet reception with John Chambers was also excellent. My biggest complaint continues to be the ominous warnings about a $100 fee to replace a lost conference badge. I've never lost mine, but this just seems patently ridiculous.

As always, though, simply meeting other networking-focused professionals and renewing friendships with people I've known from previous years was the best part of the show.

Friday, April 16, 2010

Friday, April 9, 2010

For the last week I've been reading Laura Chappell's new book, Wireshark Network Analysis. I pre-ordered the book and was looking forward to it eagerly.

Overall, it's a superb book. First and foremost, I appreciate the fact that the writing has some personality to it. I've really enjoyed the author's footnotes, anecdotes, and humor.

Second, I like the fact that it's exhaustively thorough with both features and examples. There are relatively few cases where the Chappell simply describes a feature without offering an example of how it might be used in practice. Where she does so, the example is obvious enough to be unnecessary.

Particularly good are the examples of practical Wireshark tips and tricks in the protocol-specific sections. As an advanced Wireshark user with a thorough background in the internals of common network protocol operations, I was worried that this would be just another set of explanations of how various protocols work. Those explanations are there, but they are interspersed with many great tips and tricks about how to analyze the protocols more quickly and efficiently with Wireshark. I kept thinking, "why didn't I ever think of using that trick before?"

There is a lot here for both beginning and advanced Wireshark users.

I don't have a lot of criticisms to make: I thought the security sections were interesting, but a little dated. Exotic network and transport layer attacks just aren't all that common anymore; it would have been cool to have seen some analysis of a modern application-layer attack in action.

The other thing I would have liked is a discussion of the Lua scripting-language extensions to Wireshark. There is very little out there on the Internet about this so far, and most of what exists is oriented toward expert-level programmers. I was a bit disappointed to not even find Lua in the index. Still, the book is already huge, and adding a section on scripting might have made it unreasonably long. Maybe a volume 2?

Summary: if you use Wireshark, just buy it. This is an amazingly practical, hands-on book that will make you faster and more productive when analyzing network captures.

Blogging disclaimer: I paid full retail price for the book (and managed to somehow miss all the promo coupons) and am not being compensated in any way for this review.