Building network automation solutions

6 week online course

If you’re based in Europe or Africa (or don’t mind attending a webinar very early in the morning like some of my US-based students), you can take advantage of the VPN Day Combo: if you register for both VPN webinars on July 7th, you’ll get a 20% discount on the second ticket ($39.99 instead of $49.99).

When Cisco started preaching about Borderless Networks a few months ago, we all knew the term Borderless Networks was a new fuzzily-defined paradigm revolving around the facts that:

People want to use their smartphones (and other mobile devices) to access the corporate data from anywhere at any time.

Employees have started to use third-party cloud services with unproven security or reliability without coordination with corporate IT or Security.

However, when Cisco Press launched the Securing the Borderless Network book (with the subtitle Security for the Web 2.0 World), I was hoping to get some insight into what Cisco really means with the Borderless Networks paradigm. I was also expecting some hard technical facts and solutions for the problems pestering all of us.

One of the scenarios I’m discussing in the DMVPN: Advanced and Crazy Scenarios (register here) is redundant DMVPN network with two ISPs. It’s not a particularly complex setup ... unless the ISPs decide to deploy anti-spoofing filters (more precisely: unicast RPF checks) in which case it becomes crucially important which outbound interface you use for your DMVPN tunnel.

Anyhow, I was trying to make the whole thing work in a lab and it was repeatedly failing, so I decided to log uRPF violations. According to the documentation, it’s a piece of cake:

During the last Google IPv6 Implementors Conference Donn Lee from Facebook showed how easy it is to make your content available over IPv6 and LISP ... if you happen to have the right load balancer that supports IPv6 (to view his presentation, click the slides link next to his name in the conference agenda). I would say all the excuses why your content cannot be possibly made available over IPv6 are gone (and one can only hope that a certain vendor I’m often mentioning will finally realize IPv6 is needed on more boxes than just routers and switches).

The more I think about this problem, the more I’m wondering whether we really need large-scale bridging in data centers (it looks like Google can live quite happily without it). We definitely need some bridging, but generic large-scale inter-site monstrosity? I doubt.

Please try to help me: forget all the “this is how we do it” presumptions, figure out a scenario where you absolutely need bridging and describe it in the comments.

Simplified packet header for routing efficiency. This is a new one for me (and I thought I’ve seen them all), but no less bogus than any other. Cisco’s own measurements confirm that IPv6 performs almost equally well (or marginally worse) than IPv4. There’s no improvement and no increased efficiency.

Mandatory IPSec implementation for all IPv6 devices. While this is technically true, it’s at least misleading, as most readers automatically assume it means any host-to-host communication can be easily encrypted. Not necessarily true, you’ve implemented a compliant IPSec implementation even if you just recognize the IPSec header without having any ESP (encryption) or AH (integrity/origin authentication) capabilities.

Maybe it’s time we all grow up and admit the only benefit of IPv6 is its larger address space.

Cisco has introduced Tunnel Route Selection, another “somewhat” underdocumented feature in IOS release 12.4(11)T (reading the sparse documentation, it appears to be a half-baked kludge implemented for a specific customer). I was wondering for a long time why I would ever want to use this feature, until Floris Martens asked me a question about a redundant DMVPN network using two ISPs ... and all of a sudden it all made a perfect sense.

If you want to influence traffic flow in a network, you might want to tweak routing protocol metrics to shift the traffic between paths of almost-equal cost (I would always prefer MPLS Traffic Engineering as it’s so much better, but sometimes changing a metric is faster than rebuilding your network). OSPF and IS-IS are easy: change the interface metric or interface bandwidth. EIGRP and its composite metric are trickier.

As you know, EIGRP vector metric has five components; two of which are usually ignored and MTU serves only as tie breaker. This leaves us with bandwidth and delay. Every EIGRP reference tells you to adjust interface delay, not bandwidth, and the simplistic explanation is that “bandwidth is used for QoS features, so it’s better left unchanged”. While that’s true, there are other more important reasons to focus on delay:

Yesterday I decided to get some gym climbing after a long spell of bad weather and PPP#8 was ideal companion for the drive (driving and listening to PPP#7 where those crazy guys were discussing Enterprise MPLS might not be a good idea). They started with incredibly long lead times on popular boxes (nothing new if you’ve been in the industry long enough, we had them when Cisco shipped the 4000-series routers in the previous millennium), jumped to Telepresence and IOS licensing ... and finished with the really important question whether you should encourage the geekiness in your kids; all together a nice medley of interesting topics.

Note to Ethan: This is the best ISR G2 licensing document I found so far. Note to Cisco Marketing: You know you can do better than that. Note to Cisco CA (or whoever owns this tool): Feature Navigator is BROKEN (when applied to release 15.0) and thus useless to figure out which license I need for ISR G2.

You see, everybody else is too afraid of looking stupid because they just can’t keep enough facts in their head at once to make multiple inheritance, … or multithreading, or any of that stuff work.

So they sheepishly go along with whatever faddish programming network craziness has come down from the architecture astronauts who speak at conferences and write books and articles and are so much smarter than us that they don’t realize that the stuff that they’re promoting is too hard for us.

His Guiding Principles are also excellent (and oft repeated by the old-timers who have learned their lessons the hard way):

Ethan Banks has a great article @ PACKETattack: in Assembly Required – Interconnecting 2 Ethernet Chassis Switcheshe describes various options you have when you want to connect your redundant core switches. Using more than one physical link is the obvious choice; most people are careful enough to use at least two linecards, but the true magic begins when you start considering the bandwidth allocation to individual linecards and port groups within linecards.

Yesterday I finally found time to listen to the DDoS chewing podcast on Packet Pushers. While I know quite a bit about the technical solutions, their focus on the big picture and Service Provider offerings was a truly refreshing one (after all, if you’re under attack, it’s best if your upstream SP filters the junk).

They also mentioned a few interesting application-related issues that will definitely help me streamline my web sites (for example: once your load goes above a certain threshold, start serving cached data instead of retrieving it live from the database) and discussed an interesting case study where a networking engineer (Greg, if I’m not mistaken) managed to persuade the programmers to optimize the application, thus saving the company a lot of money in the long run.

Even if DDoS protection might not be relevant to your current job position and although a lot of their discussion was spinning around SP offerings and application-level solutions, I would strongly recommend that you listen to the podcast. After all, it never hurts to glance around your sandbox and consider other perspectives (and I definitely enjoyed the view).

Bidirectional Forwarding Detection (BFD) protocol has finally been published as a series of RFCs. BFD gives you quick failure detection between L3 hops (routers) regardless of the underlying technology and equipment (modems, media converters, bridges). It’s been gradually introduced in Cisco IOS during the last few years; release 15.0M and 12.2SRE contain almost everything you’ll ever need (missing: multihop BGP support and MPLS LSP support).

It’s amazing what people would try to patent ... and it’s even more amazing what gets past the examiners. IBM has managed to patent passing ipv6 or dhcp argument to indicate an IP host should network-boot over IPv6 or using DHCP. The idea is so trivial it’s almost not worth mentioning and goes along the lines of: “usually we use BOOTP and TFTP to get network boot parameters, but imagine we could pass DHCP as the argument to the boot routine and then it would use DHCP instead of BOOTP”.

The patent supposedly covers a very specific case, but (to my untrained eye) the claims are written in a way that could cover almost any IPv6- or DHCP-assisted network boot (or at least give lawyers plenty of stuff to charge for) ... exactly what we needed with all the other roadblocks and stumbling stones to IPv6 deployment.

If a data cap doesn't affect 97% of users, why bother implementing it at all? Surely the 3% can be that significant?

A few of us immediately responded that the 3% could represent 80 (my guess) to 97% (@icemarkom) of the traffic. As I’m tracking my home Internet connection with MRTG for over a year, I was also able to get some hard facts (although the sample size is admittedly very small). We’re pretty heavy internet users (no limits on what my teenage kids are doing and I’m mostly working from home), but the average yearly utilization of my 20 Mbps pipe is only 180 Kbps or less than 1% of its capacity (still, over a year, that’s almost 700 GB of data or 350 months of AT&T’s DataPro plan).

Another article from Scott Berkun pointed me to a wonderful tool: Readability. Imagine being able to read web pages in decently-sized font on pleasant background and without all the navigational and ad clutter. It makes my reading experience infinitely more pleasurable; now I can manage to read blog posts that are several pages long without getting a headache.

Safari 5 has a similar functionality already built in: a Reader button appears next to the URL and once you click it, you get the main text of the web page in a pop-up frame. It works a bit better than Readability (which sometimes positions the text annoyingly close to the left side of the browser window), but is (contrary to Readability) not easily configurable; Steve knows best what you need. Fortunately, there’s almost always a workaround.

IS-IS has “forever” (at least since RFC 1195) supported multiple layer-3 protocol, but always with a nasty side-effect: if a link in your network did not support one of them, you could get hard-to-diagnose black holes. The problem is illustrated in the left-hand column of the following diagram. Due to a single IS-IS topology, the shortest path between A and B is the direct link and since IPv6 is not enabled on that link (click on the diagram to get an enlarged version where you'll be able to see the link colors), A and B cannot exchange IPv6 traffic even though there’s an alternate path between them.

Tired of losing half of your bandwidth to spanning tree? TRILL will solve all your problems, bring the world peace and make better coffee than Starbucks (hint: the second claim is fake and the third one is not so hard to achieve).

Undoubtedly TRILL is an interesting technology that can alleviate the spanning tree limitations. Unfortunately I’ve seen a very similar technology being heavily misused in the past (resulting in some fantastic failures) and remain skeptical about the deployment of TRILL. My worst case scenario: TRILL will make it too simple to deploy plug-and-pray bridged (vendors will call them “switched”) networks with no underlying design that will grow beyond control and implode.

Andrej Kobal from Astec shared a few interesting facts during the 3rd Slovenian IPv6 summit: they were deploying a pilot IPv6 subnet in a large network and wanted to retain tight control over the IPv6 address assignment (some people don’t consider random address chasing embraced by Windows the best use of their time), so they’ve decided to use DHCPv6. Bad luck: DHCPv6 can’t tell you the IPv6 address of the default router (like DHCP does). You need ICMPv6 RA (part of IPv6 Neighbor Discovery) to figure out who the router is.

Every so often I get a question about the MTU metric in EIGRP and whether it’s used at all or not. It actually is: if your router would have to ignore some equal-cost paths to the same destination (the number of equal-cost paths exceeds the value of the maximum-paths router configuration parameter), it ignores those with the lowest MTU metric.

A while ago Cisco Slovenija decided to set up the local CCIE club. They've sent out invitations to active CCIEs based in Slovenia and somehow a copy of it landed on the desk of an old semi-retired grump (myself). I replied saying I would be interested if they are willing to accept inactive CCIEs ... and they totally surprised me by making me an honorary member.

After such a nice gesture, I couldn't possibly decline the kind invitation to be a guest speaker in one of the regular CCIE club meetings and I chose a topic that's very close to my heart these days: the upcoming challenges of the global Internet.

On a sunnier front, I’d never got so many registrations in the first 24 hours as I did after announcing the DMVPN webinar. The number of seats might be limited (I need to check our Webex license); it’s probably a good idea to hurry up and register.

A month ago Stephen Foskett complained about lack of Microsoft’s support for FCoE. I agree with everything he wrote, but he missed an important point: Microsoft gains nothing by supporting FCoE and potentially a lot if they persuade people to move away from FCoE and deploy iSCSI.

FCoE and iSCSI are the two technologies that can help Fiber Channel gain its proper place somewhere between Tyrannosaurus and SNA. FCoE is a more evolutionary transition (after all, whole FCoE frames are encapsulated in Ethernet frames) and thus probably preferred by the more conservative engineers who are willing to scrap Fiber Channel infrastructure, but not the whole protocol stack. Using FCoE gives you the ability to retain most of the existing network management tools and the transition from FC to FCoE is almost seamless if you have switches that support both protocols (for example, the Nexus 5000).

When developing the Choose the optimal VPN service webinar, I decided to test everything I was talking about in a lab (you wouldn’t believe how much misinformation is spread across the Internet) and ended up with several DMVPN scenarios that most people would consider to be somewhere between peculiar and outrageous.

The best one: DMVPN Phase III network with ODR between spokes and level-1 hubs and OSPF inside a hierarchy of hubs ... of course fully redundant all the way down to the spokes.

The webinar has been rescheduled to July 7th (Cisco Live is taking place from June 27th to July 1st).

The design scenarios were simply too god to be left to rot on my hard drive (some of them were screaming to be documented and talked about), so I organized them into a progressively evolving story described in the DMVPN: Advanced and crazy scenarios webinar.

If you’re a CCNP/CCIE-level engineer interested in DMVPN, I’m positive you’ll enjoy this webinar (click here to register) ... and I’ll try to serve you as many curveballs as I can manage to fit within two hours.

Quick summary for the differently attentive: Even without the DNS processing, NAT-PT and NAT64 differ from the perspective of peer-to-peer applications. The differences don’t matter for IPv6 clients connecting to IPv4 servers.

Whenever I’m talking about NAT64, someone would say “we’re already using it”. As it turns out, they’re usually using NAT-PT, which looks a lot like NAT64 from afar (after all, they both allow IPv6-only clients to connect to IPv4-only servers). However, there are significant differences between the two, the most important one being DNS64, which handles DNS processing completely outside of the forwarding path (NAT-PT has embedded DNS Application Level Gateway, which was one of the major reasons NAT-PT was declared broken beyond hope).

The author

Ivan Pepelnjak (CCIE#1354 Emeritus), Independent Network Architect at ipSpace.net, has been designing and implementing large-scale data communications networks as well as teaching and writing books about advanced internetworking technologies since 1990.