Current SMB users may find themselves in a situation where they can’t run BGP. Perhaps their upstream ISP blocks the ability to establish a connection. In many cases, business class service is required with additional fees necessary to multihome. In order to take full advantage of independent links to different ISPs, two (or more) NAT configurations are required to send and receive packets correctly across the balanced connections. While technically feasible, it’s a mess to troubleshoot. It also doesn’t scale when multiple egress connections are configured. And more often that not, the configuration to make everything work correctly exists on a single router in the network, eliminating the advantages of multihoming.

LISP seeks to solve this by using a mapping database to send packets to the correct Ingress Tunnel Router (ITR) without the need for BGP. The diagram of a LISP packet looks a lot like an overlay. That’s because it is in many ways. The LISP packets are tunneled from an Egress Tunnel Router (ETR) to a LISP speaking decapsulation point. Depending on the deployment policies of LISP for a given ISP, it could be the next hop router on a connection. It could also be a router several hops upstream. LISP is capable of operating over non-LISP speaking connections, but it does eventually need decapsulation.

Where’s the Achille’s Heel in this design? LISP may solve the issue without BGP, but it does introduce the need for the LISP session to terminate on a single device (or perhaps a group of devices). This creates issues in the event the link goes down and the backup link needs to be brought online. That tunnel state won’t be preserved across the failover. It’s also a gamble to assume your ISP will support LISP. Many large ISPs should give you options to terminate LISP connections. But what about the smaller ISP that services many SMB companies? Does the local telephone company have the technical ability to configure a LISP connection? Let along making it redundant and highly available?

Right Tool For The Job

I think back to a lesson my father taught me about tools. He told me, “Son, you can use a screwdriver as a chisel if you try hard enough. But you’re better off spending the money to buy a chisel.” The argument against using BGP to multihome ISP connections has always come down to cost. I’ve gotten into heated discussions with people that always come back to the expense of upgrading to a business-class connection to run BGP or ensure availability. NAT may allow you to multihome across two residential cable modems, but why do you need 99.999% uptime across those two if you’re not willing to pay for it?

LISP solves one issue only to introduce more. I see LISP being misused the same way NAT has been. LISP was proposed by David Meyer to solve the exploding IPv4 routing table and the specter of an out-of-control IPv6 routing table. While multihoming is certainly another function that it can serve, I don’t think that was Meyer’s original idea. BGP might not be perfect, but it’s what we’ve got. We’ve been using it for a while and it seems to get the job done. LISP isn’t going to replace BGP by a long shot. All you have to do it look at LISP ALternate Topology (LISP-ALT), which was the first iteration of the mapping database before the current LISP-TREE. Guess what LISP-ALT used for mapping? That’s right, BGP.

Tom’s Take

LISP multihoming for IPv4 or IPv6 in SMEs isn’t going to fix the problem we have today with trying to create redundancy from consumer-grade connections. It is another overlay that will create some complexity and eventually not be adopted because there are still enough people out there that are willing to forgo an interesting idea simply because it came from Cisco. IPv6 multihoming can be fixed at the protocol level. Tuning router advertisements or configuring routes at the edge with BGP will get the job done, even if it isn’t as elegant as LISP. Using the right tool for the right job is the way to make multihoming happen.

Advertisements

Share this:

Like this:

Cisco announced this week that they are upgrading the venerable CCIE certification to version five. It’s been about three years since Cisco last refreshed the exam and several thousand people have gotten their digits. However, technology marches on. Cisco talked to several subject matter experts (SMEs) and decided that some changes were in order. Here are a few of the ones that I found the most interesting.

Time Is On My Side

The v5 lab exam has two pacing changes that reflect reality a bit better. The first is the ability to take some extra time on the troubleshooting section. One of my biggest peeves about the TS section was the hard 2-hour time limit. One of my failing attempts had me right on the verge of solving an issue when the time limit slammed shut on me. If I only had five more minutes, I could have solved that problem. Now, I can take those five minutes.

The TS section has an available 30 minute overflow window that can be used to extend your time. Be aware that time has to come from somewhere, since the overall exam is still eight hours. You’re borrowing time from the configuration section. Be sure you aren’t doing yourself a disservice at the beginning. In many cases, the candidates know the lab config cold. It’s the troubleshooting the need a little more time with. This is a welcome change in my eyes.

Diagnostics

The biggest addition is the new 30-minute Diagnostic section. Rather than focusing on problem solving, this section is more about problem determination. There’s no CLI. Only a set of artifacts from a system with a problem: emails, log files, etc. The idea is that the CCIE candidate should be an expert at figuring out what is wrong, not just how to fix it. This is more in line with the troubleshooting sections in the Voice and Security labs. Parsing log files for errors is a much larger part of my time than implementing routing. Teaching candidates what to look for will prevent problems in the future with newly minted CCIEs that can diagnose issues in front of customers.

Some are wondering if the Diagnostic section is going to be the new “weed out” addition, like the Open Ended Questions (OEQs) from v3 and early v4. I see the Diagnostic section as an attempt to temper the CCIE with more real world needs. While the exam has never been a test of ideal design, knowing how to fix a non-ideal design when problems occur is important. Knowing how to find out what’s screwed up is the first step. It’s high time people learned how to do that.

Be Careful What You Wish For

The CCIE v5 is seeing a lot of technology changes. The written exam is getting a new section, Network Principles. This serves to refocus candidates away from Cisco specific solutions and more toward making sure they are experts in networking. There’s a lot of opportunity to reinforce networking here and not idle trivia about config minimums and maximums. Let’s hope this pays off.

The content of the written is also being updated. Cisco is going to make sure candidates know the difference between IOS and IOS XE. Cisco Express Forwarding is going to get a focus, as is ISIS (again). Given that ISIS is important in TRILL this could be an indication of where FabricPath development is headed. The written is also getting more IPv6 topics. I’ll cover IPv6 in just a bit.

The biggest change in content is the complete removal of frame relay. It’s been banished to the same pile as ATM and ISDN. No written, no lab. In it’s place, we get Dynamic Multipoint VPN (DMVPN). I’ve talked about why Frame Relay is on the lab before. People still complained about it. Now, you get your wish. DMVPN with OSPF serves the same purpose as Frame Relay with OSPF. It’s all about Stupid Router Tricks. Using OSPF with DMVPN requires use of mGRE, which is a Non-Broadcast Multi-Access (NBMA) network. Just like Frame Relay. The fact that almost every guide today recommends you use EIGRP with DMVPN should tell you how hard it is to do. And now you’re forced to use OSPF to simulate NBMA instead of Frame Relay. Hope all you candidates are happy now.

vCCIE

The lab is also 100% virtual now. No physical equipment in either the TS or lab config sections. This is a big change. Cisco wants to reduce the amount of equipment that needs to be physically present to build a lab. They also want to be able to offer the lab in more places than San Jose and RTP. Now, with everything being software, they could offer the lab at any secured PearsonVUE testing center. They’ve tried in the past, but the access requirements caused some disaster. Now, it’s all delivered in a browser window. This will make remote labs possible. I can see a huge expansion of the testing sites around the time of the launch.

This also means that hardware-specific questions are out. Like layer 2 QoS on switches. The last reason to have a physical switch (WRR and SRR queueing) is gone. Now, all you are going to get quizzed on is software functionality. Which probably means the loss of a few easy points. With the removal of Frame Relay and L2 QoS, I bet that services section of the lab is going to be really fun now.

IPv6 Is Real

Now, for my favorite part. The JNCIE has had a robust IPv6 section for years. All routing protocols need to be configured for IPv4 and IPv6. The CCIE has always had a separate IPv6 section. Not any more. Going forward in version 5, all routing tasks will be configured for v4 and v6. Given that RIPng has been retired to the written exam only (finally), it’s a safe bet that you’re going to love working with OSPFv3 and EIGRP for IPv6.

I think it’s great that Cisco has finally caught up to the reality of the world. If CCIEs are well versed in IPv6, we should start seeing adoption numbers rise significantly. Ensuring that engineers know to configure v4 and v6 simultaneously means dual stack is going to be the preferred transition method. The only IPv6-related thing that worries me is the inclusion of an item on the written exam: IPv6 Network Address Translation. You all know I’m a huge fan of NAT. Especially NAT66, which is what I’ve been told will be the tested knowledge.

Um, why?!?

You’ve removed RIPng to the trivia section. You collapsed multicast into the main routing portions. You’re moving forward with IPv6 and making it a critical topic on the test. And now you’re dredging up NAT?!? We don’t NAT IPv6. Especially to another IPv6 address. Unique Local Addresses (ULA) is about the only thing I could see using NAT66. Ed Horley (@EHorley) thinks it’s a bad idea. Ivan Pepelnjak (@IOSHints) doesn’t think fondly of it either, but admits it may have a use in SMBs. And you want CCIEs and enterprise network engineers to understand it? Why not use LISP instead? Or maybe a better network design for enterprises that doesn’t need NAT66? Next time you need an IPv6 SME to tell you how bad this idea is, call me. I’ve got a list of people.

Tom’s Take

I’m glad to see the CCIE update. Getting rid of Frame Relay and adding more IPv6 is a great thing. I’m curious to see how the Diagnostic section will play out. The flexible time for the TS section is way overdue. The CCIE v5 looks to be pretty solid on paper. People are going to start complaining about DMVPN. Or the lack of SDN-related content. Or the fact that EIGRP is still tested. But overall, this update should carry the CCIE far enough into the future that we’ll see CCIE 60,000 before it’s refreshed again.

Like this:

I think it’s time to put up a new post on IPv6 and NAT. Mainly because I’m still getting comments on my old NAT66 post from last year. I figured it would be nice to create a new place for people to start telling me how necessary NAT is for the Internet of the future.

In the interim, though, I finally had a chance to attend the Texas IPv6 Task Force Winter 2012 meeting. I got to hear wonderful presentations from luminaries such as John Curran of ARIN, Owen DeLong of Hurricane Electric, and even Jeff Doyle of Routing TCP/IP book fame. There was a lot of great discussion about IPv6 and the direction that we need to be steering adoption of the new address paradigm. I also got some very interesting background about the formation of IPv6. When RFC 1550 was written to start soliciting ideas about a new version of IP, the Internet was a much different place. Tim Berners-Lee was just beginning to experiment with HTTP. The majority of computers connected to the Internet used FTP and Telnet. Protocols that we take for granted today didn’t exist. I knew IPSec was a creation of the IPv6 working group. But I didn’t know that DHCP wasn’t created yet (RFC 2131). Guess what? NAT wasn’t created yet either (RFC 1631). Granted, the IPng (IPv6) informational RFC 1669 was published after NAT was created, but NAT as we know and use it today wasn’t really formalized until RFC 2663. That’s right, folks.

The reason NAT66 doesn’t exist is because IPv6 was built at a time when NAT didn’t exist.

It’s like someone turned on a lightbulb. That’s why NAT66 has always felt so wrong to me. Because the people that created IPv6 had no need for something that didn’t exist. IPv6 was about creating a new protocol with advanced features like automatic address configuration and automatic network detection and assignment. I mean, take a look at the two IPv6 numbering methods. Stateless Automatic Autoconfiguration (SLAAC) can assign all manner of network information to a host. I can provide prefixes and gateways and even default routes. However, the one thing that I can’t provide in basic SLAAC is a DNS server entry. In fact, I can’t provide any of the commonly assigned DHCP options, such as NTP server or other vendor-specific fields. SLAAC is focused solely on helping hosts assign addresses to themselves and get basic IP connectivity to the global Internet. Now, take DHCPv6. This stateful protocol can keep track of options like DNS server or NTP server. It can also provide a database of assignments so I know which machine has which IP. But you know what critical piece of information it can’t provide? A default router. That’s right, DHCPv6 has no method of assigning a default router or gateway to an end node. I’m sure that’s due to the designers of DHCPv6 knowing that SLAAC and router advertisements (RA) handle the network portion of things. The two protocols need to work together to get hosts onto the internet. In 1995, that was some pretty advanced stuff. Today, we think auto addressing and network prefix assignment is pretty passé.

Instead of concentrating on solving the dilemma of increasing the adoption rate of IPv6 past the 1% mark where it currently resides, we’ve instead turned to the Anger and Bargaining phases of the Küber-Ross model, otherwise known as the Five Stages of Grief. The need for IPv6 can no longer be denied. The reality of running out of IPv4 addresses is upon us. Instead, we lash out against that which we don’t understand or threatens us. IPv6 isn’t ready for real networking. There are security risks. End-to-end communications aren’t important. IPv6 is too expensive to maintain. People aren’t smart enough to implement it. Any of those sound familiar? Maybe not those exact words, but I’ve heard arguments very similar to that leveled at IPv6 in just the two short years that I’ve been writing. Think about how John Curran of ARIN must feel twenty years after he started working on the protocol.

Anger is something I can handle. Getting yelled at or called expletives is all part of networking. It’s the Bargaining phase that scares me. Now, armed with a quiver of use cases that perhaps 5% of the population will ever take advantage of, we now must delay adoption or move to something entirely different to support those use cases. It’s the equivalent of being afraid to jump off a diving board because there is a possibility that the water will drain out of the pool on the way down. The most diabolical is Carrier Grade NAT. Let’s NAT our NATed networks to keep IPv4 around just a little longer. It won’t cause that many problems, really. After all, we’ve only got 65,536 ports that we can assign for any given PAT setup. So if we take that limit and extend it yet another level, we have 65,536 PATed PAT translations that we can assign per CGN gateway. That has real potential to break applications, and not just from an end-to-end connectivity point of view. To prove my point, fire up any connection manager and go to http://maps.google.com. See how many separate connection requests are spawned when those map tiles start loading. Now, imagine what would happen if you could only load ten or fifteen of them. There’s going to be a lot of blank spots on the that map.

Now, for the fun part. I’ve been accused of hating NAT. Yes, it’s true. I dislike any protocol that breaks basic connectivity and causes headaches for troubleshooting and end-to-end communications. I have to live with it in IPv4. I’d rather not see it carried forward. That’s the feeling of many IPv6 evangelists. If you think I dislike NAT, ask Owen DeLong his feelings on the subject. However, to say that I dislike NAT for no good reason is silly. People are angry at me for saying the emperor has no clothes. Every time I discuss the lack of need for NAT66, the same argument gets thrown in my face. Ivan Pepelnjak wrote an article about about using network prefix translation (NPT) in a very specific case. If you are multihoming your network to two different providers and not using BGP then a case for NPT can be made. It’s not the best solution, but it’s the easiest. Much like Godwin’s Law, as the length of any NAT66 argument increases, the probability of someone bringing up Ivan’s article approaches approaches one.

So, I’ve found a solution to the problem. I’m going to fix this one scenario. I’m going to dedicate my time to solving the multihoming without BGP issue. When I do that, I expect choirs of angels to sing and a chariot pulled by unicorns to arrive at my home to escort me to my new position of Savior of IPv6 Adoption. More realistically, I expect someone else to find a corner case rationale for why IPv6 isn’t the answer. Of course, that’s just another attempt at bargaining. By that point, I’ll have enough free time to solve the next issue. Until then, I suggest the following course of action:

Share this:

Like this:

Fresh off my recent fame from my NAT66 articles (older and newer), I decided first thing Monday morning that a little experiment was in order. I wanted to express my displeasure for sullying something like IPv6 with something I consider to be at best a bad idea. The only thing I could come up with was this:

New crusade: Add a Start Menu to OS X. Another OS has it and it makes my life easier. You'd have to be an elitist monk to not want it.

The response was interesting to say the least. Questions were raised. Some asked if I was playing a late April Fools joke. Others rounded up the pitchforks and torches and threatened to burn down my house if I didn’t recant on the spot. Mostly though, people made sure to express their displeasure by educating me to the fact that I should use something else to do what I wanted rather than rely on the tried-and-true metaphor of a Start Menu.

Now do you see what I’m talking about with NAT66? Some people think that NAT is needed not because it’s a technological necessity. Not because it’s solving fifteen problems that IPv6 has right now. They want NAT because they really don’t understand how things work in IPv6. It’s the same as bolting a Start Menu on to OS X. When I started using my new MacBook a few months ago, I took the time to figure out how to use things like Spotlight and Alfred. They weren’t my Start Menu, but they worked almost the same way (in many cases better). I didn’t protest the lack of a metaphor I clearly didn’t need. I adapted and overcame. And in the end I found myself happier because I found something that worked better than I had hoped.

In much the same way, people that crave NAT on IPv6 are just looking for familiar metaphors for addressing. I’m going to cast aside the multihoming argument right now because we’ve done that one to death. Yes, it exists. Yes, it needs to be addressed. Yes, NPT is the best solution we’ve got right now. However, when I started going through all the comments on my NAT66 blog post after the link from the Register article, I noticed that some of the commenters weren’t entirely sure how IPv6 worked. They did understand that the addresses being assigned to the adapters were globally routable. But some seemed to believe that a globally routable address meant that every device was going to need a firewall along with DDoS protection and ruleset monitoring. Besides the fact that every OS has had a firewall since 2002, let me ask one question. Are you tearing out your WAN firewall when you move to IPv6? Because as far as I know, you still one have one (maybe two) WAN connections that are terminated on some device. That could be a router or a firewall. In the IPv4 world, that device is doing NAT in addition to controlling which devices on the outside can talk to the inside. Configuring a service to traverse the firewall is generally a two-stage process today. You must configure a static NAT entry for the device in question and then allow one or more ports to pass through the firewall. It’s not too difficult, but it is time consuming. In IPv6, with the same firewall and no NAT, there isn’t a need to create a static NAT entry. You just permit the ports to access the devices on the inside. No NAT required. If you don’t want anyone to talk to the devices on the inside, you don’t configure any inbound rules. Simple as that. When you need to poke holes in the firewall for things like web servers, email servers, and so on, all you need to do is poke the hole and be done.

Perhaps what we really need to end this NAT issue is wildcard masking for IPv6 addresses in firewalls. I have no doubt that eventually any simple home network device that support DHCPv4 today will eventually support DHCPv6 or SLAAC in the near future. As fast as new chipsets are created to increase the processing power we install into small office/home office devices, it’s inevitable that support will come. But to support the “easy” argument, what we likely need to do is create a field in the firewall that says “Network Address”. That would be the higher ordered 48 bits of the IPv6 address. Once it’s plugged in, the hosts will use DHCPv6 or SLAAC to address themselves. Then, we select the devices from a list based on DNS name and click a couple of checkboxes to allow ports to open for inbound and outbound traffic. If a customer is forced to change their address allocation, all they need to do is change the “Network Address” field. Then, software on the backend would script changes to DHCPv6/SLAAC and all the firewall rules. DNS would update automatically and all would work again. Perhaps this idea is too far fetched right now and the scripting necessary would be difficult to write at the present time. But if it answers the “easy” outcry about IPv6 addressing without the need to add NAT to the protocol, I’m all for it. Who knows? Maybe Apple will come up with something just this easy.

Tom’s Take

For the record, I really don’t think there needs to be a Start Menu in OS X. I think Spotlight is a perfectly fine way to launch programs not located on the dock and find files on your computer. Even alternatives like Alfred and Quicksilver are fine for me. The point of my tweet and subsequent replies wasn’t meant to advocate for screwing up the UI of OS X. It was meant to show that while some people think that my distaste for NAT is silly, all it takes is the right combination of silliness to get people up in arms. To all of you that were quick to jump and offer alternatives and education for my apparent lack of vision, I say that we need to focus effort like that into educating people about how IPv6 works or spend our time figuring out how to remove the roadblocks standing in the way of adoption. If that means time writing scripting for low-end devices or figuring out easy UI options, so be it. After all, someone else has already figured out how to create a Start Menu on a Mac:

Share this:

Like this:

I think my distaste for NAT is pretty well known by now. I’ve talked time and again about how I believe that NAT is a bad idea, especially where IPv6 is concerned. I’d said my peace and had time for good conversations with my friends Ivan Pepelnjak (@ioshints) and Ed Horley (@ehorley) about the subject. I was content to just wear my “I HATE NAT” t-shirts to conferences and let bygones be bygones. Then, suddenly…

Some of you have seen my responses before. Maybe you’ve even been amused by them. Coupled with the fact that I tend to lean toward the snarky side of things, I can see where I might come off as a bit of a smart ass. But “belitted?” “chastened?” Wow. Maybe I’ve let my passion blind me to the plight of the Small-to-Medium Enterprise/Business (SME/B) network/server folks. Maybe we really should stop trying to undo years of duct tape patches to networking and embrace the fact that NAT is a great thing because it allows the little guys to spend less time changing ISPs and deciding to renumber their internal networks on a whim. In fact, I’m even considering calling all my buddies at the IETF and rescinding the whole idea of IPv6. I mean, after all what good is renumbering the Internet if it breaks such a fundamentally important protocol like NAT?

Oh, sorry. I just couldn’t keep a straight face anymore…

In all seriousness, Trevor Pott (@cakeis_not_alie) brings up some very interesting points in his discourse on the impact of IPv6 for the Small-to-Medium Enterprise/Business (SME/B). I’m even willing to admit that we might have glazed over some of the lower-end aspects of what a change like this will mean to people on the razor’s edge of IT. But in the article, the painting of networking professionals as uncaring dictators of fiat laws is almost as silly as characterizing me as a belittling jackass. Yes, I write some pretty pointed posts about NAT and IPv6. Sure, I have a lot of fun playing the heel. But I would hope that my points are made from somewhat sound networking reasoning and not simply blind hatred of NAT and those that use it. Yes, especially those on the edge of networking.

When I was an intern at IBM Global Services in 2001, I had my first real exposure to the way networking operates. I spent a lot of time configuring static IP addresses on devices and using DHCP on others. I got a real eye opening experience. It even colored my perception of networking for a few years to come, although I didn’t know it at the time. You see, as one of the “old guard” networking companies, IBM has their own registered /8 (Class A) network prefix. Everything inside IBM runs on 9.0.0.0/8. Apple similarly has 17.0.0.0/8. These folks have the luxury of a globally routable IP space large enough that they never (realistically) have to worry about running out. For many years afterwards, I couldn’t understand why I was unable to reach my 192.168.1.0/24 network at home from my university network. It wasn’t until I really started learning more about networking that I realized that RFC1918 existed and NAT was in place to allow ever-growing LANs to have Internet access in absence of registered IP space like I had enjoyed at IBM. As time moved on and I started becoming involved with more and more network services that are affected by NAT, I began to see what IBM’s /8 could offer an enterprise. The flexibility of being static. By having their own IP space, IBM didn’t have to change their address structure to suit the needs of users. They never had to worry about changing ISPs and renumbering their network. Everything just stayed the same and we went on with our lives. But, as Trevor Pott pointed out in the article, IBM and Cisco and Juniper and Apple are enterprises. They aren’t…small.

On the polar opposite end of the scale, we have the little guys. The folks that keep law offices running on a SOHO router/firewall/DHCP server. The accounting offices that can only get a /28 or a /29 IPv4 block from their ISP. Folks that look at duct tape as a solution and not a patch. The “cost conscious” customer as one might say. I can identify with many of these customers because their are my customers in my day job. I’ve had to renumber a publicly addressed network on the fly on a Saturday morning. I’ve had to reconfigure firewalls because the ISP decided to reclaim IP space from a customer. It’s a giant pain in the exhaust port. It’s not glamorous or fun. It’s a necessary evil. But is it a reason to rail against IPv6?

In my previous posts, I talked about the issues with IPv6 on the small end of the scale. Sure, we’ve got a lot of addresses to hand out. We’ve also got a lot of configuration to do. We have to reexamine our stand on firewalls and routes and DNS and a lot of other things. Yes, I will freely admit that this isn’t going to be cheap. I’ve already started building the costs of these analyses into the contracts I sign with my customers for the coming year because I know it will need to be done and I don’t want them to be surprised when they get the message from their provider that the time has come to renumber. But I’ve also got another solution in mind for some of my most “cost conscious” customers and readers. I don’t really want to spill the secret sauce, but here goes:

If it’s going to bother you that much, don’t use IPv6.

Plain and simple in black and white (and red). Unless your ISP is going to start charging you an inordinately high monthly fee to keep an IP block you’ve had for years, don’t change. Stay on IPv4. There’s a whole world out there that is never going to move from IPv4 and be perfectly happy. People who run equipment that will never have enough memory or CPU power to process a naked IPv6 packet (let alone a NATed or NPTed packet). People who are mandated to use translated addresses because of some kind of regulatory oversight like the Payment Card Industry (PCI). I really don’t mean to sound like I’m snorting derisively with this advice. If the additional cost of maintaining an IPv6 network with things like link local addresses and proper DNS resolution and multiple firewall translations isn’t worth it to you and your customer base, then stay where you are. No one will come to your office and put a gun to your head to make you move. The issues we have with address space exhaustion inside enterprises are a wholly different animal than keeping the small office going. Honestly, you folks will stay in business for years to come serving a subset of the Internet populace. There may come a time when there is an opportunity cost of being able to reach new customers that are IPv6-only, but that cost will either be balanced by the need to trade out your “low cost” equipment for something that will run newer IPv6 features or it will be balanced out by your need to get any business to offset falling revenues due to IPv6-only customers no longer being able to reach your site on IPv4.

If you’re an SME/B network admin that’s still reading this, I’d highly recommend that you take a moment to think about something though. Is IPv6’s insistence on one-to-one communications and move away from NAT really disrupting the way the Internet works? Or is it moving back to the way things were before? Setting right what once went wrong? One of the funny things about information technology that I’ve noticed can best be summed up by a quote from the new Battlestar Galactica, “All of this has happened before. All of this will happen again.” Think about mainframes. We used to do all of our work from a dumb terminal that gave us a window to a large processing system that housed everything. Then we decided we didn’t like that and wanted all the processing power to live locally in minicomputers and client/server architecture. Now, with the advent of things like virtualization and virtual desktop infrastructure (VDI), we’ve once again come back to using dumb terminals to access resources on large central computers. All of this has happened before. And when we get constrained on our big hypervsior/VDI servers, we’ll go right back to decentralized processing in a minicomputer or client/server model once more. All of this will happen again.

In networking, we moved from globally routable address space on all of our nodes to running them all behind a translated boundary. At first we did it to prevent exhaustion of the address pool before a suitable replacement was ready. But as often is the case in networking (and information technology for that matter), we patched something and forgot to really fix the problem. NAT became a convenient crutch that allowed network admins to not have to worry about address renumbering and creating complex (even if appropriate) firewall rules. I’m just as guilty of this as anyone. It was only when I realized that many of the things that I want to do with networking, such as video conferencing, require more effort to work with NAT than they would otherwise. We spent so much time trying to patch things to work with the patch that we forgot what things looked like before the patch. I’d argue that getting back to end-to-end communications to “fix” protocols like SIP and FTP is just as important as anything. Relying on Skype to do VoIP/video communications just because it doesn’t care about NAT and firewalls isn’t good design. It’s just an inexpensive way to avoid the problem for a little longer. The funny thing about IPv6 is that while there is a huge amount of configuration up front and a lot of design work, when things are configured correctly, stuff just works. Absent of a firewall in the middle, I can easily configure an end-to-end connection directly to a system. Before you say that something like that is only important to an enterprise, think about something like remotely supporting a small office. I no longer have to poke holes in firewalls and create one-to-one NAT translations to remotely connect to servers. I can just fire up my RDP client (or your screen sharing tool of choice) and connect easily. No fuss, no muss, and no NAT needed.

I’ve also said before that I now see that there is a use case for Network Prefix Translation (NPT). Ivan has talked about it before and showed me the light from the networking side. Ed Horley has also given me a perspective from the Microsoft side of things that has changed my mind. But exhorting NPT as a solution to all of our NAT problems in IPv6 is like using a butter knife as a screwdriver. NPT was designed to solve one really huge issue – IPv6 multihoming. Announcing address space to two different upstream providers, which is easier to do with NAT in IPv4 than it currently is in IPv6 absent of the solution provided in RFC6296. NPT for multihoming is a good idea in my mind because of the inherent issues with advertising multiple address spaces to different providers and configuring those addresses on all the internal links in an organization. I also believe that NPT is a transition mechanism and will allow us to start “doing it right” down the road when we’ve overcome some of the thinking that we’ve used with IPv4 for so long. One-to-one NAT makes no sense to me in IPv6. Why are you hiding your address? The idea is that the device is reachable, whether it be a web server or a video conferencing unit. Why force a translation at the edge for no apparent reason? Is it because you don’t want to have to re-address your internal network devices?

Absent the aforementioned multihoming issues, let’s talk about renumbering for a second. How often to you really renumber your internal network? At the company that I work for, we’ve done it once in ten years. That’s not because we were forced to. It was because we ran out of space and needed to move from a /24 to a /23 (and now maybe to a /22). We didn’t even renumber half the devices in the network. We just changed a couple of subnet masks and started adding things in new subnets that were created. Now, granted, that was with an RFC1918 private address space internally. However, with SLAAC/DHCPv6 and IPv6, renumbering isn’t that big of a pain. You just change the network ID that is being handed to your end nodes. Thanks to EUI-64 addressing, the host portion of the address won’t change one bit. And Trevor Pott points out in the article that enterprises assume that DNS resolution will take care of the changeover just before he snorts derisively about how no one has managed to make it work yet. I’d argue that he’s more right than he knows. I have the IP addresses of hundreds of customers memorized. Most of them are RFC1918. Some are not. All of them are dotted decimal octets. I know that when I move these customers to IPv6, I will be relying on DNS resolution to reach these end nodes. My days of memorizing IP addresses are most definitely coming to a middle. And for those that might scoff at the ability of a DHCP server to register and maintain a database of DNS-to-host address mappings, you might take a look at what Active Directory has been doing for the last twelve years. I say that because in my experience, many SME/B networks run some form of Microsoft operating systems, even if it is just for directory services.

I’d like to take a moment to talk about “small” enterprises versus “large” enterprises. For most people, the breaking point is usually measure in costs or in devices. As an example, if you have more than 1000 devices, you’re large. If you have less than 50, you’re small. Otherwise, you’re in the middle (medium). Me? I don’t like those definitions. 10,000 devices in a flat Layer 2 network is (relatively) simple. A 10-person shop doing BGP multihoming and DMVPN is more of an enterprise than the previous example. For those networking admins that are running tens or even hundreds of servers, ask yourself what you really consider yourself to be. Are you a small enterprise because you have a Linksys/D-Link/Netgear Swiss Army Box at your edge? Or are you really a medium-to-large enterprise because of what you’re doing with all that horsepower? Now ask yourself if you want your network to be easy to configure because that’s the way networks should be, or is it really because you’re understaffed and running far more infrastructure that you should be? I’m not going to sit here and say that you just throw more people at the problem. That’s never the right answer. In fact, it’s usually the one that gets you laughed at (or worse). Instead, you should examine what you’re doing to see why wholesale renumbering or network changes are even necessary in the first place. One of the main points of the article is that IPv6 will allow network admins to finally be able to create hundreds of VMs on a single physical server and make them reachable from the global Internet. I would counter with the idea that if the only thing truly holding you back from doing that has been address space, the SME/B that you work for has really been a large enterprise wolf in small enterprise sheep’s clothing all along.

Now, if you’re still with me this far you should congratulate yourself. I’ve expounded a lot of thoughts about the technical reasons behind the way IPv6 behaves and why there are difficulties in applying it to the SME/B. I also wrote all that in isolation on an airplane. As soon as I stepped off and got my Internet lifeline back, I checked up on the original article and noticed that Trevor Pott had clarified his original intent at the bottom of the post with a long comment. Being no stranger to this myself, I read on with measured intent. What I came away with galvanized my original thoughts even further. Allow me to restate my original point a little more pointedly:

If “cheap” and “simple” are your two primary design goals, IPv6 probably isn’t for you.

We’ve gone through this whole problem before in the infancy on the Internet. Last year, Vint Cerf gave a talk at Interop about the problem of protocol adoption. One of the stories I love from this talk involved Mr. Cerf’s attempt to spur the adoption of TCP/IP over the then-dominant NCP protocol. He needed to drive people away from NCP, which wouldn’t scale into the future, and force them to adopt TCP/IP. But adoption rates plateaued quite often as network operators just became comfortable that NCP would always be there to do all the work. Mr. Cerf eventually solved his adoption issues. How did he do it? He turned off NCP for a couple of hours. Then for a day. Then for a week. He drove adoption of a better protocol through sheer force of will and an on/off switch. Now, we all know that we can’t do that today. The Internet is too vital to our global economy to just start shutting things off willy-nilly. Despite that, “cheap” and “simple” aren’t design goals for the Internet core or even the ISP distribution layer. We have to have a protocol that will scale out to support the explosion of connected devices both now and in the coming years. Enterprise providers like Cisco and Juniper and Brocade are leading the charge to provide equipment and services to support this in-state transition. There will be no shutdown of IPv4. This is a steady-state parallel migration to IPv6. These kinds of things don’t come without a cost of some kind. It may not be in the form of a purchase order for a new network core. It may not even be in the form of a service contract to a consultant to help engineer a renumbering and migration plan. The cost may be extra hours reconfiguring servers. It may be taking more time to read RFCs and understand the challenges inherent in reconfiguring the largest single creation in the history of mankind at a fundamental level.

Economies of scale are a good thing. They bring us amazing products every day. They also enable us to spend less time configuring or working and spend more time on creating solutions. The first time you tried to ride a bicycle was probably difficult. As you practiced and progressed it became easier. Soon, you could ride a bike without thinking about it. You might even be able to ride a bike with holding the handlebars or ride it standing on the seat (I never could). That kind of practice and refinement is what is needed in IPv6. We have to make it work on a large scale first to get the kinks worked out. Every network vendor does this. Yes, even the ones that only sell their wares at the local big box retailer. Once you can make something work on a big scale, you can start winnowing down the pieces that are necessary to make it work on the small scale. That’s where “cheap” and “simple” come from. No magic wand. No easy button. Just hard work and investment of time and money.

Tom’s Take

Spurring us “priestly” networking people to change the way things work is a very valid goal and should be lauded. Doing it by accusing us of being obstinate and condescending is the wrong way to do things. I don’t consider myself to be a member of the Cabal of IETF High Priests. I’m not even a member of the IETF. Or the IEEE. I’m a solutions guy. I take what the academics come up with and I make it work in real life. Yes, much like Trevor Potts, I’m a blogger. I like to take positions on things and write interesting articles. Yes, I lampoon those that would seek to hobble a protocol I have high hopes for with thinking from fifteen years ago for the sake of making things “simple”. I’d rather be spending my time working on ways to reduce the time and effort needed to roll out IPv6 everywhere. I’d rather focus on ways to make it easier to renumber the “hundreds” of VMs I typically see at my local small business. In the end, I want what everyone else wants. I want an Internet that works. I know that it may take the rest of my career to get there. But at the end of the day, if I’m forced to choose between making the best Internet I can for the sake of everyone or making it “cheap” or “simple”, then I’ll sacrifice and pay a little more in time and costs. It’s the least I can do.

Share this:

Like this:

Welcome to my first NAT post of 2012. After spending some time during the holidays unwrapping new tech toys and trying to get them to work on my home network, I’m full of enough vitriol that I need to direct it somewhere. Based on the number of searches for “double NAT” that end up on my blog, I thought it was only fitting that I direct some hate toward NAT444, also called carrier-grade NAT or large-scale NAT.

Carrier-grade NAT is the brainchild of the ISP world. It turns out that we may be running out of IP addresses. Shocking, right? We’ve all known for at least a year that we were on the verge of running out of IPv4 addresses. I even said as much last February. The ISPs seem to have decided that IPv4 is still a very important business model for them and the need to continue using it over IPv6 is equally important. My best guess is that many consumer-oriented ISPs looked at their traffic patterns and found that the majority of them were dominated by outbound connections. This isn’t shocking when you consider that the majority of devices in the home aren’t focused around serving content. In fact, many residential ISPs (like mine) tend to block connections on well-known server ports like 25 and 80. This serves to discourage consumer users from firing up their own mail and web servers and forces them to use those of the ISP. It also makes the traffic patterns outflow dominant.

With the lack of availability of IPv4 addresses, the ISP need to find a way to condense their existing and new traffic onto an ever-dwindling pool of available resources. Hence, NAT444. Rather than handing the customer an global IPv4 address for use, the ISP NATs all traffic between their exit points and the customer premise equipment (CPE):

In this example, the subscribers may have an address space on their devices in the 192.168.x.x/24 space. The ISP would then assign an address to the CPE device in the 172.16.x.x./16 space or the 10.x.x.x/8 space. That traffic would then bent sent through some kind of NAT gateway device or cluster of devices. Those devices would function in the same way that your home DSL/Cable router functions when translating addresses, only on a much larger scale. The amount of addresses the ISP current has in their pool would not need to be significantly increased to compensate for a larger number of subscribers, just as if buying a new XBox doesn’t require you to get a new IP address from your ISP.

NAT444 has its appealing points. It’s helpful in staving off the final depletion of the IPv4 address space from the provider side of things. It will help keep IPv4 up and running until IPv6 can be implemented and reduce the pressure on the address space. Yeah, that’s about it…

NAT444 has drawbacks. Lots of them. First, you are adding a whole new layer of complexity onto your ISP’s network. Keeping track of all those state tables and translations for things like lawful intercept is going to be a pain. Not to mention that the NAT gateway devices are going to need to be huge, or at the very least clustered well. Think about how many translations are going through your CPE device at home. Now multiply that by the number of people on your ISP’s network. Each of those connections now has to have a corresponding translation in the NAT table. That means RAM and CPU power. Stupidly big boxes for that purpose. What about applications? We’ve already seen that things like VoIP don’t like NAT, especially when SIP hardcodes the IP address of the endpoint into all of its messages. Lucky for me, a group already did some testing and published their results as a draft RFC. Their findings? Not so great if you like using SIP or seeding files with BitTorrent (hey, it has legitmate uses…). They also tested things like XBox Live and Netflix. Those appear to have been bad as late as last year, but may have gotten better as of the last test. Although, I don’t think testing Netflix streaming for 15 minutes was a fair assessment. You can also forget about hosting anything from your own network. No web, no email, no peer-to-peer gaming sessions over a NAT444 setup. I’m sure your ISP will be more than happy to provide you with a non-NAT444 setup provided you want to upgrade to “premium” service or move to a business account with all the associated fees.

I leave you with a this small reminder…

Tom’s Take

I had one of those funny epiphanies when writing this post. I kept holding down the shift key when typing, so NAT444 kept turning into NAT$$$. That’s when it hit me. NAT444 isn’t about providing better service for the customers. It’s about keeping the whole mess running just a little while longer with the same old equipment. If the ISPs can put off upgrading to IPv6 for another year or two, that’s one more year they don’t have to spend their budgets on new stuff. Who cares if it’s a little harder to troubleshoot things?

In the end, I think NAT444 will be dead on arrival, or at the most shortly thereafter. Why? Because too many things that end users depend on today will be horribly broken. Sure, I can grouse about how NAT444 breaks the Internet and is horrible from a design perspective. I am the I Hate NAT Guy, after all. But try telling the average suburban household that they won’t be able to watch a streaming Netflix movie or play Call of Duty over XBox live anymore because we didn’t plan to keep the Internet running with a new set of addresses. Those people won’t wax intellectual about their existential quandary on a blog. They’ll vote with their dollars and go to an ISP that doesn’t use NAT444 so all their shiny new technology works the way they want it to. In the end, NAT444 will end up costing the ISPs big $$$.

Share this:

Like this:

I’m sure that you’ve probably seen the now-famous Double Rainbow video somewhere on the Internet or television. It has spawned thousands of time-wasting videos. Allow me to make that 1,001.

Gerren Murphy (@Smurficus) threw down the gauntlet on Sept. 27th with this tweet. I thought it might be fun to try, so I fired up the Flip and promptly got writers block. How to show what double NAT really looked like? That’s when I stumbled across the Wikipedia page for carrier-grade NAT. The picture there served my purpose just fine, so I pulled it up in full screen and let my best method acting take over:

I would like to thank the Academy for any awards I might win in the future. Although I apologize to my readers for not being as…augmented as the original video director.

If you’d like to learn more about carrier-grade NAT (also called NAT444), please head to Ivan Pepelnjak’s blog and check out his NAT444 post.