The trials and tribulations of a few network engineers

Category: IPv6

As part of our IPv6 deployment we had to upgrade the firmware on our CPEs. We have a small variety of different models, but the majority of them are based on a Broadcom chipset. This firmware upgrade included all the features we needed for IPv6, the DHCPv6 client for the WAN, RA announcements on the LAN etc, but it also included other non-related IPv6 fixes and enhancements.

We spend a lot of time and effort regression testing these firmware pushes, and are generally pretty confident in it by the time we go to mass push it out via TR69. However, shortly after the firmware upgrade we started hearing complaints that this firmware had broken a very specific use case that we hadn’t obviously tested for.

IPv6 tunnels such as the 6in4 ones offered for free by Hurricane Electric. Odd, we hadn’t started the enablement of native IPv6 prefixes for these customers yet, but we did deploy it with ULA RAs enabled, could that be affecting things? We didn’t think so, but we had to investigate obviously.

Problem Statement:

Tunnel establishes correctly. Client gets an IPv6 prefix, can ping6 tunnel end-point as well as other v6 connected servers on the internet.

However, TCP sessions don’t establish over the tunnel.

Diagnosing:

ICMPv6 echo requests and replies are passing fine, so it doesn’t appear to be an obvious routing issue.

First step is to bust out netcat, tcpdump on the client and watch the TCP 3-way handshake and see how far we get.

We see the initial SYN go out fine, and then the reply SYN/ACK from the remote v6 host and we send the replying ACK. All looks good, myth busted, time for the pub….. but wait.. there comes in a duplicate SYN/ACK. Wait what? We send another ACK but yet another duplicate SYN/ACK comes back.

OK, sounds like our ACKs aren’t making it back to the remote host; Time to check the remote side of the session and confirm.

Hrm nope this is IPv6, the minimum MTU size is 1280 and that’s definitely enough for a small ACK. Check the he-ipv6 tunnel iface mtu, 1480. Yup, makes sense, 1500-20 bytes for the 6in4 encapsulation. Quick check to make sure we can get at least 1280 through unfragged:

Yup, we’re golden. Can also ping6 up to 1432 bytes unfragged which matches the advertised MSS value above once we include the 8 byte ICMP header (but I didn’t screen-cap that). Side note: There’s also another 40 bytes for the IPv6 header which adds up to the 1480 byte tunnel MTU we confirmed above.

Right, let’s tcpdump on the WAN interface of the router and have a look at the 6in4 encapsulated packets to see what’s up.

Now this is where it gets fun. Tcpdump on the router’s WAN interface…. and the 3-way TCP handshake establishes. Wait….. what? Stop the tcpdump and no new sessions establish successfully.

Riiiiiiiight.. is the kernel looking at the 6in4 payload destination address and throwing it on the ground unless promiscuous mode is enabled? Nope, tcpdumping with -p (–no-promiscuous-mode) and the TCP sessions work again. OK, what if we tcpdump on the internal bridge interface instead of the WAN? That results in the broken behaviour!

We spent a wee while going back and forth over a few hypothesis and then proving them wrong. Then after a quick chat with one of our CPE developers, he informs us of an MTU issue they saw a while ago affecting large packets when using the hardware forwarding function of the Broadcom chipset known as “fastpath”. But this shouldn’t affect our small 72 byte packet, and besides, it was fixed ages ago.

This gets us thinking; It would explain why tcpdumping could “fix” the issue by punting the packets to the CPU and forcing software forwarding. Let’s test this theory and disable the hardware fastpath… hrmm nope no dice. Let’s disable another fastpath function, this time a software fastpath function called “flow cache”. Lo and behold, the TCP sessions establish!

Myth confirmed, bug logged upstream with Broadcom, time for a well earned pint.

Cheers to @NickMurison for raising the issue and helping me with the diagnostics.

DNS is one of those ancillary services that can often get overlooked, be it recursive, authoritative forward or reverse.

Assigning a Recursive DNS server (RDNSS):

There are 2 main ways to tell a client what RDNSS to use, stateless DHCPv6 (or with stateful if you want to use DHCP for IPv6 address assignment), or built in to the ICMPv6 Router Advertisements (RA) being sent by a router, a la RFC6106.

Sadly Microsoft and Google(Android) disagree on which of these methods is the best; Microsoft will only use DHCPv6 and Android will only use RAs. The fallout of this means that if you want both types of systems on your IPv6 network, you need to announce your RDNSS via both RAs and DHCPv6. The alternative is to just rely on the RDNSS being handed out via DHCPv4 if you’re dualstacked, obviously not a future proofed solution.

Resolving AAAA Records:

AAAA or “quad As” are the IPv6 equivalent of the IPv4 A record, used for forward resolution from a domain name to an IP address. They are intrinsic to your use of the internet, without them your client would not know which server to connect to.

Now to dispel one misconception right off the bat, it is not essential that your RDNSS be contactable over IPv6 in order to deliver an AAAA record. Ie. A dual stacked host can use an IPv4 RDNSS and still be able to browse the IPv6 internet.

For anyone starting to deploy IPv6 in a dual stack environment, it’s important to realise that clients will start requesting both an A and an AAAA record for each resolution attempt, which will effectively double the load on your RDNSS infrastructure.

But when should a client start requesting an AAAA record? The easy answer is obvious, when the client has a public IPv6 address, however in reality it’s not quite as simple as that. As we discovered above, different operating systems, and even different clients within said operating systems, behave differently.

We found that Windows (7/8/10) and OSX all start requesting AAAA records as soon as they have an IPv6 address that isn’t a link-local or Teredo address. This means that even if you haven’t enabled IPv6 on the WAN side just yet, you will still see a drastic increase in RDNSS load when you upgrade your CPEs firmware to support IPv6 if they start handing out Unique Local Address (ULA) addresses.

Oddly enough, Android itself and Chrome on any OS do not seem to request AAAA records when only presented with a ULA address. To me this seems broken, as routable networks can be built using only ULA addressing, and thus could quite feasible want forward DNS entries. ie. Just like on an IPv4 LAN using RFC1918.

There’s a lot of support for deploying IPv6, no one really is saying that you shouldn’t, but the lackluster uptake from large eyeball ISPs tends to grate on those most vocal IPv6 evangelicals. Rest assured however that most of the larger ones have been silently working away on this for some time, working on making their large scale deployments to the mass market as seamless as possible. Those smaller ISPs that lack budget, or those that are flush with IPv4 space, are happy to ride it out letting others deal with the 0day vendor bugs. That’s fine, with the already tight margins in this market, why should they spend money when they don’t need to?

Anywho, I decided to document some of the issues we’ve faced and have had to overcome in the past few years as we start to descend over the precipice of our residential mass market IPv6 rollout. This is by no means a complete list, just a few of the interesting ones that spring to mind.

Authentication

PPPoE with CHAP for authentication will continue to be fine. Your PPP session establishes as normal, IPv6CP negotiates link-local addressing, and then DHCPv6 hands out an IA_NA and/or IA_PD over the top.

IPoE on the other hand, authentication is typically done when the BNG receives a DHCPv4 DISCOVER or DHCPv6 SOLICIT. It’s done with what’s colloquially referred to as “port based authentication”, using either the Circuit-ID or Remote-ID that’s inserted via the DHCP Relay Agent on the Access Node, ie. the DSLAM or OLT. This is often referred to in DHCPv4 land simply as “Option 82” with Circuit-ID being sub-option 1, and Remote-ID sub-option 2.

First issue: not all Access Nodes, or their service providers will support DHCPv6 LDRA insertion of Remote-ID or Interface-ID (the v6 equivalent of Circuit-ID). Openreach being our primary example here in the UK, although their Huawei MSANs do actually support it, Openreach don’t. The impact of this means you’re reliant on DHCPv4 for gleaning this information, and can’t go single-stacked native IPv6 just yet. Doesn’t seem like that big a deal, right? Which brings us to the second issue:

In lieu of having the native Remote-ID/Interface-ID, some BNGs can attribute the v6 session to the same as the v4 if the DHCPv6 SOLICIT is received within a few seconds of the DISCOVER. Nice wee kludge that works when a CPE is freshly booted, but can fall out of sync depending on timers, if v6 is enabled after it’s online, or other CPE quirks. If this happens, the CPE’s existing PD may become non-routable causing end user’s traffic to be blackholed, or the CPE just won’t get a new PD.

OK, so you own your own access network and your DSLAM/OLT supports Lightweight DHCPv6 Relay Agent to insert a Remote-ID, great we now know who sent us that DHCPv6 SOLICIT. Except that DHCPv4 Option 82.1 Remote-ID is different to the DHCPv6 Option 37 Remote-ID. RFC4649 prepends an extra 4 bytes to the front of the Remote-ID, to include the IANA registered Enterprise-ID of the relay agent vendor. Now you’re going to need extra RADIUS logic to strip off those first 4 bytes; to do that it needs to be able to reliably identify an IPv6 triggered ACCESS-REQUEST, which can also be a challenge in itself as the the different format Remote-IDs get inserted in to the same RADIUS attribute, Broadband Forum’s “Agent-Remote-ID”.

Resourcing

On a BNG, like other routers, you have to keep a close eye on the resources being used. On the Alcatel-Lucent 7750-SR, each subscriber connected chews up what they call a “host resource”. As soon as you dual stack a subscriber, that takes up another host resource, thus halving the total number of subscribers you can host on that one BNG.

If you hand out an IA_NA as well as an IA_PD, that uses up yet another host resource. To avoid this waste of resources, you don’t have to assign a public IPv6 point-to-point addressing for use on the CPE’s WAN interface (IA_NA), instead they can just use link local for BNG<->CPE communication.

Side note: In a later R12 release of SR OS it no longer uses a host resource each for IA_NA and IA_PD.

CPE

As we’re not allocating an IA_NA address for use on the WAN interface but rather using the link local addressing, the CPE no longer has a public IP address. Not really a major issue apart from perhaps confusing some helpdesk agents or end users who can no longer ping their CPE as proof their connection is up. We mitigate this with a small custom tweak on the CPE, it claims the first ::1 address from the PD and uses it as a loopback of sorts.

For firewalling, we’ve chosen to follow the RFC6092 recommendations on CPE IPv6 security. Which means we’ll be, by default, allowing all inbound IPSec but blocking all other non-solicited inbound traffic.

That poses an issue that we haven’t really resolved, and I’m unsure as to what the exact impact will be. As a result of NAT on IPv4 a lot of applications utilise UPnP to open up inbound ports for connectivity, this means not just a DNAT entry, but also an inbound firewall rule. Whilst we no longer need NAT with IPv6, that inbound firewall rule will still be required on CPEs that have a default deny policy.

As mentioned in a previous post, there are new UPnP functions which allow for the dynamic creation of IPv6 firewall rules and are actually mandated in the IGD:2 specs; Sadly not many CPEs meet these IGD:2 specifications yet, and even then it will require application developers to update their applications to make use of these new functions.

Another potential issue I foresee with firewalling is more of an end-user training one as opposed to a technical one. Most modern OS will make use of privacy addressing. This is a method of an end-host pseudo-randomly assigning itself temporary addresses to use for outbound connections and then deprecating them after a while to be replaced with new ones. The end result is that an end-host will have a multitude of IPv6 addresses on an interface, including:

Several of the aforementioned Privacy Extension addresses. (Only 1 being used for new outbound connections, but possibly multiple deprecated addresses that were used for older flows)

Hopefully people will realise pretty quickly that the first two aren’t globally routable and aren’t to be used for inbound firewall rules. The issue is that the last two types of addresses are both assigned out of the same prefix handed out via RAs from the CPE and aren’t instantly recognisable by their format. Thankfully most OS will include the word “temporary” next to the privacy addresses, which will hopefully steer end-users to use the EUI-64 address for any IPv6 firewalling rules they decide to manually enter.

Right that’s enough for now, and that’s just a small snippet focusing on the very end of the Internet chain. Hopefully it helps some people, or gives others an idea of what kind of things they should be looking at when doing their own IPv6 deployment.

Apple have recently made waves in the networking world by announcing IPv6 support will be mandatory for apps in iOS9. Now, this is not actually as scary as it originally sounds for app developers; all it really means is that they must avoid the use of IPv4 literals and make use of iOS’s high-level network API’s. In this way, the network is abstracted from the developer and they don’t really need to care about the underlying transport. iOS will decide what to use. If only IPv4 is available, it will use that. If only IPv6, it will use that, although perhaps with the help of DNS64/NAT64 in the providers network if the content itself is IPv4-only. Things get more interesting when both IPv4 and IPv6 are available, the decision is then more complicated.

In dual-stack networks, you could be forgiven for thinking that iOS will make the ”right” decision when presented with a dual-stack connection, unfortunately, real world experience is not proving this to be the case. Of course, the “right decision” means different things to different people. For example, Facebook have recently shown that they see a significant performance increase (Up to 40% decrease in page load-time) when using IPv6 on a major US mobile network. The interesting thing here though, is that they also report that iOS will only choose the IPv6 path ~20% of the time. This seems to be because Apple are basing their decision on RTT even though HTTP throughput is faster via IPv6. This has actually led Facebook to develop mobile proxygen so they have control over the protocol selection process.

So what are developers meant to do? How will they even know if their apps are IPv6 compliant? Fortunately, Apple have thought of this and are providing OS X with the ability to create an IPv6-only hotspot. As far as I know, the details of exactly how this works are not yet public but the best guess seems to be that they will present an IPv6-only connection and then use NAT64/DNS64 to provide connectivity to any content that may be IPv4-only.

This is all great news but there is an elephant in the room: 464XLAT. This is a technique that combines existing translation mechanisms (RFC6146, and RFC6145) to provide IPv4 connectivity to devices connected to an IPv6-only network. This is achieved by providing a client-side IPv4 CLAT, traffic is allowed to enter as IPv4 and is then mapped to IPv6 for transport across an IPv6-only network. This is great for the mobile world; it means UEs (User Equipment) that support 464XLAT can be connected to operators IPv6-only networks without fear of poorly coded apps breaking. The UE itself “fakes” IPv4 connectivity towards any applications that can’t live without it. So why have Apple stubbornly refused to support 464XLAT and show no sign of changing this stance?

It is important to remember that 464XLAT solves a very specific problem; IPv6-only networks, even with DNS64/NAT64 cannot achieve parity with IPv4-only networks when some apps break on an IPv6-only network. This typically happens because the app is using IPv4 literals or making use of legacy IPv4-only APIs. 464XLAT gets around this by presenting IPv4 connectivity so these apps can continue to work. Because of this, major mobile networks (T-Mobile US being a prime example) have been able to provide IPv6-only to their Android users. But Apple are in a unique position, they exercise great control over their store and apps must meet certain requirements to gain entry. One of these requirements will now be that the app *must* work on an IPv6-only network. If developers have coded apps in such a way that they break without IPv4, they will be required to fix this for iOS9. This is an excellent move, just like so many things before, Apple are consigning “IPv4-only” to the history books.

There is still plenty more for Apple (And the rest of the industry) to do regarding IPv6 but I can’t help but be optimistic after hearing this announcement. I hope this move really does help weed out legacy code in apps and Apple listen to experiences gained by Facebook, and others, around improvements to protocol selection. I guess we will find out over the coming months.

There are two rather big misconceptions I often see mentioned when it comes to the IPv6 over Carrier Grade NAT debate.

Firstly that CGN and IPv6 are mutually exclusive. Some folks seem to think that it’s an either/or approach when it comes to deploying these technologies, rather than having them work alongside each other in parallel.

IPv4 isn’t going away anytime soon, CGN should be treated as a transitioning mechanism, not as a method to procrastinate the deployment of IPv6. CGN is capex intensive, requiring special hardware to NAT and keep track of the translation state table. This CGN hardware is typically going to be limited by number of flows and bandwidth throughput, i.e, how many customers and how much they’re using. Providing an IPv6 prefix delegation dual stacked alongside a private CGN IP allows for native IPv6 traffic to bypass these resource (and financially) intensive CGN gateways, meaning more capacity out of them, better IP saving efficiency and most importantly, better user experience.

ISPs don’t want to deploy CGN, it costs us, our customers don’t like it, not to mention the compliance issues involved; however, until the IPv4 internet gets turned off, we’re going to need some form of CGN, NAT64, 464XLAT, 4RD etc.

The second misconception is that IPv6 is a silver bullet, solving all of our NAT issues and providing ubiquitous end-to-end connectivity.

I can’t imagine that many ISPs are going to be deploying IPv6 to their end-users with the CPE firewall disabled by default, so there goes that theory. If you’re lucky your ISP will provide you a CPE that has a nice web UI with an IPv6 firewall that allows you to open up ports to your public IPv6 addresses on your LAN. If you’re really lucky your CPE will allow IPSec to your end hosts by default, as per RFC6092. But what about users that don’t want to or care how to do manual firewall rules, let alone know that they should use the EUI-64 address instead of the privacy extension addresses? What happens if the dynamic PD changes, do they have to login and update the CPE firewall rules each time?

To make sure IPv6 doesn’t negatively impact the user experience we need some sort of dynamic firewall and port forwarding similar to what UPnP used to provide us in IPv4. Well, there is a UPnP function to allow this, but it’s a (relatively) new one specified in IGD:2 that requires both CPE and application support. I don’t know the stats on how widely this is supported by residential CPEs, but my personal experience has indicated “not very”, and even if it is, it requires every application that currently does an IPv4 AddPortMapping() request to do a new IPv6 AddPinhole().

Thankfully, as of the 30th March 2015, the UPnP Forum has deprecated IGD:1 so hopefully we will see a bit more up take amongst the commodity CPEs vendors for this.

As a side note, this IGD:2 spec also includes an AddAnyPortMapping() function that allows a CPE to return a different port to the one requested, in case the original one is in use or otherwise unavailable. This would be great for CGN deployments that incorporate PCP to allow for dynamic port forwarding, but once again requires application developers to support the new function, as well as a UPnP-PCP Interworking function on the CPE.