Category Archives: IPv6

I have been using RHEL/CentOS lineage Linuces for a good while now, and I believe the strength and distro mindset is their dedication to package dependencies. Some packages or software that require bleeding edge or even not so recently updated/released libraries/CPAN perl modules etc don’t behave well or won’t install/compile without breaking those dependencies without installing source packages. This paradoxally is the weakness of RHEL/CentOS, their dedication to dependencies They occasionally are a bit behind when it comes to the latest and greatest, and you cat get handcuffed if you want to remain within the confines of the dependancy sandbox.

In any case I wanted to toy with some software that clearly seems written for Ubuntu (first tell-tale sign is that you need to add a bunch of user-contributed repositories so you can install it….) This software requires IPv6 connectivity so I decided to trash a BIND virtual server I was using and replace it with Ubuntu 14.04 from the .ISO. I am not totally foreign to Ubuntu as I use Ubuntu desktop on a couple laptops, however in a server environment I have never bothered to consider it out of laziness, mostly. Here are a few observations, some of which I am still trying to wrap my head around.

BIND startup

My first challenge was getting BIND to start correctly. I usually use “listen-on-v6” to specify what interfaces I allow BIND to bind to, and was seeing this incredibly annoying message on bootup and BIND was not binding to any IPv6 addresses, defeating the purpose of running BIND on this server in the first place:

apparently, Ubuntu is in such a hurry to boot that it starts daemons before the interface init is even finished (!) Major-wtf. So it would seem that it is prudent (and necessary?) to put the IPv6 definition of eth0 inet6 *before* IPv4. Guess that is good to know Once thats done, BIND starts correctly and we can move on.

IPv6 default gateway gong-show

My next hurdle was not far away. Even if I manually set the gateway, for some reason Ubuntu seems to feel it necessary to send a router solicitation… My network has regular router advertisements disabled, just because I don’t want SLAAC to work in that particular test environment. So even if I statically configure Ubuntu with a prefix, mask and gateway, it still seems to feel a need to go exploring for routers and sends out an ICMPv6 router solicitation…so when I check the routing table:

now wait a minute! My network does not send periodic RAs, I have statically defined my gateway, and Ubuntu overrides whatever I defined with auto-learned crap?? After 500 seconds, the RS-learned gateways disappear and whatever I defined as default then remains, and sometimes no gateway at all…..

Link-Local Gateway

Remember I mentioned above the bit about putting the IPv6 config before IPv4? My network uses HSRP as first-hop redundancy, and IOS didn’t allow for global addressing (and some supported IOS trains do not have this feature) until “recently”. In any case, I want to use a link-local address as a gateway. For some reason, if the IPv6 interface in Ubuntu is defined *after* the IPv4 address, the LL gateway sometimes is ignored (what?) So not only do I get possibly bogus gateways via ICMPv6 router solicitation, I might end up with no gateway at all once those RAs lifetime is up. not good.

The solution is to disable router-advertisement learning in /etc/sysctl.conf by adding the following:

and contrary to RHEL/CentOS (and pretty much any environment where a link-local is used to route something that I have seen), it is not necessary to specify the interface – I suppose Ubuntu is doing some logic there:

So we note the absence of %device. I would have to test if with multiple NICs this still works. Hopefully it does

Network restart broken

My last adventure was trying to restart networking in some controlled way after making changes. I believed I had broken my install when I got the following as a result:

After some googling, apparently this is known. So shucks, another thing that will hopefully get fixed At the rate Ubuntu seems to update packages, I suppose it might not be a long wait

I have been running around trying to figure out why my Samsung Galaxy S4 (Android 4.3) seems to lose its IPv6 prefix when it has been sleeping for a while. At first I blamed my edge CPE device, as I had rebuilt a OpenWRT recently and switched from WIDE-DHCPv6+RADVD to 6relayd. And I had never noticed if the previous implementation had this issue or not.

Symptoms: If I leave the device idle for a certain amount of time (which apparently would be longer than the preferred life of the prefix), when I open the browser I note that v6-sites are not reachable until my router sends an unsolicited RA. I installed some free apps such as MyIPv6 and Network Info ii and although all seemed normal during phone use, after a certain time the prefixes would be non-functional, and would even not appear in these apps as existing, until the next RA sent by the router and all would resume correctly.

As well, if for whatever reason the prefix received by prefix delegation changes, an idle device (read: screen off….) will only learn of the new prefix once the route sends an RA containing that new prefix.

OK, well I tested this, as did the reporter of this issue. Simply locking the phone (turning off the screen), with an ICMP ping to the IPv4 and IPv6 addresses respectively. The IPv6 address of my phone ceases to reply to ICMP echo requests when I turn off the screen (as expected) – YET IPv4 REQUESTS ARE NOT IGNORED and replies are sent by the GS4! So what gives, Samsung devops? This is with or without the power saving features enabled. I don’t believe this cockamamie notion about running down batteries – its either laziness, incompetence or an outright dismissal of a bug.

And frankly, I wouldn’t really care if ICMP echo/echo-reply data was discarded, however ND, RA and other critical v6 traffic shouldn’t be. Or, at the very least, send a solicited RA or reset the IPv6 stack entirely, because when the phone wakes from sleep or is unlocked, the IPv6 stack is broken up until a router sends an RA. And in my case, 6relayd seems to send one every 7 minutes or so, with a randomized +/- interval of a few seconds. And some networks I have connected to send RAs every hour, not every few seconds, since IPv6 protocols allow for address assignment to occur without RAs….So it could be a long wait! And even if a network sends an unsolicited-RA every 30 seconds, that is 30 seconds where your connectivity is v4-only when you benefit from a dual-stacked environment. Not very modern-ish behaviour by a so-called industry leader!

I hate to admit it, but I tested iPhone4, 4S, 5 and iPad mini as well as Windows XP/7, and they all behave correctly when woken from whatever state they were in post-prefix-expiry. I will be borrowing some other Android devices as I can get my hands on them to compare results. But as of now I don’t have a very high regard for Samsung’s IPv6 implementation!

Samsung coders need to get a clue! “End users can connect to networks continually by IPv4” is not an acceptable answer. As IPv6 grows in popularity and necessity, that is a pretty bone-headed opinion. Need I point out that at the time of this writing, that forum post is less than 6 months old!

Somebody at Cisco obviously found this too easy, so it is now required to re-engineer this functionality into “Flexible Netflow”. It does allow you, via the creation of a flow record, customize your own format, which is beyond this simple blog post and likely my current understanding as well ;).

For my purposes, I simply want to export IPv6 flows to my collector for business as usual. This is done by defining a flow exporter and flow monitor, attaching the exporter to the monitor, in the gobal configuration, and applying it to the interface somewhat as before:

I am starting to experiment with IPv6-only networks and require the use of a DNS64 service to be used in conjunction with NAT64. Luckily, BIND 9.8.x offers this capability. However, I don’t want my DNS server to reply with a quad-A record for every query now that I have enabled DNS64, since this DNS server is used by non-v6 clients, as well as dual-stack clients that I do not want to send them searching for 64:ff9b::/96. :S I therefore added a secondary IPv6 address on the DNS server for my v6-only clients to use specifically for DNS64, and put it in its own view. In the config below, the BIND DNS64 service will utilize 2001:db8::64 🙂

I don’t know if this is a late night test, however apparently Facebook is publishing (I am guessing deliberately) a AAAA via DNS, even for “non-whitelisted” resolvers…

They had announced in April that they would enable AAAA for “beta.facebook.com” for May 18th, yet the plan at that time was to enable AAAA on the www. on June 6th like everyone else. However dig currently has results that push forward that deadline:

Their initial IP on June 8th 2011 was [2620:0:1cfe:face:b00c::3] (aka www.v6.facebook.com, and www.facebook.com for the 24 hours of World IPv6 Day 2011) , which is within a /40 out of ARIN space. beta.facebook.com and www.facebook.com do not have the same IP so I am guessing they are not necessarily the same source.

It also seems they got their own /32, from RIPE?! Facebook Ireland LTD? What the….

We are a month away from world IPv6-day(bis), which was arguably a yawnfest last year. It was a huge success in the fact that it happened, and other than a small handful of people, nobody noticed 😛 Essentially, many heavy hitters on the Internet such as Google, Facebook, and a bunch of others, turned on their quad-As for 24 hours, as an “experiment”. The Internet didn’t blow up, everybody pretty much went on business as usual – and for some reason, that tiny fraction of people with broken IPv6 stacks was enough to have those AAAAs removed from DNS.

June 6th 2012 will be the second coming of “World IPv6 Day”, and this time, the theory is that AAAAs will be permanently added to DNS……ZMFG!

And as network operators, we are striving to dual-stack our networks end to end, so that all our users can benefit from having IPv6 connectivity on their networks before that day, right?! Well unfortunately not all of us have managed that 🙁 My company included. I guess I have nobody to blame but myself. We have some progress made, but not as much as I would have liked. Considering many companies still have zero deployment, by north-American standards I am doing ok 😉

However, this does not mean that nobody will be, or is currently using, some form of IPv6, be it over a Hurricane Electric tunnel broker, 6RD, or even 6to4. 6to4 is particularily relevant since Windows7 (Vista as well?), Windows 2003 and probably other OSs have 6to4 enabled interfaces on by default. I personally don’t like 6to4 very much, and I hope that it dies very, very soon; however if some folks are necessarily using it, and you haven’t gotten around to dual-stacking everyone, might as well make 6to4 slightly more useable, or at the very least, help it limp along. One of the inherent operational issues with 6to4 is that it depends on finding a relay server, in order to go from the IPv4 world to IPv6, and vice-versa. It would be a good idea to set up, as a network operator, your own 6to4 relay, in order to at least know where in the network your users are connecting, since 6to4 traffic will “find” the nearest one, and it can be anywhere; best it be on your network.

There is an RFC which deals specifically with 6to4 from the network operator’s perspective.

I won’t try and explain all of 6to4 here, however in essence, IPv6 prefix 2002::/16 is a reserved prefix that IPv4 hosts can derive their 6to4 address from. As an example, 192.168.1.1 would derive its 6to4 IP to be:

This becomes, in 6to4 – 2002:C0A8:101::/48 This unique prefix can then be routed back and forth from v4 to v6 and back over 6to4 relays. It is the 2002:V4ADDR::/48 prefix. A reserved anycast address, 192.88.99.1 (which microsoft seems to discover by querying 6to4.ipv6.microsoft.com), is used as a relay to the IPv6 network. (note: RFC 1918 addressing and 6to4 prefixes cannot exist or work on the Internet, I only use them here for example addresses….)

To illustrate the point, if I disable my 6to4 relay, I see that Hurricane Electric is nicely anycasting 192.88.99.1 and would be the nearest 6to4 relay I can use. A traceroute to ARIN over 6to4:

Neat! 25ms or so, a fraction of the round trip time the “nearest” 6to4 relay could offer me. Success!

Of course, it must be noted that as a network operator, you can only influence the relay used towards the IPv6 network – the nearest relay from the destination node back to the IPv4 network will be used. That is one of the drawbacks of 6to4 – although there can be relays peppered all over the Internet, the one used to get to IPv6 will in most likelyhood not be the one used to get back to IPv4. But the point is to improve 6to4, so hosting your own relay is favourable for your customers.

The setup is insanely and trivially easy. Here is an example Cisco config:

voila! In the above examples, substitute 2002:C0A8:101:: and 192.168.1.1 for the IP address of your router used for 6to4 and you are off to the races. Make sure you distribute 192.88.99.0/24 and 2002::/16 in your IGP.

IPv6, fragmentation and MTU

Unlike IPv4 where routers can (and do) fragment packets when a larger MTU packet must be forwarded over a path that does not support it. Assuming the packet doesn’t have the df-bit set, the IPv4 router will fragment a packet and forward two or more packets downstream for reassembly by the end host.

In the IPv6 world, routers or intermediate nodes do not fragment packets. A packet received by a router that is too large to forward will cause the router to drop the packet. This practice was likely designed into IPv6 for certain security vulnerabilities, where sending fragments could hide protocol level header information and cause reassembly surprises, DoS a host or router when trying to process these fragments or reassemble them, and a slew of various issues that made fragmentation become a IPv4 four-letter word.

In IPv6, although the packet is dropped, the router or node that drops the packet sends a ICMPv6 message “packet too big” back to the source host (Minimum IPv6 MTU has been defined as 1280). The source host then must perform the fragmentation and re-transmit the ensuing flow at a smaller MTU. The end node performs packet reassembly. This behaviour clearly indicates that arbitrarily dropping ICMP, which was a common practice by clueless IPv4 system administrators of yesteryear, is now a deprecated practice 😉

Ensuring ICMPv6 messaging delivery is particularly important in today’s reality, as a significant proportion of IPv6-capable networks are utilizing a tunneling mechanism of some sort, be it Hurricane Electric’s excellent tunnel broker service, 6RD, or static IPv6-over-IPv4 tunnels; a 1500 MTU cannot be assumed. The use of path mtu discovery (PMTUD), is critical.

The use of tracepath6, becomes particularily interesting – my distro (RedHat et al) seems to include it in the iputils package:

Here is an example of non-tunnelled, native dual-stack tracepath from a node to www.ripe.net:

Apologies for the ::ffff:154.0.0.xx IPv4 mapped addresses, AS174 (Cogent…) seems to return them in traceroutes…Yuck! And the asymm warning, which normally can indicate an asymmetrical path, is apparently caused by my local network’s use of HSRP. It can also appear in certain tracepaths when an intermediate router is a Juniper, which I am told reduces the TTL by one before transmitting ICMP, and tracepath(6) uses TTL to determine path symmetry. I don’t have a Juniper to test this <:-)

Anyway, here is an example of a tracepath through a IPv6-over-IPv4 tunnel:

tracepath6 is a useful tool, especially in today’s Internet where IPv6 connectivity is not necessarily the usual 1500-byte MTU standard of the past and many hosts use tunneling transition mechanisms to access IPv6 content. It will be useful to identify broken ICMPv6 paths, which for v6 traffic is essential for reliable communication. Until dual-stack or even IPv6-only nodes are the norm, PMTUD and ICMPv6 will be critically important. Here is hoping sysadmins will be wise enough to follow best practices and allow ICMPv6 messages, especially “packet too big” and “destination unreachable” messages.