Hampering Eyeballs - Observations on Two "Happy Eyeballs" ImplementationsIn this article we look at two "Happy Eyeballs" implementations, that aim to reduce degraded user experience as the result of broken dual-stack configurations. We call this degraded user experience "Unhappy Eyeballs". The Chrome web browser implementation seems to succeed in this aim, while Apple's Mac OS X Lion operating system only partially succeeds in avoiding "Unhappy Eyeballs". Furthermore we show that Mac OS X Lion, in combination with the Safari web browser results in "Happy Eyeballs", but leaves well performing native IPv6 capacity unused in roughly half of the cases we measured, a condition we name "Hampering Eyeballs".https://labs.ripe.net/Members/emileaben/hampered-eyeballshttps://labs.ripe.net/logo.png

In this article we look at two "Happy Eyeballs" implementations, that aim to reduce degraded user experience as the result of broken dual-stack configurations. We call this degraded user experience "Unhappy Eyeballs". The Chrome web browser implementation seems to succeed in this aim, while Apple's Mac OS X Lion operating system only partially succeeds in avoiding "Unhappy Eyeballs". Furthermore we show that Mac OS X Lion, in combination with the Safari web browser results in "Happy Eyeballs", but leaves well performing native IPv6 capacity unused in roughly half of the cases we measured, a condition we name "Hampering Eyeballs".

IPv6 deployment is hampered due to end users that have broken IPv6 connectivity. Luckily this only affects an ever decreasing fraction of end users, but the fact that small amounts of broken IPv6 connectivity exists makes some people not deploy IPv6. When a user who is dual-stacked connects to a dual-stack website, his or her system (operating system plus application) will traditionally try to connect over IPv6 first (as per
RFC 3484
). When IPv6 doesn't work, this connection has to time out, before IPv4 will be attempted. Since timing out happens in the order of tens of seconds, the resulting user experience is horrible: Users end up staring at a blank screen or spinning wheels and will likely give up before the timeout occurs.

The selection of what IP address to connect to (IPv6 or IPv4) is typically done by picking the first result from the
getaddrinfo()
system call (ie. something that is baked into an operating system), and when that first result doesn't respond the next result from
getaddrinfo()
is tried. RFC 3484 defines in what order types of addresses should be tried. Roughly, RFC 3484 dictates that native IPv6 be tried first, then IPv4, and then IPv6-through-transition-technologies.

The "Happy Eyeballs" concept aims to address the situation of bad performance when a user encounters broken IPv6. "Happy Eyeballs" is a description of implementations of operating systems and/or applications that take measures so users ("eyeballs") won't see noticeable performance degradation when encountering broken connectivity in one protocol but not the other. Analogous one could call the users that do encounter the slow fall-back behaviour to broken dual-stack hosts using classic RFC 3484 "Unhappy Eyeballs". The term "Happy Eyeballs" itself was first coined in an
IETF Internet Draft
describing a specific implementation, which has evolved to describe the general principles of the mechanisms involved. See below a quote from the "Happy Eyeballs" draft:

A Happy Eyeballs algorithm has two primary goals:
1. Provides fast connection for users, by quickly attempting to connect using IPv6 and (if that connection attempt is not quickly successful) to connect using IPv4.
2. Avoids thrashing the network, by not (always) making simultaneous connection attempts on both IPv6 and IPv4.

There have been at least two different flavours of "Happy Eyeballs" deployed in commonly used operating systems and web browsers in the last few months. Below I will detail some of the behaviour I've seen from these two different flavours.

Chrome

The first "Happy Eyeballs" implementation that I'm aware of that made it into production software is included in the Chrome webbrowser since version 11.0.696.71 which was released on 24 May 2011. In
Chrome's issue tracker
the way
getaddrinfo()
-results are handled is described as follows:

When a hostname has both IPv6 and IPv4 addresses, and the IPv6 address is listed first, we start a timer (300ms) [...]. If the timer fires, that means the IPv6 connect() hasn't completed yet, and we start a second socket connect() where we give it the same AddressList, except we move all IPv6 addresses that are in front of the first IPv4 address to the end. That way, we will use the first IPv4 address. We will race these two connect()s and pass the first one to complete to ConnectJob::set_socket()

In real life this works pretty well. I've put a web server on an intentionally broken dual-stack hostname (
http://intentionally.broken.dualstack.wdm.sg.ripe.net/
, which redirects to www.ripe.net when accessed on IPv4 and has an IPv6 address that leads nowhere). Anytime I visited that intentionally broken dual-stack host using a recent version of Chrome, the fallback from the intentionally broken IPv6 to IPv4 was not noticeable to me.

The Chrome approach is simple but effective: If your IPv6 isn't fast enough (or not working at all), it's not going to be used. Period.

Mac OS X Lion

The second "Happy Eyeballs" implementation is included in the Mac OS X Lion operating system, released on 20 July 2011. In a
post by an Apple engineer
on the Apple IPv6 mailinglist, it is described in two parts. The first part is about
getaddrinfo()
:

Results from getaddrinfo are now sorted using routing statistics (destination
with the lowest min round trip time wins). If the statistics can not determine
which destination is better, an implementation of RFC3484 is used. The default
RFC3484 policy is read only.

The second part is about two Apple frameworks:

CF and NS layer frameworks that use CFSocketStream do not use getaddrinfo.
Those APIs use something similar to happy eyeballs. The A and AAAA queries are
started at the same time but the responses are handled as they are received.
When an answer is received, it is sorted in to a list of destination addresses.
If there are no more addresses coming in (this was the last answer in the DNS
packet or mDNSResponder has no more answers in the cache), a connection is
started to the first destination on the sorted list. The DNS resolve operation
is left running and more answers are processed as they arrive. A timer is setup
for a period of time in which we would expect the connection to complete, based
on the routing statistics. If the timer fires before the connection is
established, a connection to the next best address will be started while the
existing connection continues to try and make progress. A similar timer is
setup and the process repeats until a connection is established or we run out
of addresses to try. The code keeps track of whether or not it has received
both A and AAAA response (whether the answer was a list of addresses or no
address). If the connection is established before both A and AAAA responses
come back, the resolve is kept open for up to a second to allow mDNSResponder
to receive a slow response and store it in the cache. This way, subsequent
connections to the same host in a short period of time will have all answers in
the cache.

I also tested this to my intentionally broken dual-stack hostname, and the results are, well, complicated, so bear with me.

On Lion with Firefox 7 one has to wait for over 20 seconds before failover between intentionally broken IPv6 to IPv4 happens. My guess is that initially there are no routing statistics, so RFC 3484 is used and my Firefox waits 20 seconds in vain for a non-working IPv6, before falling back to IPv4. So Mac OS X Lion doesn't solve the broken dual-stack problem in this case. Firefox 7 does have a "Happy Eyeballs" mechanism, but it is disabled by default (which is what I tested). This default has not changed in Firefox 8, which I haven't extensively tested yet.

On Lion, Chrome's "Happy Eyeballs" saves the day: the 300ms fallback-mechanism as described in the previous paragraph about Chrome just works. If the Firefox 7 "Happy Eyeballs" mechanism is activated, it works as well as Chrome. This is done by setting the value of
network.http.fast-fallback-to-IPv4
in
about:config
to
true.

But Lion with Safari is further different. I'm guessing because it doesn't use
getaddrinfo()
for this, but one of the two frameworks described in the post above. A tcpdump reveals that indeed both connections, one IPv6, the other IPv4, are tried at the same time. And since the IPv6 in my test was intentionally broken, the IPv4 connection wins.

When visiting the broken dual-stack website first with Safari, and then with Firefox 7 things are different again. In this case Firefox 7 doesn't have the noticeable performance degradation, like it has without first using Safari. (Note: You may have to try a couple of times in Safari and wait a couple of minutes for the operating system to detect that IPv6 is not working before the Firefox 7 performance degradation due to slow fail-over goes away). This may seem weird but makes sense if the routing statistics of both IPv4 and IPv6 connection attempts by Safari have been cached, and
getaddrinfo()
will return a list with the fastest protocol first. Because IPv6 failed, in this case this will be IPv4.

Are you confused yet? It looks like OS X Lion makes debugging connectivity issues to dual-stacked websites a royal pain, with the associated cost for support staff. While the Mac OS X Lion + Safari combination looks to be a true "Happy Eyeballs" implementation, the combination of OS X Lion with anything that just uses
getaddrinfo()
does not make the eyeballs much happier it seems.

Debugging Dual-Stack On Mac OS X Lion

I'm not aware of any tools to see what, and in what order,
getaddrinfo()
in Mac OS X Lion actually returns IP addresses. This matters because this can be different on every host due to differences in performance statistics and how much of this is cached. The post on the Apple IPv6 mailing list mentions
nettop -n -m route
to dump a live view of the routing statistics. However, this doesn't help you to match up the IP addresses that are associated with a hostname and find out which one was seen as faster easily.

Luckily it's easy to code up a tool for this or if you're lazy like me you can even find
existing code online
that will do the job. This will help in debugging issues with Mac OS X Lion. If people are aware of other useful tools here (bonus points if they are pre-installed on Mac OS X Lion), please comment below.

Windows 7

Windows 7 doesn't have a "Happy Eyeballs" implementation that I'm aware of. I tested both Firefox 8 and the latest Internet Explorer 9 beta against the intentionally broken dual-stack website, and in both cases encountered a long timeout before fall-back to IPv4.

The Effect of "Happy Eyeballs" on IPv6 deployment Metrics

The RIPE NCC runs a measurement infrastructure to monitor deployment of IPv6 at end-users, as we
described on RIPE Labs earlier
. Since we have over three months of data for both "Happy Eyeballs" implementations described above, we can see what the effects are on IPv6 traffic levels, and IPv6 deployment statistics that are coming out of infrastructures like ours (for instance
Google
's, or the one that
APNIC is running
and the RIPE NCC is actively supporting).

With our measurement setup we can measure the effect of both "Happy Eyeballs" implementations on communication between dual-stacked hosts. In the the old non-"Happy Eyeballs"-world two native dual-stacked hosts would always try to communicate over IPv6 first. In the new, not necessarily better, world, this is not necessarily the case.

In our measurements, hosts will try to fetch objects that are available in IPv4-only, IPv6-only and objects that are available over both (ie. dual-stacked). I looked at the hosts that are able to fetch an IPv6-only object using unicast IPv6 (determined by having a non-Teredo, non-6to4 IPv6 source address). For these native-IPv6 capable hosts I looked at what percentage would fetch the dual-stacked object over IPv4. This is the tell-tale sign of "Happy Eyeballs" or other deviations from RFC 3484 at work, determining that IPv4 should be used instead of IPv6 for connections from that host to our measurement server. Let's call this behaviour "Hampering Eyeballs", since these hosts are using the legacy IPv4 protocol, while they have perfectly fine native IPv6 available, which is a pretty good approximation for this dictionary entry I found:

hamper:
hinder or impede the movement or progress of

2011-08

2011-09

2011-10

Mac OS X Snow Leopard
(1265)

0.73%

0.71%

0.55%

Mac OS X Lion
(1628)

43%

43%

43%

Other OS
(12154)

5.0%

4.1%

9.1%

Table 1: Percentage of "Hampered Eyeballs" for Mac OS X Snow Leopard, Mac OS X Lion, and all other operating systems. The numbers in brackets are the minimum number of observations per month in this category.

As you can see in Table 1, the latest two versions of Mac OS X show quite some different behaviour. Mac OS X Snow Leopard shows a very good track record, while almost half of the clients using Mac OS X Lion are "Hampering Eyeballs", and the fact that they have perfectly fine working native IPv6 connectivity is masked. Remember that we excluded hosts with non-working native IPv6 connectivity from these numbers.

2011-08

2011-09

2011-10

Chrome "Happy Eyeballs" version >= 11.0.696.71
(3044)

8.1%

7.7%

7.5%

Chrome version < 11.0.696.71
(450)

5.1%

7.0%

7.1%

Other browsers
(11720)

8.7%

8.6%

14%

Table 2: Percentage of "Hampered Eyeballs" for pre- and post-Happy Eyeballs implementation in Chrome, as compared to other browsers.
The numbers in brackets are the minimum number of observations per month in this category.

As you can see in Table 2, the "Happy Eyeballs" versions of Chrome hardly have a noticeable effect on what percentage of hosts was seen with the "Hampering Eyeballs" behaviour, as compared to older versions of Chrome, or other browsers. So implementing "Happy Eyeballs" the Chrome-way doesn't cause significant IPv6 capacity to be masked. At least not to hosts that have comparable IPv4 and IPv6 performance, like our measurement host. Since in Table 2 there is no distinction between operating systems, it is quite likely that OS X Snow Leopard causes some (most?) of the "Hampering Eyeballs" observed in all three categories.

2011-08

2011-09

2011-10

Mac OS X Lion + Safari
(1284)

48%

48%

48%

Mac OS X Lion + other browser
(344)

20%

26%

29%

Other combinations
(13419)

4.5%

3.7%

8.3%

Table 3: Percentage of "Hampered Eyeballs" for Lion + Safari, as compared to Lion + other browsers, or other combinations of operating system and browser.
The numbers in brackets are the minimum number of observations per month in this category.

In Table 3, the combination of OS X Lion + Safari is compared to OS X Lion with other browsers and you can see that there is a difference. When using Lion + Safari, roughly half of the hosts measured, show the prefer-IPv4-to-reach-dualstack behaviour. What I expected to see is that Lion + other browsers would be as low as the category "Other combinations". Remember that
getaddrinfo()
on Mac OS X still uses RFC 3484 when no performance statistics about the destination IP address are known. This can be caused by hosts having performance statistics in the cache for the measurement host that we use to perform the IPv4, IPv6 and dual-stack tests, or there are other browsers that don't use
getaddrinfo()
anymore.

Conclusion

If the way Mac OS X implemented "Happy Eyeballs"-like functionality becomes more widespread (please don't do this Microsoft!) it will cause perfectly fine dual-stack hosts with roughly equal performance over both protocols, to select the legacy IPv4 protocol in roughly half of the cases. This makes the IPv6 capacity in networks under-used and that is bad for a few reasons:

There will be less incentive to improve the IPv6 network further (performance, more interconnects)

It will put more strain on the IPv4 network, including middleboxes that are put in this network

IPv6 deployment statistics will under-report IPv6 capacity, which will make informed decisions based on these harder

Prolonging the transition to IPv6 unnecessarily long (ie. causing "Hampering Eyeballs") is not going to help the Internet.

The Chrome implementation of "Happy Eyeballs" on the other hand seems to adequately avoid a degraded user experience ("Unhappy Eyeballs") when visiting broken dual-stacked hosts, and at the same time doesn't cause "Hampering Eyeballs" in significant numbers.

If people have other observations on "Happy Eyeballs", please comment below.

Note: After a few comments from readers I decided to change the technically incorrect "Hampered Eyeballs" (as the users are not hampered in viewing web pages), into "Hampering Eyeballs" (as the users are, inadvertently, hampering the transition to IPv6).

18 Comments

Tore Anderson
says:

13 Nov, 2011 12:26 PM

I immediately get an ICMPv6 Hop Limit Exceeded from 2001:67c:2048:e:102:4144:0:b when I attempt to access the intentionally broken hostname. This makes failover to IPv4 happen without any discernable delay, no Happy Eyeballs needed (at least not on Linux). In order to truly simulate brokenness and test HE, you'll probably need to instead use a destination that's completely blackholed over IPv6. One example of this is http://broken.redpill-linpro.com/.

Nit: s/classif/classic/

Tore

Emile Aben
says:

13 Nov, 2011 04:41 PM

I've fixed this. The situation you encountered was due to the fact that after I did all my tests with a broken dual-stack using an IPv6 address in the documentation prefix, I changed the broken dual-stack IPv6 address to an address that was in a RIS routing beacon prefix. I did this because I was worried that somebody could announce the documentation prefix and receive traffic for it due to this article. As you pointed out this didn't cause the broken dual-stack hostname to be truly broken all of the time. Now it should be truly broken again.

I asked a Firefox developer, and he said that their version of Happy Eyeballs is fixed up and turned on by default in Firefox 10, now in alpha/Aurora and due to be released in 11 weeks.

John Mann
says:

14 Nov, 2011 11:29 PM

Nit: s/gethostbyaddr/getaddrinfo/

Josh
says:

15 Nov, 2011 09:38 PM

Hampered Eyeballs? Can your eyeball tell the difference between a web page loaded over IPv6 and one loaded over IPv4? If a page loads faster over IPv4, what is the advantage of using IPv6?

Emile Aben
says:

16 Nov, 2011 11:04 AM

Hi Josh,

They are not hampered in viewing a web page, but they are hindering/impeding the transition to IPv6. They also could have used IPv6 and not notice the difference. Maybe "Hampering Eyeballs" is a better term, I'll change it in the article and acknowledge it was changed.

Hi Jan,Interesting. Proves the point that this whole thing with using performance statistics is tricky to debug (even for mr. IPv6-Slovenia).

Andrew Yourtchenko
says:

01 Mar, 2012 09:04 PM

Jan, if I were to have two routes to go with my car (my eyeballs) to get to the destination (viewing it), why exactly should I choose the slower one ? What makes the pings on IPv6 slower than on IPv4 ? Would be worth checking whether there is maybe something that needs to be fixed. Alternatively, just put a netem in front of IPv4 leg and bump up the delay to 14ms so IPv6 becomes better :-) (*ducks*)

Alex
says:

17 Jun, 2013 09:38 AM

Mountain Lion still has this stupid behavior, I hope "Mavericks" fixes it (although I doubt it will)

I could accept the behavior if the v6 hosts are much slower than v4 hosts (by like 100ms or so), but in my case some hosts are so close to me (hop wise) that the timing difference between them is <5ms in cases, so subsequent calls to getaddrinfo() basically have the IP addresses jumping around randomly due to minute fluctuations.

Any application that wants consistent behavior would have to implement RFC 3484 themselves, and completely ignore the order in which OS X returns addresses in.

Peter Green
says:

21 Aug, 2013 06:25 PM

Happy eyeballs cannot be implemented properly at the OS level for applications that use the traditional APIs because the traditional APIs work by the application first looking up an address and then making a connection.

Properly implemented happy eyeballs requires making multiple connection attempts at once to allow speedy connections when one protocol is broken while allowing a connection to be made eventually when all protocols are slow.The application has to be involved in the move to happy eyeballs either by implementing it entirely in the application (as chrome does) or by using a different interface from app to OS (as safari seems to be doing).

The fact apple seems to have ignored the "you should still preffer ipv6" part of the "spec" is also concerning.

Paul Marks
says:

27 Oct, 2013 09:15 PM

When I browse dual-stack sites on 10.9 Mavericks with Chrome+IPvFoo, I'm no longer seeing unexpected IPv4 addresses. So it looks like Apple may have fixed this.

osxusers
says:

12 Nov, 2013 10:43 PM

I'm also on Osx maverick, but unlike Paul, I still experience the same behaviour as in mountain lion, so I guess this issue is still not fixed yet. I don't think my tunnel broker (he.net) is to blame, since on windows 8.1, ipv6 is prefered as per happy eyeballs policy

Dylan Neild
says:

24 Jan, 2014 07:16 AM

Paul Marks: Apple hasn't fixed this; it's Chrome that is working properly. Try Safari and you'll see the problem still exists.

Henning Søilen
says:

28 Apr, 2014 11:07 AM

And the same flaw is present with a dual stacked iphone running IOS 7.1.1

The RIPE NCC uses cookies. Some of these cookies may have been set already. More information about our cookies can be found in our privacy policy. You can accept our cookies either by clicking here or by continuing to use the site.