An interview with Owen DeLong, IPv6 Evangelist, Hurricane Electric

This is an email interview with Owen DeLong , the “IPv6 Evangelist” for Hurricane Electric, one of the largest ISPs in the world. They are also operators of tunnelbroker.net, one of the largest IPv6 tunnel providers in the world.

Owen has been in the industry in many roles, including work at Sun, Exodus Communications and TellMe. In addition to his “day job”, he’s also on the advisory council for ARIN, teaches SCUBA, and makes lots of contributions to network-related mailing lists. He’s also one of the very few people I know who has an AS number for his home net!

We’ve been wanting to do this interview for quite a while, but things have been rather busy the past year. When I ran into Owen (again) at LISA in San Diego a few weeks back, we talked about IPv6 and this interview. Owen was kind enough to work on this with me over the holiday.

Owen, you’ve been an “IPv6 Evanagelist” for Hurricane Electric for a while now. How did you become the IPv6 Evangelist? What does that entail, and what did you do before IPv6 needed an evangelist?

I’ve been various forms of Systems Administrator, Network Engineer, Backbone Engineer, Senior Backbone Engineer, Network Architect and even Operations Manager over the years. When my last startup was on the verge of imploding, I mentioned to someone at HE who does similar work that I was interested in doing something similar. We talked extensively in Manila (APNIC conference) and the rest as they say, is history.

Can you tell me a little about Hurricane Electric?

We are actually one of the largest backbone network providers in the world. We are the number 1 most peered IPv6 network and the number 3 most peered IPv4 network according to CAIDA. We operate multiple 10-Gig backbones throughout North America, Europe, and Asia.

We’ve run into each other at several conferences such as Game Developer and LISA. What other conferences do you attend, and what’s the feedback from people about IPv6?

Most of the audiences I speak to are pretty receptive. Certainly there is more and more interest and less and less skepticism over the last 3 years.

IPv6 has been a standard now for almost 18 years (if you use RFC 1883 as the reference). Why has adoption been so slow?

Because we made the mistake of introducing NAT and people are busy. A network not perceived as broken tends to limp along with whatever level of brokenness it has come to accept until perturbed by an additional factor such as CGN, address exhaustion, or other external pressures like the continued ability to reach destinations in Asia.

So far, IPv4 runout hasn’t really exerted those pressures to any great extent on anyone yet, but at least people are starting to see that it is coming and IPv6 adoption is starting to accelerate quite a bit now.

According to Google, IPv6 was around 0.25% of internet traffic at the beginning of 2010 and 2011. The biggest difference between the two being that at the beginning of 2010, it was about 60% 6to4/Teredo and at the beginning of 2011 it was about 80% native, but the overall growth was pretty flat. By the end of 2011, it was 0.4% and almost entirely native with 6to4 and Teredo almost unmeasurable. As we approach the end of 2012, we’re seeing more than 1.1% with 6to4 and Teredo almost eliminated.

At first blush, those seem like pretty low numbers. However, 1% of internet traffic today is a lot more than 100% of internet traffic when IPv4 was 18 years old. Also, consider in 2011, that’s a 100% growth rate and in 2012, it’s approximately 175% growth, so not only is adoption growing, but the rate of adoption is accelerating quite rapidly.

If the curve continues to ramp towards vertical, we might see more than 5% by the end of next year and more than 20% by the end of 2014. That’s without factoring in the additional pressures from actually running out of IPv4 addresses at the service providers and the fact that ARIN will be basically out of address space most likely in the middle of next year.

On the technical side, what’s the biggest impediment to implementation? Is that the same for ISPs, Enterprises and end users, or do they all face different challenges?

No, they each face unique challenges.

For the end user, it’s the lack of IPv6 support in consumer products. There’s a host of products that lack IPv6 support today which is severely impacting consumers. These include familiar household names like Playstation, TiVO, Yamaha, Samsung, LG, and all of the Matsushita brands (JVC, Technics, Pioneer) and more. In fact, I don’t know of a single CE manufacturer that has embraced IPv6 in their products yet. IMHO, this is inexcusable.

None of these vendors have yet seen fit to deploy IPv6 at the consumer level in their products and this is becoming a major impediment.

To look at ISPs, we really need to divide them into two principal categories… The B2B and/or Backbone-oriented ISP, such as Hurricane Electric, where there really are very few remaining technical impediments and the few remaining impediments are primarily related to educating executive management.

The other category is what I will call the last-mile ISP. The providers that serve SOHO, Residential, and Small Business customers and perhaps some medium sized businesses as well. These ISPs are still facing shortcomings in CPE and in their provisioning, management, and deployment systems. Many DSLAMs and BRAS units have limited or no IPv6 capability. Several of the CMTS vendors still have show-stopping bugs to work out (This means you, brand C and others).

For the enterprise, there are a few technical impediments, but these can mostly be addressed by current technologies. The larger hurdles in business really are the education of executive management and communicating to the enterprise that this isn’t just a networking issue.

How about on the Business side?

I think this is easier to answer. Across the board for all of those organizations, the number one business challenge is educating executive management to the point that they understand this to be a critical organization-wide issue that must be addressed starting now (if they haven’t started already).

Any organization considering IPv6 seriously needs to understand that this is an organization-wide issue and that it touches EVERY group within the organization. Systems administrators need to be involved because this will touch every application, every system, every server, and will impact how they do their jobs on a daily basis. The help desk needs to get involved because this will represent some pretty major changes in their troubleshooting scripts and will also require them to understand more about how things operate in a dual-stack environment. Application developers, maintainers, administrators, etc. are involved because any application that communicates via the network needs to be examined and tested against a dual-stack and eventually an IPv6 only environment. The mail room has to expect that there’s going to be a lot of logistical support required for getting all of these equipment updates to the right locations at the right times. Management at all levels needs to recognize the need to make this an organization-wide priority and address it in an orderly fashion before it becomes an outright emergency.

Lots of networks seem to be taking a dual-stack approach and adding IPv6 to their existing IPv4 network. What do you think of this approach?

I think it’s the only feasible approach for the time being. We’ll reach a point where we have to turn off IPv4 and move forward to IPv6 single-stack and that’s a good thing. It should always be the end goal. However, just as we didn’t turn off Novell the day we turned on IP in that great migration of the enterprise, we’re not going to be able to turn off IPv4 and turn on IPv6 overnight. We’re going to have to live with both for some time to come.

For people that want to learn more about IPv6, or even implement it on their home networks, where should they start?

I highly recommend our on-line training available at http://tunnelbroker.net. It’s pretty easy to get through, but it does require you to not only learn about IPv6, but to demonstrate that you can actually make things work with IPv6 in order to obtain the higher certification levels.

You’ve mentioned in the past that there are new IPv6-only customers coming online in areas where the IPv4 space is already depleted. How many people are we talking about and where are they? How many potential customers could a company “miss” by delaying an IPv6 roll out?

Well, there are currently a little more than 2.5 billion people on the internet. There are more than 6.8 billion people on earth. So, over time, I expect we’ll see most of the remaining 4.3+ billion people (and growing) connecting on IPv6 and unable to get IPv4 addresses. Immediately, it’s relatively small numbers, probably a few thousand her and there. However, that number can only continue to grow as time progresses.

One of the biggest complaints we all hear is that there’s no “killer app” and “no ROI” for implementing IPv6. What risks (if any) are companies taking by delaying IPv6 adoption?

If it isn’t clear from what I’ve said above, I’ll put it this way. The IPv4 internet cannot keep growing much larger than it currently is. The IPv6 internet will, on the other hand, continue to grow and for a while, that growth will continue to accelerate. As a result, it won’t be very many years before the fraction of the internet that is reachable via IPv4 is much smaller than that portion which is reachable via IPv6. If your business depends on being connected to the whole internet, then, that’s the killer app. for IPv6. If you’re willing to live with only being connected to the part of the internet that exists today, then you may have a few years before you have to worry about that shrinking, but, I suspect it’s not as many years as a lot of people will claim.

You speak as if IPv6 is urgent even though we haven’t actually run out of IPv4 addresses yet. Why is that?

The deployment of IPv6 is no small undertaking. In fact, the effort required in most organizations, especially enterprises will be on par with the Y2K process. However, because unlike Y2K, IPv4 does not come with a certain deadline, it has been much harder to get management to focus proper attention on the issue.

Just like Y2K, if we had waited until Dec. 30, 1999 to begin our preparations, we would have had a number of serious problems. Instead, most organizations had multi-stakeholder task forces assigned to Y2K more than 5 years in advance. Depending on how you measure IPv4 runout, we’re somewhere between already there and less than 2 years away, so it’s already too late to start 5 years in advance. The good news is that a lot of the necessary vendor work has already been in progress. The bad news is that there’s still a whole lot of organizational work and vendor work that hasn’t yet begun.

At the very least, it’s quite urgent for each organization to go through an IPv6 gap analysis (an analysis of what portions of the infrastructure are not yet ready for IPv6 to be deployed and what is required to correct those limitations) as soon as possible. Until that process is completed, an organization has no idea even how long it will take to prepare for IPv6 or what it will cost. This close to runout, the lack of that information is a major risk to any organization which depends on the internet for its operations.

Owen, thanks for all the great IPv6 info! We’ll have to stay in touch, especially this year, as we see more IPv6 rollouts and everyone gets more operational experience.

That wraps up the interview with Owen. Stay up to date with IPv6 news and deployment strategies at:

You write that there is “no killer app” and “no ROI” and “we haven’t actually run out of IPv4”. But that is no longer the case. We are starting to see severe issues cropping up due to IPv4 exhaustion. For example Unitymedia and (to a lesser degree) Kabeldeutschland, two of the largest and fastest growing ISPs in Germany are moving to DS-lite as they have run out of IPv4 addresses. All new customers and some of the existing customers are no longer provided with native IPv4. Allthough IPv4 only webpages continue to be available through DS-lite tunnels, any inbound connections are no longer possible to these customers with IPv4. Amongst many other issues that this has caused, various forums are full of complaints that Unitymedia customers can no longer play Xbox live games as Microsoft does not support IPv6 and the needed IPv4 peer to peer connections aren’t available any more due to a lack of IPv4 and port forwarding.

I don’t know what the situation is like in Asia, but given that APNIC ran out of IPv4 in 2011 I would imagine it is even worse than in Europe. So companies like Microsoft with Xbox live are starting to loose significant numbers of customers who can no longer reach their IPv4 only services.

Those are all great and valid issues. No doubt and I agree completely that they emphasize the importance of companies going through the transition.

However, neither of those is a “killer app”, or “ROI”: they are not new capabilities and do not (yet) contribute to the bottom line. This has been the problem for the past 15 years. There were no new capabilities, and no new customers on IPv6-land.

But, as Owen (and you) point out, there *will soon* be a very real ROI when there is a growing “IPv6 only” customer base out there. The argument then will be that companies are ignoring an important and growing market, while their existing market will shrink (as current v4 customers transistion to v6). In other words they will need to transition in order to not only get all those new never-were-IPv4 customers, but also to *keep* their current customers, after the customers transition.