Network Working Group C. Donley
Internet-Draft CableLabs
Intended status: Informational L. Howard
Expires: November 15, 2012 Time Warner Cable
V. Kuarsingh
Rogers Communications
J. Berg
CableLabs
J. Doshi
University of Colorado
May 14, 2012
Assessing the Impact of Carrier-Grade NAT on Network Applicationsdraft-donley-nat444-impacts-04
Abstract
NAT444 is an IPv4 extension technology being considered by Service
Providers to continue offering IPv4 service to customers while
transitioning to IPv6. This technology adds an extra Carrier-Grade
NAT ("CGN") in the Service Provider network, often resulting in two
NATs. CableLabs, Time Warner Cable, and Rogers Communications
independently tested the impacts of NAT444 on many popular Internet
services using a variety of test scenarios, network topologies, and
vendor equipment. This document identifies areas where adding a
second layer of NAT disrupts the communication channel for common
Internet applications. This document was updated to also include
Dual-Stack Lite impacts.
Status of this Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on November 15, 2012.
Copyright Notice
Donley, et al. Expires November 15, 2012 [Page 1]

Internet-Draft NAT444 impacts May 20121. Introduction
IANA and APNIC exhausted their IPv4 address space in 2011. Current
projections suggest that RIPE and ARIN may exhaust their free pools
of IPv4 addresses in 2012. IPv6 is the solution to the IPv4
depletion problem; however, the transition to IPv6 will not be
completed prior to IPv4 exhaustion. NAT444 [I-D.shirasaki-nat444]
and Dual-Stack Lite ([RFC6333]) are transition mechanisms that will
allow Service Providers to multiplex customers behind a single IPv4
address, which will allow many legacy devices and applications some
IPv4 connectivity. While both NAT444 and Dual-Stack Lite do provide
basic IPv4 connectivity, they impact a number of advanced
applications. This document describes suboptimal behaviors of NAT444
and DS-Lite in our test environments.
In July-August 2010, CableLabs, Time Warner Cable, and Rogers
Communications tested the impact of NAT444 on common applications
using Carrier Grade NAT (CGN) devices. This testing was focused on a
wide array of real time usage scenarios designed to evaluate the user
experience over the public Internet using NAT444, in both single ISP
and dual ISP environments. The purpose of this testing was to
identify applications where the technology either breaks or
significantly impacts the user experience. The outcome of the
testing revealed that applications such as video streaming, video
gaming and peer-to-peer file sharing are impacted by NAT444.
From June - October 2011, CableLabs conducted additional testing of
CGN technologies, including both NAT444 and Dual-Stack Lite. The
testing focused on working with several vendors including A10,
Alcatel-Lucent, and Juniper to optimize the performance of those
applications that experienced negative impacts during earlier CGN
testing and to expand the testing to DS-Lite.
Applications that were tested included but were not necessarily
limited to the following:
1. Video/Audio streaming, e.g. Silverlight-based applications,
Netflix, YouTube, Pandora 2.
2. Peer-to-peer applications, e.g. video gaming, uTorrent
3. On line gaming, e.g. Xbox
4. Large file transfers using File Transfer Protocol (FTP)
5. Session Initiation Protocol (SIP) calls via X-Lite, Skype
Donley, et al. Expires November 15, 2012 [Page 4]

Internet-Draft NAT444 impacts May 20122.1.4. Case4: Two Clients, Two Home Networks, Two Service Providers Cross ISP
^^^^^^^^ ^^^^^^^^
( ISP A ) ( ISP B )
Vvvvvvvv vvvvvvvv
| |
+---------------+ +---------------+
| LSN | | LSN |
+---------------+ +---------------+
| |
+---------------+ +---------------+
| CMTS | | CMTS |
+---------------+ +---------------+
| |
+---------------+ +---------------+
| CM | | CM |
+---------------+ +---------------+
| |
+-------------------------+ +-------------------------+
| Home Router | | Home Router |
+-------------------------+ +-------------------------+
| |
+---------------+ +---------------+
| Client | | Client |
+---------------+ +---------------+
This test case is similar to Case 1 but with the addition of another
identical ISP. This topology allows us to test traffic between two
residential customers connected across the Internet. We focused on
client-to-client applications such as IM and peer-to-peer.
2.2. General Test Environment
The lab environment was intended to emulate multiple service provider
networks with a CGN deployed, and with connectivity to the public
IPv4 or IPv6 internet (as dictated by the co-existence technology
under test). This was accomplished by configuring a CGN behind
multiple CMTSes and setting up multiple home networks for each ISP.
Testing involved sending traffic to and from the public internet in
both single and dual ISP environments, using both single and multiple
home networks. The following equipment was used for testing:
o CGN
o CMTS
Donley, et al. Expires November 15, 2012 [Page 8]

Internet-Draft NAT444 impacts May 2012
11. Internet Archive - Video and Audio streaming; large file
downloads
12. Video streaming using iClips
13. SIP Calls - X-Lite, Skype, PJSIP
14. MS Smooth Streaming (Silverlight)
15. Video chat - Skype, OoVoo
The following CPE devices were used for testing these applications on
one or more home networks:
1. Windows 7, XP and Vista based laptops
2. MAC OS X laptop
3. iPad
4. Xbox gaming consoles
5. iPhone and Android smartphones
6. LG Blu-Ray player (test applications such as Netflix, Vudu, etc.)
7. Home routers - Netgear, Linksys, D-Link, Cisco, Apple
2.3. Test Metrics
Metrics data that were collected during the course of testing were
related to throughput, latency, and jitter. These metrics were
evaluated under three conditions:
1. Initial finding on the CGN configuration used for testing
2. Retest of the same test scenario with the CGN removed from the
network
3. Retest with a new configuration (optimized) on the CGN (when
possible)
In our testing, we found no significant differences with respect to
latency or jitter when the CGN was in the network versus when it was
not present in the network. It should be noted that we did not
conduct any performance testing and metrics gathered were limited to
single session scenarios. Also, bandwidth was not restricted on the
DOCSIS network. Simulated homes shared a single DOCSIS upstream and
Donley, et al. Expires November 15, 2012 [Page 10]

Internet-Draft NAT444 impacts May 2012
downstream channel.
Note: Performance testing as defined by CableLabs includes load
testing, induction of impairments on the network, etc. This type of
testing was out of scope for CGN testing.
2.4. Test Scenarios Executed
The following test scenarios were executed using the aforementioned
applications and test equipment:
1. Single ISP, Single Home Network with Single User
2. Single ISP, Two Home Networks With One User on Each Network
3. Dual ISPs, Single Home Network with Single User on each ISP
4. Dual ISPs, One Home Network With One User ISP-A; Two Home
Networks with one user on each for ISP-B
These test scenarios were executed for both NAT444 and DS-Lite
technologies.
2.5. General Test Methodologies
The CGN was configured for optimal setting for the specific test
being executed for NAT444 or DS-Lite. Individual vendors provided
validation of the configuration used for the co-existence technology
under test prior to the start of testing. Some NAT444 testing used
private [RFC1918] IPv4 space between the CGN and CPE router; other
tests used public (non-[RFC1918]) IPv4 space between the CGN and CPE
router. With the exception of 6to4 ([RFC3056]) traffic, we observed
no difference in test results whether private or public address space
was used. 6to4 failed when public space was used between the CGN and
CPE router was public, but CPE routers did not initiate 6to4 when
private space was used.
CPE gateways and client devices were configured with IPv4 or IPv6
addresses using DHCP or manual configuration as required by each of
the devices used in the test.
All devices were brought to operational state. Connectivity of CPE
devices to provider network and public Internet were verified prior
to start of each test.
IP sniffers and metrics tools were configured as required before
starting tests. IP capture and metrics data was collected for all
failed test scenarios. Sniffing was configured behind the home
Donley, et al. Expires November 15, 2012 [Page 11]

Internet-Draft NAT444 impacts May 2012
routers, north and south of the CMTS, and north and south of the CGN.
The test technician executed test scenarios listed above, for single
and dual ISP environments, testing multiple users on multiple home
networks, using the applications described above, where applicable to
the each specific test scenario. Results checklists were compiled
for all tests executed and for each combination of devices tested.
3. Observed CGN Impacts
CGN testing revealed that basic services such as e-mail and web
browsing worked normally and as expected. However, there were some
service affecting issues noted for applications that fall into two
categories; dropped service and performance impacted service. In
addition, for some specific applications in which the performance was
impacted, throughput, latency and jitter measurements were taken. We
observed that performance often differs from vendor to vendor and
from test environment to test environment, and the results are
somewhat difficult to predict. So as to not become a comparison
between different vendor implementations, these results are presented
in summary form. When issues were identified, we worked with the
vendors involved to confirm the specific issues and explore
workarounds. Except where noted, impacts to NAT444 and DS-Lite were
similar.
In 2010 testing, we identified that IPv6 transition technologies such
as 6to4 [RFC3056] and Teredo [RFC4380]) fail outright or are subject
to severe service degradation. We did not repeat transition
technology testing in 2011.
3.1. Dropped Services
Several peer-to-peer applications, specifically peer-to-peer gaming
using Xbox and peer-to-peer SIP calls using the PJSIP client, failed
in both the NAT444 and Dual-Stack Lite environments. Many CGN
devices use "full cone" NAT so that once the CGN maps a port for
outbound services, it will accept incoming connections to that port.
However, some applications did not first send outgoing traffic and
hence did not open an incoming port through the CGN. Other
applications try to open a particular fixed port through the CGN;
while service will work for a single subscriber behind the CGN, it
fails when multiple subscribers try to use that port.
PJSIP and other SIP software worked when clients used a registration
server to initiate calls, provided that the client inside the CGN
initiated the traffic first and that only one SIP user was active
behind a single IPv4 address at any given time. However, in our
Donley, et al. Expires November 15, 2012 [Page 12]

Internet-Draft NAT444 impacts May 2012
testing, we observed that when making a direct client-to-client SIP
call across two home networks on a single ISP, or when calling from a
single home network across dual ISPs, calls could neither be
initiated nor received.
In the case of peer-to-peer gaming between two Xbox 360 users in
different home networks on the same ISP, the game could not be
connected between the two users. Both users shared an outside IP
address, and tried to connect to the same port, causing a connection
failure. There are some interesting nuances to this problem. In the
case where two users are in the same home network and the scenario is
through a single ISP, when the Xbox tries to register with the Xbox
server, the server sees that both Xboxes are coming through the same
public IP address and directs the devices to connect using their
internal IP addresses. So, the connection ultimately gets
established directly between both Xboxes via the home gateway, rather
than the Xbox server. In the case where there are two Xbox users on
two different home networks using a single ISP, and the CGN is
configured with only one public IPv4 address, this scenario will not
work because the route between the two users cannot be determined.
However, if the CGN is configured with two public NAT IP addresses
this scenario will work because now there is a unique IP address to
communicate with. This is not an ideal solution, however, because it
means that there is a one-to-one relationship between IP addresses in
the public NAT and the number of Xbox users on each network.
Update: in December, 2011, Microsoft released an update for Xbox.
While we did not conduct thorough testing using the new release,
preliminary testing indicates that Xboxes that upgraded to the latest
version can play head-to-head behind a CGN, at least for some games.
Other peer-to-peer applications that were noted to fail were seeding
sessions initiated on Bittorent and uTorrent. In our test, torrent
seeding was initiated on a client inside the CGN. Leeching was
initiated using a client on the public Internet. It was observed
that direct peer-to-peer seeding did not work. However, the torrent
session typically redirected the leeching client to a proxy server,
in which case the torrent session was set up successfully.
Additionally, with the proxy in the network, re-seeding via
additional leech clients worked as would be expected for a typical
torrent session. Finally, uTorrent tries to use STUN to identify its
outside address. In working with vendors, we learned that increasing
the STUN timeout to 4 minutes improved uTorrent seeding performance
behind a CGN, resulting in the ability for the uTorrent client to
open a port and successfully seed content.
Donley, et al. Expires November 15, 2012 [Page 13]

Internet-Draft NAT444 impacts May 20123.2. Performance Impacted Services
Large size file transfers and multiple video streaming sessions
initiated on a single client on the same home network behind the CGN
experienced reduced performance in our environment. We measured
these variations in user experience against a baseline IPv4
environment where NAT is not deployed.
In our testing, we tried large file transfers from several FTP sites,
as well as downloading sizable audio and video files (750MB - 1.4 GB)
from the Internet Archive. We observed that when Dual-Stack Lite was
implemented for some specific home router and client combinations,
the transfer rate was markedly slower. For example, PC1 using one
operating system behind the same home router as PC2 using a different
operating system yielded a transfer rate of 120Kbps for PC1, versus
250Kbps for PC2. Our conclusion is that varying combinations of home
routers and CE client devices may result in a user experience that is
less than what the user would expect for typical applications. It is
also difficult to predict which combinations of CPE routers and CE
devices will produce a reduced experience for the user. We did not
analyze the root cause of the divergence in performance across CE
devices, as this was beyond the scope of our testing. However, as
this issue was specific to Dual-Stack Lite, we suspect that it is
related to the MTU.
While video streaming sessions for a single user generally performed
well, testing revealed that video streaming sessions such as
Microsoft Smooth Streaming technology (i.e. Silverlight) or Netflix
might also exhibit some service impacting behavior. In particular,
this was observed on one older, yet popular and well-known CPE router
where the first session was severely degraded when a second session
was initiated in the same home network. Traffic from the first
session ceased for 8 s once the second session was initiated. While
we are tempted to write this off as a problematic home router, its
popularity suggests that home router interactions may cause issues in
NAT444 deployments (newer routers that support DS-Lite were not
observed to experience this condition). Overall, longer buffering
times for video sessions were noted for most client devices behind
all types of home routers. However, once the initial buffering was
complete, the video streams were consistently smooth. In addition,
there were varying degrees as to how well multiple video sessions
were displayed on various client devices across the CPE routers
tested. Some video playback devices performed better than others.
3.3. Improvements since 2010
Since CableLabs completed initial CGN testing in 2010, there have
been quantifiable improvements in performance over CGN since that
Donley, et al. Expires November 15, 2012 [Page 14]

Internet-Draft NAT444 impacts May 2012
time. These improvements may be categorized as follows:
o Content provider updates
o Application updates
o Improvements on the CGNs themselves
In terms of content provider updates, we have noted improvements in
the overall performance of streaming applications in the CGN
environment. Whereas applications such as streaming video were very
problematic a year ago with regard to jitter and latency, our most
recent testing revealed that there is less of an issue with these
conditions, except in some cases when multiple video streaming
sessions were initiated on the same client using specific types of
home routers. Applications such as MS Smooth Streaming appear to
have addressed these issues to some degree.
As far as application updates, use of STUN and/or proxy servers to
offset some of the limitations of NAT and tunneling in the network
are more evident as workarounds to the peer-to-peer issues.
Applications appear to have incorporated other mechanisms for
delivering content faster, even if buffering times are somewhat
slower and the content is not rendered as quickly.
CGN vendors have also upgraded their devices to mitigate several
known issues with specific applications. With regard to addressing
peer-to-peer SIP call applications, port reservations appear to be a
workaround to the problem. However, this approach has limitations
because of there are limited numbers of users that can have port
reservations at any given time. For example, one CGN implementation
allowed a port reservation to be made on port 5060 (default SIP port)
but this was the only port that could be configured for the SIP
client. This means that only one user can be granted the port
reservation.
3.4. Additional CGN Challenges
There are other challenges that arise when using shared IPv4 address
space, as with NAT444. Some of these challenges include:
o Loss of geolocation information - Often, translation zones will
cross traditional geographic boundaries. Since the source
addresses of packets traversing an LSN are set to the external
address of the LSN, it is difficult for external entities to
associate IP/Port information to specific locations/areas.
Donley, et al. Expires November 15, 2012 [Page 15]