NA Server Roadmap Update: Addressing the Q's in Quality Connections

**EDIT:** Thanks everyone for your questions! I missed a few ISP partners originally, so they've been added to the list at the bottom of the post. We'll continue updating you as work on the South Bridge and Phase 3 of the Roadmap develops!
> *TL;DR (because this post is a big one!): There’s more that goes into lag than just your ping, so we’re attacking the problem from multiple angles. Due to some delays and a typhoon in China affecting our network gear supplier, our delivery date for hardware build out of the South Bridge has shifted from end of March to late April/early May, and we’ve added several new ISPs to our partner list! Find them at the very bottom of this post. For those of you hungry for the nitty-gritty, read on!*
Hey, everyone! Riot Ahab here again with a progress update on the [NA Server Roadmap](http://boards.na.leagueoflegends.com/en/c/help-support/WH8doH76-na-servers-and-the-future). Last update, we covered Phase 2 of the roadmap: optimizing internet connections for League and you. Specifically, we touched on building points of presences (PoPs) and establishing peering agreements across the “North Bridge” of the network, covering the northern US and Canada. You can check out that post [here](http://boards.na.leagueoflegends.com/en/c/help-support/0hVNFnPH-na-server-roadmap-update-pops-peering-and-the-north-bridge).
As we’ve mentioned previously, Phase 1 (new server infrastructure) and Phase 2 (connection optimization) of the Server Roadmap are unlikely to bring a significant impact to your ping. So why even bother? Because those phases instead focus on delivering a quality connection while you play League.
##*So what do you mean by “quality connection”? Just fix my ping, Rito!!*
We’re working on that! The difficulty is that many people equate “ping” with the deciding factor between a good and bad connection. The more complicated truth is there are many factors beyond ping that impact the millions of pieces of data coursing between your keyboard and the game servers. For sanity’s sake, though, let’s focus here on the big three: connection stability, round trip time (ping), and packet loss.
##Connection Stability
Connection stability refers to a consistent, predictable schedule when the server is online. Ideally, you’d never have to think about this part of your connection! Stability is negatively impacted by malicious attacks, DDoS, and larger-scale hardware problems (or your pet chewed on the router cable again).
Our philosophy is that there are only two reasons you shouldn’t be able to play League: scheduled downtime for content patches and scheduled infrastructure maintenance. Which is why we upgraded our more outdated server hardware during Phase 1 of the NA Server Roadmap. Odds are you didn’t feel much difference with the upgraded hardware - but in the background the new server architecture has already been working hard to mitigate large-scale malicious attacks like ones that plagued NA last year and providing much better uptime when unexpected hardware-related issues occur.
##Round Trip Time
This usually is referred to as good-old “ping”. Your round trip time is a measurement of the time it takes for a packet of information to be sent from your machine to our servers and back again, and is itself impacted by a number of additional factors:
Physical Distance: Data travels fast (near the speed of light), but it’s still impacted by the distance it needs to travel. As a general rule, the further away you physically are from a server destination, the higher your ping will be.
Routing: Physical distance can (and often does) differ from routing distance. Data doesn’t travel as the crow flies, but follows copper and fiber optic cables that make up the many different transit networks across the US and Canada. Routing algorithms determine the specific path taken by chunks of data, and they don’t always work in the most efficient manner. You could be right down the street from the League servers, but if your ISP’s routing algorithms are funky, your data could bounce to several different locations before finally arriving at the servers just next door.
Congestion: Internet traffic can slow down when there’s a spike in the volume of local data. This can occur at multiple levels: from getting slowed down on your home network by a family member who’s binge streaming House of Cards in HD to your local neighborhood circuit getting overloaded because everyone is streaming even more House of Cards. (House of Cards is a great show, btw). Peak times and high traffic can cause local ping spikes that slow down your League traffic.
Ping is often the most cited measure of what makes a quality connection - and as the only “connection” indicator we include in-game, it’s natural for players to equate a quality connection with this one number. While ping is important, it’s not the only thing that makes or breaks a good connection. As a result, we can’t just move the servers and call it a day - we’ve had to take a multi-step approach that addresses all of the factors that affect connection quality.
##Packet Loss
Before we even begin, let me just tell you: **this is the big one**. It’s less catastrophic than poor stability (nobody can play when the servers are down) - but arguably more damaging to games than high ping. Packet loss is the silent killer that leaves you rubber-banding all over the rift, or freezes you up in the middle of the team fight just to come back to a grey screen of death.
So here’s how packet loss goes about its nefarious business: in any League game, there is an ongoing exchange of information between your machine and the League servers. Along that route, there are multiple “handoff” exchanges, where packets have to navigate their way across the internet from system to system. Every exchange introduces a potential point of failure or congestion, where packets can become “lost,” and fall off of their happy journey between your computer and the League servers. If that happens, the game servers are left with incomplete information and aren’t sure what your last command was, and have to ask your machine again. Hence hang time, frustration, table-flipping, and grey screens.
While ping may be more heralded, the inherently inconsistent and sporadic nature of packet loss makes it uniquely damaging to gameplay and, as result, makes it a big priority for us to address as part of the NA Server Roadmap. By comparison, when running internal tests simulating various connection issues we see that players can “adjust” to higher ping as long as the connection is consistent and packet loss is minimal. That’s not to say high ping isn’t awful as long as it’s consistent (it most certainly is), but it does indicate how important packet loss is as part of the overall connection picture.
-------------------------------
##*OK, so lots of things can go wrong with my connection. What’re you doing about it, Riot?*
This is why the NA Server Roadmap is divided up into multiple components - they’re all geared to address different parts of what make up a quality connection.
* **Phase 1:** The architecture upgrade [back in November](http://boards.na.leagueoflegends.com/en/c/riot-official/0YOGUwHG-stability-upgrade-and-planned-migration-for-na-servers) has been actively fending off malicious attacks with greater efficiency. DDoS attacks will continue to be a challenge, but by moving to a new location and upgrading our hardware we’re much less susceptible than before. Plus the older Frankenstein’d design was more prone to hardware failures and was harder to maintain without planned and unplanned downtime.
* **Phase 2:** Connection optimization is focused on removing potential points of failure and putting your data on the most direct route between your machine and the servers. Since I’ll be talking about “optimization” quite a bit, check out the images below to help illustrate the idea:
* Here’s your connection. On one end is the local network between you and your ISP, and on the other end is the NA League game server. When playing, your data needs to get from your machine to the servers and back again. Ideally, your data would take a straight shot across all the networks involved.
http://i.imgur.com/AwpoMOQ.jpg
* But as you have likely already figured out, your data doesn’t always take the most sensible path. Most ISPs focus on moving as MUCH data as possible, not as fast as possible, which often puts your data on a wonky path, which increases route distance (ping) and potential points of failure (packet loss).
http://i.imgur.com/Z3ZEGZ4.jpg
* This is where the direct network comes into play. By setting up hardware in strategic locations across NA, we can channel local traffic onto our network, which removes additional hops, points of failure, and routing distance, which all cuts down on packet loss (and to a much lesser extent, ping!). For this example, let’s presume you are playing League in Boston, MA, closest to the New York, NY PoP we’ve recently set up as an access point to our direct network:
http://i.imgur.com/Yuqd4rm.jpg
* Now your data is on a much more direct route, but there’s still some inefficiency between the ISP and your closest PoP hardware location. This is what we mean when we talk about “route optimization and tweaking” once hardware and peering agreements are online - we’re not only working with ISPs to hand off your data to our network, but to also route your traffic in the most direct way to the nearest PoP through ongoing work on route optimization.
http://i.imgur.com/83C75NH.jpg
* And better yet, when optimized this network allows us to redirect traffic with ISPs should any one hardware station fail. Let’s say you’re still playing in Boston, MA, and our New York, NY hardware fails for some reason. The network will redirect your traffic to the next closest PoP (in this case Ashburn, VA) to get you back onto the network seamlessly!
http://i.imgur.com/InBYbUu.jpg
* **Phase 3:** the last phase of the roadmap will see the game servers move from their current location on the West Coast to a more centralized location in NA. This move will (for a majority of players) improve the geographic distance factors between them and the game servers in order to lower ping values; and in broad strokes should improve network quality for everyone.
* Now you may have zeroed in on how I wrote “for a majority of players.” What overly cautious PR corporate speak you sling, Ahab! As many of you following these updates have already theorized, moving the servers to a central location will alleviate the excessively high East Coast ping numbers, but may elevate West Coast ping numbers as result. The direct network we’re currently building will, however, mitigate much of the ping a move like this would normally bring, since the routing pain will be largely optimized.
* Between the new architecture, direct network pathing and local network optimization, by the time Phase 3 is complete everyone on NA should be feeling an improved network quality connection. That’s both our goal and our theory, which we’ll keep testing out, working towards and updating you about as we go. Also it's important to call out that if you live really far from a major population center it's going to be harder for us to optimize your network connection to the same level of quality as a player that lives closer.
-------------------------------
Whew! You made it. That’s about all we have to share this time around - thanks for reading! The last few bits of information we wanted to share were a few quick updates to the network. Last time, we cited a goal to have all network hardware placed and installed by the end of March. However due to the length of time involved in negotiations for space and circuits, and because a typhoon in China delayed some of our network equipment shipments, we are now targeting the end of April or early May to have the complete southern expansion done. I will keep you updated as more PoPs come online!
The last bit of information to share is an update to the list of ISPs we’re currently working with to get you on our network via the shortest route possible. Some big names have been added! I’ve got the entire list of partnered ISPs just below, with the latest additions at the top. Remember: this is an ongoing list! Just because you don’t see your ISP doesn’t mean they’re not part of the plan - odds are we’re already in talks with them. Also many dozens of smaller ISPs are getting directly connected to Riot through well managed public peering exchanges, we will work to get that list up on the web somewhere soon.
Thanks everyone for your patience and help improving the NA server experience. ~~The team and I will be hanging around the thread for a while answering whatever questions we can, so feel free to fire away!~~
Cheers,
-Riot Ahab
---
**Partnered ISPs:**
*(new additions)*
Time Warner Cable
Verizon (very early stages - we have an agreement but haven't implemented big changes yet!)
CenturyLink
Frontier Communications
WOW!
TDS Telecom
Videotron Ltee
Google Fiber
Pavlov Media
CTS Communications Corp
*(original list)*
Atlas Networks Corporation
Charter Communications
Clear Wireless
Cogent Communications
Comcast Cable Communications
Eltopia
Hurricane Electric
Interconnected Associates
LS Networks
NetRiver
NTT America
Pocketinet Communications
Rogers Cable Communications
Saskatchewan Telecommunications (SaskTel)
Shaw Communications
Syringa Networks
TekSavvy Solutions
TELUS Communications
TeraGo Networks
Threshold Communications
Vision Net
WiscNet
Worldlink
_Thanks everyone for your questions! This thread's comments are being disabled and archived - please jump back [here](http://boards.na.leagueoflegends.com/en/c/help-support/WH8doH76-na-servers-and-the-future) to the main post and scroll to the bottom to find the most recent NA Server Roadmap update if you have any questions you'd like addressed!_

Since he asked about a company I'll ask as well anything from Midcontinent Communications? Since I see WOW! on there I was hoping they would be as well. If nothing yet should I reach out to them and talk to them?

> [{quoted}](name=Riot WizardOTL,realm=NA,application-id=osqw6G4M,discussion-id=APgXgX6W,comment-id=00000000,timestamp=2015-03-05T22:39:39.738+0000)
>
> We have reached out to Cablevision, but we have not received a reply from their peering group as of yet. Still hoping to hear from them though!
Is there anything customers can do to help the message get across? Long Islander here that has Optimum Online.

> [{quoted}](name=Riot WizardOTL,realm=NA,application-id=osqw6G4M,discussion-id=APgXgX6W,comment-id=00000000,timestamp=2015-03-05T22:39:39.738+0000)
>
> We have reached out to Cablevision, but we have not received a reply from their peering group as of yet. Still hoping to hear from them though!
I'd like to help this come to fruition, should I call Optimum? What should I say?
edit: You guys are doing amazing work by the way, where's the tip jar?

> [{quoted}](name=Riot WizardOTL,realm=NA,application-id=osqw6G4M,discussion-id=APgXgX6W,comment-id=00000000,timestamp=2015-03-05T22:39:39.738+0000)
>
> We have reached out to Cablevision, but we have not received a reply from their peering group as of yet. Still hoping to hear from them though!
How about Brighthouse Networks in the eastern central part of Florida? I currently use ATT uverse but brighthouse is the other one in my area incase ATT cant reach a deal

Not sure where you Colo in NY area but if it is at Halsey Street in Newark(NJ) request to not peer on the old switches for NYIIX, they need to bring the whole bundle down if a port is to be added or removed. Makes for a nightmare if a port goes bad on their end. I may or may not have experience peering with them there ;). However, their NY connections are up-to-date and just fine. I guess previously working at a CDN that might be down the road from Riot has given me some tips I can share.

> [{quoted}](name=Riot WizardOTL,realm=NA,application-id=osqw6G4M,discussion-id=APgXgX6W,comment-id=00000000,timestamp=2015-03-05T22:39:39.738+0000)
>
> We have reached out to Cablevision, but we have not received a reply from their peering group as of yet. Still hoping to hear from them though!
How can we help you guys hook up our more little ISP? Is it a hard process or just a few things on both your ends?

If your ISP happens to have a network presence in the same facilities as us, it makes it much easier for our networks to connect (providing your ISP is willing!).
We publicly list our network presences where we are able to peer with any willing ISPs [here](http://as6507.peeringdb.net).
In the event an ISP is not at a mutual location, it does make it more difficult - but not impossible!
Let us know of your ISP and we'll see what we can do.

Speaking from the perspective of someone on Comcast in Silicon Valley, I've gone from 42ms low packet loss, down to 33ms nearly-zero packet loss a few months ago, back up to 42ms but with nearly-zero packet loss as of what feels like 3-4 weeks ago.
Was that short time where I had 33ms the result of test runs for route optimization, or was that a fluke that I shouldn't expect to see back anytime soon?
I hadn't run any traceroutes at the time but now I wish I had so I could compare.

That should have been due to the optimization. Can you run some traceroutes and open a ticket with Comcast? We can work with them and see if there is an issue on their network.
Shoot the traceroutes to riotwizardotl@riotgames.com

Well my ISP is GVTC Communications (Guadalupe Valley Telecommunication Coop I think) sadly I dont know who they coop with but they service most of the hill country in Texas. Would yall be able to reach out to them and see if you could optimize the routing for them?
My main question is WHEN will phase 3 be put into effect if you are able to give an estimation cause I miss my original 40-50 ping I had now I get like 80+ ping no matter what other than on the PBE server. The PBE server ill see 53 ping regularly.

Part of our strategy for the smaller ISPs like yours was that, in the event we were unable to set up a regular peering arrangement we'd do the next best thing in the interim - purchase tier 1 IP Transit from the common upstreams.
AGOC do not appear* (on face value, I did a few quick database searches) to have the ability to peer with Riot at this stage, but we do share three mutual tier 1 IP Transit providers with them (Level3, Cogent and Zayo/AboveNet).
This doesn't directly peer Riot with your ISP, but it reduces the number of intermediary networks to just one.
*The above statement is not necessarily true! If you would like to raise a support ticket with your ISP and link them to this post and if they are interested, reach out to us at peering@riotgames.com!

In [this previous NA Server Roadmap Update](http://boards.na.leagueoflegends.com/en/c/help-support/0hVNFnPH-na-server-roadmap-update-pops-peering-and-the-north-bridge) there was mention of a PoP in Hillsboro, Oregon. In this update you mention you are working with Frontier Communications. Both of these facts gave me great hope as I live there and use that ISP. Unfortunately a coworker and I continue to see higher pings, more fluctuations in the ping times, and most recently enough packet loss to cause in-game warping.
What should I be doing about this to help figure out what is happening? I have been running continuous pings to google.com during my games and do not see packet loss or high ping times. I don't know how to generate any useful data for you about how service to Riot is getting worse while you work directly with my city and ISP.
As anecdotal information I have fluctuated daily in ping values between 30ms and 140ms.
Small followup question, why is Riot working in Hillsboro and not Portland (the much larger neighbor city). I am enthused but wonder if the large Intel presence in Hillsboro has that kind of sway.

Frontier does not peer in Portland. We currently peer with them in Seattle (and other places, but that is the closest IX to you). Can you traceroute to 192.64.170.1 and send the result to riotwizardotl@riotgames.com? After that we may ask you to open a ticket with Frontier so they can allocate people to look at it with us.

Hey thanks for doing this! Also I have HUGE packet loss 20+ per minute in every single game (still managed to climb to silver 5) and I realize most of it is on my end, forced to play on wifi, have no control over the router, ISP is bronze 5, What can I do on my computer to fix this packet loss?

In the province of Quebec which is pretty much the Eastern Coast of Canada, the big twos are Bell Canada and Videotron. I read that you guys were still trying to communicate with the former, so what about the latter? Videotron is really one of the biggest here, and pretty much the best along with Bell Canada in terms of Internet quality around here.
Either way, thank you so much for your work up until way and rock on. :)

I know you can't answer yes or no to if you've spoken to them or not, but is Windstream Communications an isp on the radar for potential talks for the routing? I know they lease fiber to Verizon in some areas and is huge here in the midwest united states (I know this because my dad works there and they are a fortune 500 company)

Thank you for this update! I can already tell that i am no longer experiencing packet loss as i had earlier this year (thanks to the PoP in NY). I hope that connection quality will continue to improve for many other players as the "South Bridge" slowly comes online and as more ISPs get partnered.
And we can already tell that there hasn't been a significant reduction major service interruptions since the hardware upgrade. (I can only recall maybe one interruption, as compared to almost a problem at least once a month).
Keep it up Rito!

Is there any way Riot could provide us with a small client that could simulate connecting to your game servers, then give us feedback on ping and packet loss? I am hesitant to play at night from the East Coast as I have no idea until I get in game if the game will be playable or not. Rather than playing Russian roulette with my LP I'd like to know beforehand if I am going to have a chance or not.

Hey Riot! I'm no networking genius so some of this didn't make sense to me, but the illustrations helped a ton. If I'm wrong correct me but does the NY server mean that players on East Coast will get better ping? I play at a consistent 100 and I see this players from the west coast with like 20. Can players from the East Coast be expecting better ping soon or is that still a WIP?

To clarify -- the hardware placed in New York is more like an on-ramp to the network rather than a free-standing server. Once your local traffic gets to that PoP, it'll put you on the direct network to wherever the servers are.
We've been seeing varying impacts on ping (some are unaffected, some are seeing an improvement of 20ms), but this network will likely not affect your ping as it will your connection stability and packet loss. The more direct your route to the servers, the less likely you are to drop packets along the way.
Phase 3 is likely to bring the biggest impact to players ping on the East Coast, which is still WIP.

**Culture**
This is the place to get player-to-player assistance for Tech or general questions and knowledge! Riot Support lurks from time to time but if you would like to get official support, please reach out to our team [here](https://support.riotgames.com/hc/en-us/requests/new).
**Etiquette**
***DO NOT*** post bug reports here. Instead, head on over to our [Report a Bug](http://boards.na.leagueoflegends.com/en/c/bug-report) board.
> *** {{champion:74}} : I require my adjustable reverse-ratcheting magnetic box wrench!***