Tag Archives: Routing Problem

Quick analysis of BSNL-Level3 bad routing issue

I can see BSNL having pretty high latency again with most of Europe again. It seems like they are using Level3 Communications AS 3356 along with Tata-VSNL for upstream. With Level3 transit BSNL has badly screwed up reverse path causing very high latency and awful bandwidth.

1

2

3

4

5

6

7

8

9

10

11

12

anurag@laptop:~$ping server7-c5

PING server7.anuragbhatia.com(178.238.225.247)56(84)bytes of data.

64bytes from server7.anuragbhatia.com(178.238.225.247):icmp_req=1ttl=52time=320ms

64bytes from server7.anuragbhatia.com(178.238.225.247):icmp_req=2ttl=52time=320ms

64bytes from server7.anuragbhatia.com(178.238.225.247):icmp_req=3ttl=52time=319ms

64bytes from server7.anuragbhatia.com(178.238.225.247):icmp_req=4ttl=52time=327ms

64bytes from server7.anuragbhatia.com(178.238.225.247):icmp_req=5ttl=52time=320ms

---server7.anuragbhatia.com ping statistics---

5packets transmitted,5received,0%packet loss,time4004ms

rtt min/avg/max/mdev=319.880/<strong>321.765</strong>/327.384/2.828ms

anurag@laptop:~$

Expected latency values here should be around 150ms. A packet should not take more then 150ms round trip between Radaur, Haryana to Munich located server.

Clearly hop 3 is New Delhi (30ms latency), hop 4 is Mumbai (again as per latency values). Hop 5 is London Level3. Seems like BSNL used Europe-India gateway link here (a submarine cable from Mumbai to London owned by multiple providers including BSNL and Bharti Airtel along with Global Crossing which is now owned by Level3). Also, as far as I know Level3 does not has a ISP license in India (doT’s list here) and thus they cannot sell bandwidth at Mumbai. Likely BSNL is using its own ILD license in this case and thus BSNL is responsible for purchase of bandwidth in London.

Thus, as per that traceroute and fact that BSNL is one who is purchasing transit from Level3 in London, BSNL should be having BGP session in London and should be exchanging it’s routing table in turn for global routing table provided by transit. While latency jumps as soon as we hit London as per that traceroute. Clearing BSNL > Level3 path seems OK while return path on Level3 > BSNL is faulty.

Using Level3’s looking glass, we can have a quick check on traceroute to my IP:

1

Show Level3(London,England)Traceroute to117.207.48.1<br/>1ae-51-51.csw1.London1.Level3.net(4.69.139.88)0msec<br/>ae-52-52.csw2.London1.Level3.net(4.69.139.120)0msec<br/>ae-51-51.csw1.London1.Level3.net(4.69.139.88)0msec<br/>2ae-227-3603.edge3.London1.Level3.net(4.69.166.154)0msec<br/>ae-117-3503.edge3.London1.Level3.net(4.69.166.138)0msec<br/><strong>ae-226-3602.edge3.London1.Level3.net(4.69.166.150)32msec</strong><br/><strong>3gblx-level3-50g.London1.Level3.net(4.68.110.158)8msec4msec0msec</strong><br/>4ae6.scr4.LON3.gblx.net(67.17.106.150)[AS3549{GBLX}]0msec0msec<br/>ae5.scr3.LON3.gblx.net(67.17.72.22)[AS3549{GBLX}]4msec<br/><strong>5so5-0-0-2488M.ar1.NYC1.gblx.net(67.17.64.146)[AS3549{GBLX}]104msec</strong><br/>so6-0-0-2488M.ar1.NYC1.gblx.net(67.17.64.154)[AS3549{GBLX}]68msec68msec<br/><strong>6BHARTIBSNL.so-7-0-0.ar1.NYC1.gblx.net(64.210.30.70)[AS3549{GBLX}]268msec268msec264msec</strong><br/>7218.248.255.101[AS9829{APNIC-AS-3-BLOCK}]276msec272msec276msec<br/>8117.207.48.1[AS9829{APNIC-AS-3-BLOCK}]272msec280msec276msec

Hop3 – Level3, hop4 is Gblx (which is now owned by Level3), hop 5 is Gblx New York and hop 6 is BSNL router in New York. The target BSNL ip is coming from 117.207.48.0/20. Now interesting thing here is BSNL uses Level3 + Gblx both for transit. So return path via Gblx is not an issue but the path London > New York > India is surely an issue.

Clearly here’s the issue. BSNL again is doing selective BGP announcement of prefixes at New York only and that is why Europe to India traffic is being routed via New York. BSNL is allowing entry path into it’s network from outside India only at New York and few other selected locations which causes serious damage to latency.

Time for me to get back on work of routing packets! Thanks for reading. 🙂

So what’s really wrong? Why VSNL-AS4755 has high latency while AS6453 low – even when AS4755 routes (most of) international traffic via AS6453 which is very popular system (and one of tier 1 networks).

Pretty much good. So whole problem is between routing tables for AS4755 which used to be VSNL – Govt, owned monopoly in India and AS6453 which used to TeleGlobe – both of which now belongs to Tata Communications.

Both of these Autonomous Systems play an important role since AS6453 is one of big tier 1 backbone network, while AS4755 has significant presence in India and most of domestic ISP’s purchase bandwidth from Tata Communications via AS4755 backbone. The extent of brokeness can be seen from a end user BSNL connection – e.g a traceroute to Mumbai based PoP of AS6453 is goes like:

With hope that Tata Communications will eventually fix these broken backbones, I get back to my Java exam preparation! 🙂

Disclaimer: Analysis done in this post is purely based on facts found from publically available information. I have no intention to harm Tata Communications from this. If anyone finds anything incorrect in this post, please feel free to contact me, and I would be happy to fix it.