Jio Or Airtel – Who Will Win The LTE Battle ?

The 2015 State of Mobile Performance report contained limited coverage on the Indian market. We saw that Wifi was much faster than LTE; sometimes as much as 3 times faster, as in Mumbai. We recognized Airtel’s dominance in the carrier space, we did a comparison of Tier I and Tier 2 cities, and noted a consistent performance from LTE for the majority of users.

Due to strong interest and the size of the Indian market, we decided to expand our coverage for 2016 in order to include more topics and explore how the results have evolved from the previous year. More specifically, we will look at how the carriers rank across 4 categories (performance, quality, variability, coverage), how Jio, a new entrant in the LTE market, is performing, and whether iOS or Android perform better.

Here are the key highlights of the India market:

LTE sees stunning 5x growth in traffic, but overall share is still at 23%. Most of the traffic is over HSPA.

70% of API responses are smaller than 10KB; the most commonly used image size is between 10KB and 25KB.

Jio emerges as a valid LTE competitor to Airtel. While it is not performing as fast as Airtel, it is second best in quality—the experience is more consistent and the coverage much better.

Contrary to 2015, LTE is now much faster than WiFi across almost all cities in India. Even in Mumbai where Wifi was 3x faster, LTE has now caught up.

There are no differences in performance between Tier I cities, like Mumbai and Bengaluru and Tier 2 cities, like Jaipur and Pune.

Android is 35% faster than iOS for both LTE and WiFi.

LTE is the most variable network of all. Fast speeds on LTE are 2.7 times bigger than the median.

Optimal image size for guaranteed delivery under 200ms is 12.5KB over 3G and 13.5KB over LTE.

1. LTE Sees Extraordinary Growth In 2016

In 2015, traffic over LTE was minimal. It was barely crossing the 5% mark. 3G and HSPA were dominant, accounting for almost 85% overall mobile traffic. While these two continue to remain dominant with 75% of the total market, LTE has seen extraordinary growth. Traffic over LTE is now 5x greater than in 2015 at 23% of total traffic.

Considering the fact that Airtel has already deployed advanced LTE in some areas of India, we should expect the total LTE share to increase substantially in the coming months. There is definitely a lot of room for growth. On the other side of the spectrum, 2G traffic is steadily shrinking.

2. Developers In India Are Content Size Conscious

Drilling into the sizes that developers are using for APIs and images, we observe just how size conscious they are.

Expecting less than optimal speeds due to high 3G and HSPA adoption, developers tend to keep payloads small. 75% of images requested over the network are less than 50KB, and the median image size is just 22KB.

In comparison, the median in the US is 50% bigger—75% of images are also less than 50KB. Seems that 50KB Surfaces as the gold standard for image sizes globally.

On the API front, 75% of all API requests are less than 25KB, with a median size of 8KB. The majority of the API calls are 0<10KB. Developers try to keep their responses small in an effort to bring the results back to the user as quickly as possible. That said, our data indicates that this may not always be enough, since 40%-50% of total completion time for APIs is spent processing on the backend.

Given that LTE usage in India has risen 5 times in just 6 months and speeds will naturally increase as a result of that, it will be interesting to see how the image file size distribution evolves.

3. Airtel Is The Fastest, Jio The Most Consistent

When we looked at carrier performance in our previous report, Airtel had been by far the most dominant. This year, we decided to take a deeper look into all carriers in India and increase the number of categories we use to rank them. Specifically:

– Performance in terms of speed when downloading static files such as images

– Performance variability between the fastest and the average users

– Quality of the network in terms of network response time

– LTE Coverage as measured by the likelihood that an LTE user will access an LTE network when using an app

The results are revealing. First, we notice speeds in the range of 1-2 mbps across the board. Airtel tops our rankings in performance across 2G, 3G, HSPA, and LTE.

It is also has the highest quality network. Jio, which launched in 2016, is slightly slower than Airtel, but tops our charts in consistency, which suggests that it provides a more uniform experience for its users. Jio also ranks first in our coverage metric, which is somewhat expected given that it is a 4G/LTE only network.

It is important to highlight that these are live performance numbers as experienced by mobile users in India. Static files, such as images, provide a good proxy of network performance, since the backend serving the images has to perform a simple memory lookup. We excluded dynamic files, such as JSON, from our results to avoid cases where slow performance would be an outcome of slow backends.

We measured performance for all networks (2G, 3G, HSPA, and LTE) and found that Airtel is the fastest carrier across the board. In LTE, Airtel is 1st (15% faster than Tata Docomo) followed by Aircel and Reliance in 3rd and 4th, and BSNL and JIO in 5th and 6th. To put this performance ranking into perspective, it would take on average 222ms to download a 40KB image on LTE over Airtel.

When looking at HSPA, Airtel’s dominance is undeniable, measuring 24% faster than Vodafone and 40% faster than 3rd ranked Tata Docomo. Given that HSPA is the most dominant network technology in the Indian market, Airtel and Vodafone emerge as the best carriers (for now) in India.

**We only focused on LTE coverage as this is the fastest network

Airtel and Tata Docomo are definitive performance leaders in 3G, averaging about 300ms-400ms faster when downloading a 40KB image. Aircel, Vodafone and BSNL are ranked as 3rd, 4th and 5th respectively. Overall, we are seeing a 20% bump in speed from 3G to HSPA and 100% increase from HSPA to LTE. Finally, when comparing the performance of HSPA-Plus to HSPA, we see that numbers are identical. So HSPA-Plus deployments have not led to improvements in speeds experienced by users.

Jio is the Most Consistent in Performance

In our performance variability rankings, Jio has the smallest variability index at 1.6. This means that faster Jio users experience about 60% better performance than average Jio users. The gap is bigger for Airtel, Tata Docomo and Vodafone at 70%. Idea is the most variable carrier, with a variability index of 2.2.

Airtel Has the Best Network Quality

Looking at network quality as measured by network response times, Airtel is the best in LTE and HSPA. Reliance is second in LTE with 17ms, BSNL is third with 21ms slower response times. All measurements are for 75% of users. In general, it takes devices about 100ms to access a network.

In HSPA, Airtel tops our charts with 107ms, Tata Docomo is second with 113ms, and Reliance is third with 125ms. Network response times are similar for 3G networks. The top 2 carriers are in the 115ms range. However, slower networks are in the 210-240ms range, which is fairly slow. For example, Vodafone has a fairly high response time of 213ms to reach our accelerators.

In the graphs below, we plotted quality against speed to see if there is a correlation between the two. We observe that in LTE, Airtel, Reliance and Jio are on the top right, indicating networks of both high quality and high performance. For HSPA Airtel, Vodafone and Tata Docomo and for 3G Airtel and Tata Docomo. These are the carriers that combine high quality and speed at the same time.

Jio Has the Best LTE Coverage

Jio tops our LTE coverage rankings with Idea and Airtel in 2nd and 3rd respectively. Aircel has the lowest LTE coverage for the time being. While this is not a sign of how broad LTE coverage is, it is a sign that Airtel and Idea subscribers have a higher chance of using an HSPA-Plus, HSPA or 3G network instead of LTE.

4. LTE Is Faster Than Wifi In Most Cities

In our 2015 report, Wifi outperformed LTE in most major cities. In some cities, the difference was considerable, e.g in Mumbai Wifi was 3x faster than LTE. In 2016, this picture has changed dramatically. We measured performance in the top 10 Indian cities and in all except Mumbai, LTE is faster than Wifi. Coupled with an increase in traffic share, LTE is improving in performance as well. Even in Mumbai, where Wifi was 3 times faster, LTE and Wifi are now tied.

Looking at rankings by city, Pune and Chennai had the fastest performance while Ahmedabad, Surat and Mumbai had the slowest. Chennai and Bengaluru boast the fastest Wifi, while Jaipur and Ahmedabad have the slowest.

Finally, grouping the cities into Tier I (e.g. Mumbai and Bengaluru) and Tier II (e.g. Jaipur and Pune), we do not observe any notable differences in performance. This result is congruent with our results from 2015 that also showed no advantage for Tier I cities.

5. LTE: Better Performance But Also Greater Variability

When looking at the the performance variability by network, we observe that Wifi is the most variable of all. This can potentially be attributed to the high variability in Wifi speeds offered by ISPs. On the cellular network side, variability has decreased compared to 2015. 3G is the least variable of the networks. LTE, while the best performing, is also the most variable. Fast users get 170% better performance than median users.

We look at Network Quality from two angles – Network access Time And Packet Loss

For network access time, the patterns we observe in 2016 are somewhat expected. For 90% of users, network access times for LTE and HSPA are very similar. LTE is marginally better by 50ms, meaning it takes a device up to 180-235ms to get a response from one of our accelerators. Naturally, access times are higher for 3G and 2G networks; it takes 1.2sec for the slowest users to get a response from the network for 3G. A quick comparison with the US reveals that LTE response times lag for about 75% of users, but only by 30ms. Therefore, India and the US are fairly similar as it pertains to the quality of their LTE networks

Packet loss is another metric we use to evaluate a network’s quality. Packet loss is commonly an issue with 2G and 3G networks. LTE was meant to be a solution to this problem and our analysis for India proves that this may just be the case.

Using retransmit rates to estimate the impact of true packet loss on performance, we observe a 6% packet loss in 2G and 2.2% in 3G. While these numbers could slightly overestimate the true packet loss value, they are good proxies and show how big of an issue it is for the 20% of the Indian population that uses 3G. Packet loss is less of an issue for HSPA users (0.9%) and LTE users (0.3%).

7. Android Is 35% Faster Than iOS On LTE And WiFi

India is predominantly an Android market. Looking at our data, Samsung and Motorola devices account for the majority of usage. However, the vast difference in performance between Android and iOS is interesting. We found that it takes on average 415ms to download a 40KB image on iOS, while it takes 240ms (40% slower) to download the same image on Android over an LTE network.

The graph below shows the top 20 devices in India and the time it takes them to download a 40KB image over LTE.

Looking at similar data for Wifi, we see again that Android users are faster than iOS users. They download images 25% faster (248ms vs 322ms).

8. 14 KB Images For Optimal Download

These are still early days for LTE in India. As we saw above, just quarter of overall traffic flows over it and the performance is not on par with the LTE promise. Nevertheless, developers should build their apps with LTE (and Wifi) in mind.

When thinking about the specifications required for images to guarantee an optimal delivery of 200ms to the device, we found the magic number to be 13.5KB on LTE and 15KB in Wifi. At these sizes, the user is guaranteed to see the images load in less than 200ms.

Last year, we ran the same analysis for an optimal delivery time of 2 seconds, as we felt that would be more realistic given the low quality of performance in networks in India. The numbers were 40KB for Wifi and 80KB for LTE. To showcase how much progress there has been, we re-did our analysis with a 2sec threshold. The optimal size came at 90KB for LTE and 95KB for Wifi. There is clearly an upward trend for both Wifi and LTE.

Considerations When Releasing an App in India

India is one of the largest markets in the world with hundreds of millions of mobile app users. Here are some insights for you to consider when planning the release of your app in the Indian market.

LTE is almost there. It grew 5x since last year and covers 23% of the market. At this rate, by the time your app is live, LTE will likely account for the majority of your users.

Expect your LTE users to have a fairly variable experience in terms of speed. Account for cases of low performance even when on LTE.

Spend more time optimizing the performance of your iOS apps. Users tend to have a slower experience on iOS vs Android.

When you have to make a network call to fetch static content, tweak the payload to 14KB for under 200ms (instant) delivery.

Be conscious of API response size to account for slowness in your backend. 67% of our customers’ API traffic is less than 10KB.

Always measure performance of dynamic payloads in terms of time spent on device, network, and the server to better debug performance issues. Just the payload size is not enough.

Even with all of these insights in your tool belt, it’s important to remember that networks are constantly evolving. Whatever improvements you put in place today, you will have to rigorously monitor their effectiveness over time.

(Disclaimer: This is a guest post submitted on Techstory by the mentioned authors.All the contents and images in the article have been provided to Techstory by the authors of the article. Techstory is not responsible or liable for any content in this article.)

About The Author:

Kartik Chandrayana is the Co-founder and CEO of Twin Prime. He is an IIT graduate, and holds a PhD in Computer & Systems Engineering from the US. He leads the company’s strategic direction and business execution. He brings more than 15 years of networking and engineering experience having led engineering teams at Blue Coat Systems and Cisco Systems.

Kartik is also a seasoned speaker having presented his views at premier conferences – both academic and industry-specific, like the recent CDN Summit in NYC which is a premier conference for CDN and networking technology.