Friday, December 2, 2016

As a follow up to my November 1 post on AT&T's plans to deploy
wireless local loops as part of their obligations under Connect America
Fund Phase II (see "Hello Wireless Loops. Goodbye Fiber? Part 2"), the Connect America Fund Phase II eligible areas are now loaded onto the California Interactive Broadband Map. As shown in the legend to the left, four carriers' territories are shown in light blue (AT&T), dark blue (Consolidated), gray (Frontier), and red (Verizon, now also Frontier). To view the areas, go to the map's menu on the right and expand the FCC Data menu. Check the box next to "Connect America Fund Phase II Locations."

Monday, November 21, 2016

Those of you new to CalSPEED often ask how it compares to other speed
testing apps. Here are answers to some of the more common questions.

How Did CalSPEED Begin?CalSPEED was originally funded by a State Broadband
Initiative grant from the National Telecommunications and Information
Administration. The testing program began in spring 2012 and has now completed
10 rounds. The test program collects not only speed data, but also speed
variation, latency, jitter, and packet loss. With these data, we are able to
estimate performance for “over-the-top” streaming voice and video service. With ten rounds of semi-annual mobile testing completed, the CPUC has one of the largest public data sets of mobile broadband performance.

Where Are Tests Performed? The CPUC tests the same 1,990 locations twice a year. The
breakdown is 37% urban locations, 56% rural locations, and 7% tribal locations, which were randomly generated.
The field test relies on two devices from four major providers (AT&T,
Sprint, T-Mobile, and Verizon). 80 TCP tests are performed for each provider,
on each device, at each of the 1,990 locations.

Is 1,990 Locations Enough? Using advanced geo-statistical
methods, we are able tointerpolate
service characteristics likely to be experienced by a user located anywhere in
the state. The CPUC
designed the mix of test locations to cover not only urban places where people
live and work, but also rural locations where people may be passing through,
such as rural highways and state and national parks. All tests are performed
along roads navigable by automobile.

Why Not Test In Every Census Block?Performing
field tests in all of California’s 710,145 census blocks would be prohibitively
expensive, impractical and unnecessary. For this reason, neither the CPUC nor
mobile providers like Verizon, AT&T, T-Mobile, and Sprint perform tests in
every census block, but instead use statistical techniques to approximate
service characteristics in between tested locations.

Why Two Servers? Most testing applications use only one, generally nearby, server.This method understates latency and overstates
throughput as compared to using multiple, geographically-diverse servers.Testing to a nearby server results in speeds
likely to be experienced for applications such as streaming movies, where
content is often cached locally due to its popularity. However, much of the content broadband users
access is not cached locally, so CalSPEED tests two two servers -- one in Arlington, Virginia, the other in San Jose, California -- to
understand the role of back haul networks in each provider’s delivery of mobile
broadband. While using more than two test servers in disparate locations
across the globe would be desirable, using both east coast and west coast
servers yields more representative results that testing to only one
server.

Is This Better Than SpeedTest.net or the FCC's Speed Test? As shown in
a study published
by Novarum in 2014 comparing Ookla, FCC, and CalSPEED testing applications,
results for Ookla and FCC tests tend to be higher because both intentionally
select test servers for lowest latency, which tend to be geographically closer.
Moreover, Ookla’s test further biases results by discarding the bottom half of
upstream results and bottom third of downstream results. By consistently
testing to the same two servers, one on each coast of the continent, CalSPEED
provides a reliable backhaul performance metric for each of the four mobile
providers. Since we began testing in 2012, we have seen the performance
(latency) difference between east and west servers decrease.How Else Does CalSPEED Differ From Other Speed Tests? Most speed test applications rely on crowd sourcing. Crowd
sourcing has an inherent selection bias of only collecting data from where it
is chosen to be used. Where data is collected, it is biased towards who
collected it, why, when and where. In contrast, the CalSPEED methodology has
testers return to the same location every time, and the geographic distribution
of test locations provides a more complete picture of mobile broadband across
the state.

How Many TCP Threads Does CalSPEED Use?Multi-threading
means opening more than one connection to the host and combining them in order
to boost overall throughput and is used by many speed test applications. When
the CPUC designed CalSPEED, we examined the effect of using multiple threads
(“flows”) and concluded there was no material difference in mobile throughput
between four threads versus eight threads or sixteen threads. The current test
design has 4 threads, each divided into ten 1-second tests for upstream to the
west server, then again to the east server. The same is true for downstream.
This is then repeated a second time, totaling eighty 1-second tests. Most applications only use one thread.

Why Do Speed Testing? Carriers Already Have Coverage Maps. Most provider maps show a single coverage color and say things like "4G/LTE Coverage." Through CalSPEED, the CPUC has been able to discern more subtle speed and coverage differences by region. Some
providers advertise speeds, but we have observed that those speeds are not
ubiquitous, that is, they are not available everywhere providers claim to offer
service. Speeds vary
widely depending on if you are in an urban, rural, or tribal location. For this
reason, we create a heat map of speeds based on actual field test data, and the
heat map shows how speeds vary across the state.

Why Not Use Average Speed, Like Mean or Median? The CPUC has
demonstrated[1]
through years of methodical field testing that mean and median speeds, by
themselves, are unreliable indicators of what consumers can expect to
experience reliably at a location.CalSPEED takes
observed variability into account to determine speeds that consumers can
consistently expect to receive.As mean
throughput increases, so does the amount of variability around the mean.

Tuesday, November 1, 2016

Last week, I posted three scenarios examining AT&T's mobile LTE coverage of high cost-eligible homes under the FCC's Connect America Fund Phase II program. That analysis assumed AT&T would be using their existing mobile network and licensed LTE spectrum...

Turns out, that was a rosy assumption.

Since last Friday, I received more information from Steve Blum about AT&T's plans to
deploy wireless local loops (thanks, Steve!). Rather than revisit the issues, I want to make some corrections and comments to my previous post.

AT&T plans to deploy a new technology for wireless local loops whose "success in the marketplace is thus unproven." (paragraph 51 of their SEC 425 filing). For more, see #1 below.

Rather than use licensed LTE spectrum, which includes the incredibly valuable 700 MHz band, it appears AT&T will be using a higher frequency (2,300 MHz, or 2.3 GHz), which requires line of sight between cell tower and customer location. For more, see #2 below.

Other risksinclude spectrum constraints on the number of subscribers who can use the service simultaneously during peak usage, and this "reduce[s] the prospects for a successful rollout by AT&T of fixed WLL as a standalone product." For more, see #3 below.

1. Did AT&T Say They'll Need More Cell Sites?

Group President and Chief Strategy Officer John Stankey's testimony, which is part of AT&T's Form 425 filing for the DIRECTV acquisition, states: "A fixed [wireless local loop] service requires substantial upfront investments. AT&T must install additional antennas and other equipment at each cell site in areas it seeks to serve." (paragraph 50).

There's no mention of needing more cell sites.

Modifying existing cell sites is one
thing. Acquiring, permitting, and building new sites is completely different.
As one El Dorado County supervisor says, new sites are very controversial
because of their "aesthetic downside." Aside from having to find
property and possibly develop it, there's the sticky problem of getting
community buy-in for more cell towers.

If AT&T plans to use a higher frequency like 2,300 MHz for wireless, they will need to add cell sites. That takes time. Finding additional sites, much less building new ones, will have a significant impact on AT&T's ability to deploy wireless local loop services in a timely manner. It also puts more pressure on local governments to expedite site approvals.

How many new sites are needed? To
get an idea, here is AT&T's presentation to the El Dorado County
Board of Supervisors. AT&T estimates they'll need to modify 4 cell towers
and add 34 new locations -- either collocated on existing buildings or newly
constructed cell sites. 38 locations seems insufficient to serve all of El
Dorado County’s high cost-eligible CAF II households, so maybe this is the
first phase in AT&T’s five-year rollout.

2. Frequency Matters

There are solid engineering
arguments why 700 MHz is more valuable than 2,300 MHz for delivering wireless
broadband to homes. I’ll address only the line of sight argument. The larger-wavelength
700 MHz frequency penetrates buildings and foliage better than the shorter-wavelength
2,300 MHz frequency. For 2,300 MHz service to reliably deliver broadband, there
needs to be line of sight, meaning you can physically see, without obstruction,
between cell tower antenna and customer antenna. Without line of sight, service
will be less reliable than with 700 MHz. Many parts of rural California are mountainous and have trees, which makes line of sight broadband difficult and expensive to deploy.

Home in El Dorado County with Trees
Obstructing Line of Sight

Diagram showing line of sight wireless broadband. No obstructions allowed.

Diagram showing non-line of sight
wireless broadband, e.g. 700 MHz

Credit: L-Com Connectivity, www.L-com.com

3. Is Wireless Local Loop Service Viable Without Video?

AT&T’s Form 425 testimony
mentions the importance to their revenue stream of bundling broadband (through
WLL, U-Verse, or their fiber product, GigaPower) with video (through DIRECTV),
voice, and even mobile. As John Stankey stated, capacity constraints reduce the
prospects of WLL as a viable standalone service. Put another way, WLL might be
financially viable only if a subscriber opts to subscribe to video and/or
mobile as well.

The prospect of adding new satellite and cable television
subscriptions is dubious in today’s marketplace. One study
claims that more than 40% of satellite and cable subscribers are planning on cutting
back or dropping their pay TV service. Broadband data from the CPUC's 2013 annual report on the Digital Infrastructure and Video Competition Actindicates that California's state video franchise holders now provide more broadband service than video service.

That doesn’t bode well for AT&T’s
DIRECTV subscribers, some of whom may be switching to over-the-top services
like Netflix or Amazon Prime and require only a reliable internet connection to
watch shows. To underscore that point, AT&T's Form 425 filing for the DIRECTV acquisition includes an economic assessment, which states "higher Internet access speeds will erode the traditional advantage that DIRECTV's satellite network has given it" over video on demand providers like Netflix. (page 54)

More on AT&T’s proposed WLL
broadband service (from the El Dorado County video):

Broadband speeds must meet or exceed
10 megabits per second down and 1 up

Latency should not exceed 100
milliseconds

Initial minimum usage allowance of
150 gigabytes per month

Monthly rates must be comparable to
fixed wireline urban rates for the same level of service in urban areas or at
or below the FCC Annual National Rate Benchmark (approx. $72/month)

Our testers are nearly finished with the 10th round of statewide field testing with 82% complete as of last Friday. Here's a map of the completed locations in red, and those still needing to be tested in blue:

Friday, October 28, 2016

Last weekend, River City Rowing Club teammates and I competed in the
Senior Master Four race at the annual Head of the Charles Regatta in
Boston. While there, I ran several speed tests at the AirBnB where we
stayed -- both on the WiFi provided and on AT&T's LTE network. Here
are the results:

Head of the Charles racecourse. Test location shown in red point below.

Wireless Loops As Copper Substitute
There have been reports that California's largest local exchange carrier and Connect America Fund Phase II recipient, AT&T California (maps of California's CAF II areas), plans to use wireless loops to deliver broadband to high cost-eligible households -- in other words, use mobile (i.e. "wireless loops") as a substitute for copper telephone lines to offer speeds of 10 megabits per second down and 1 megabit per second up or faster (10/1). How the FCC measures AT&T's progress in deploying wireless 10/1 could greatly influence the level of investment AT&T needs to make. This post analyzes several scenarios where AT&T uses mobile LTE spectrum to deliver wireless loops. Read Part 2 for corrections and comments to this analysis.
(photo credit: Owen Rochte)

Background
Last year, AT&T, Verizon, Frontier, and Consolidated agreed to receive from the FCC, collectively, $105 million annually
for five years to upgrade California homes so they can get at least a 10/1 connection to the internet. The federal program that pays for this is
called Connect America Fund Phase II (CAF II), and it is funded by
surcharges on our telephone bills.

Fiber Optic Reality Check
As we enter the last
quarter of 2016, it's time for a reality check. For those of you living
in CAF II high cost areas (primarily rural California) who've been
holding out hope for fiber optic broadband access to your home, you may
be waiting in vain. That's because CAF II doesn't pay enough for
fiber to the home, and carriers are not obligated to build fiber to the
home. Furthermore, once the CPUC's California Advanced Services Fund runs out
of money, there's arguably no funding mechanism, either federal or
state, to bring fiber to rural California homes.

Can Mobile Deliver?
If engineered as a wireline substitute, mobile can deliver 10 megabits per second downstream and 1 megabit per second upstream (10/1) to the home. The question is what will be used as proof that AT&T's 141,000 CAF II-eligible homes have been upgraded to receive that level of service? We know from mobile field tests that mobile speeds are highly variable from one moment to the next, so in order to deliver a reliable 10/1 service, AT&T may need to augment their network to deliver considerably faster speeds on average. Refer to the CPUC's comments to the FCC on GN Docket No. 15-191 regarding the highly variable nature of mobile broadband and the importance of accounting for reliability when measuring broadband speeds.

The "LTE" Indicator On Your Phone Doesn't Mean Fast Speeds
The FCC has made available to the public some of AT&T's mobile broadband deployment data. Unfortunately, speed data are missing. What remains is a file showing areas with access to LTE (4G "Long Term Evolution") service, as well as other, lower levels of service. We know from experience that mobile coverage shrinks as speeds increase, so it's impossible to tell how large a 10/1 service area extends across California.

Three CAF II Wireless Loop ScenariosIf AT&T is going to
use LTE wireless loops (i.e. home broadband access over the mobile network),
here are three initial takes on how many CAF II-eligible households are
theoretically covered today using various data sets. Based on these scenarios,
AT&T has to make either very little investment to offer 10/1 service to
their CAF II high cost households, or they have to make a significant
investment. The answer hinges on what validation method the FCC uses to ensure AT&T has met its CAF II obligations to offer 10/1 or faster.

Scenario 1: FCC 477 Deployment Data - Using the December 2015 deployment data AT&T filed with the FCC, 85% of AT&T's CAF II high cost households have LTE coverage. Note that "LTE" is not the same as 10/1 or faster. However, if the FCC were to use 477 data as proof of meeting CAF II obligations, AT&T could conceivably claim subsidies for most of the 141,000 high cost households without doing any network upgrades.

Scenario 2: Interpolated Average Speed Data - Based on interpolated average speeds from Spring 2016 mobile field testing, over half of AT&T's CAF II high cost households appear to have access to speeds of 10/1 or faster.

Scenario 3: Interpolated Mean Minus 2 Standard Deviations - Using this stricter standard, where we reduce the average speed at each of the 1,990 field test locations by two standard deviations and create an interpolated surface using those adjusted speeds, nearly all of the 141,000 households appear to remain CAF II eligible. This suggests that AT&T would need to make a significant investment in their mobile network to deliver a reliable 10/1 or faster service to their high cost households.

Conclusion
Of these three scenarios, the answer to how much additional coverage is necessary is likely to be somewhere between scenarios 2 and 3, because scenario 1 is does not rely on a speed threshold, but rather simply the presence of LTE service. "LTE" offers a wide range of speeds, including zero kilobits per second. For the sake of reliability, the FCC should implement a robust mobile measurement program, such as CalSPEED, to verify AT&T has met its CAF II obligations using wireless loops.

Wednesday, October 19, 2016

This week, we were treated to an amazing photo by tester Steve Crews of a roadrunner with mountains in the background. The team finished Week 2 having completed 46% of the 1,990 locations (see map image below, completed location shown in red).

Average speeds shown in table below are in kilobits per second (Kbps), so divide by 1,000 to get megabits per second (Mbps).

Friday, October 14, 2016

As I've said in earlier posts, back when we launched CalSPEED in 2013, we didn't restrict distribution to only the United States. We added the restriction because we didn't want people running tests all over the world. CalSPEED is designed to be used by California residents so they can understand how their mobile provider is performing. Nevertheless, we occasionally still see tests appearing in unexpected parts of the world.

At the end of 2014, there were three tests performed at geographic coordinates that appear to be in the City of Homs, Syria. A quick look at those locations in Google Earth reveals the destruction of that city from years of war. See before-and-after photos in The Guardian here.

A New York Times' article on the destruction of another Syrian city, Aleppo, estimates the number of displaced people around the world
is equivalent in number to the entire population of the United Kingdom or
France. It would seem that number includes all displaced people, not just Syrians, but these images of Syria, even from Google Earth, are devastating.

Monday, October 10, 2016

I am constantly amazed at the geographic diversity of California, and the photos coming in from our field testers are proof. Their photos are wonderful. I'm adding a few of them from Week 1, but feel free to follow their progress on Instagram at cpuc_broadband_testing.

Featured Post

About Me

Beginning in May 2012, the California Public Utilities Commission began testing mobile broadband performance of the four major providers and reporting on the results. The CPUC tests 1,990 locations twice a year. Tests include throughput (TCP), latency, UDP packet loss, and jitter. With these metrics, we are able to estimate network quality for streaming voice (VoIP), streaming video, and interactive video. The results are available on the CPUC interactive broadband map as well as on the CPUC's website. We also offer both iPhone and Android speed test apps for the public to do crowdsourced testing. Search for "CalSPEED" on iTunes and Google Play. The results are uploaded the California Interactive Broadband map every 24 hours.