Can We Validate the Cost of Connectivity?

The latest addition of the New America Foundation’s “Cost of Connectivity” report is successfully gaining attention from bloggers and journalists who accept its claims at face value. Some of the stories about the report have appeared in major publications, and nearly all the sources that generally deal with Internet policy have covered it:

So here we have ten stories that accept the NAF’s claims without the slightest bit of skepticism or critical examination. Each of them repeats the tale that American broadband is slower than broadband in the rest of the world without any need to verify or validate it by examining additional data or asking hard questions. The closest any of these stories gets to asking questions is the piece in the New York Times by Claire Cain Miller who at least asked one critic (me) for an alternate point of view, even though she used the smallest possible part of my interview.

Another study earlier this year found that the United States had an average connection speed of 10.5 Mbps.

I’ve noted that reporters don’t understand what the Akamai State of the Internet report measures, and here we see that claim in action. New America surveyed some ads and ranked nations according to the speeds offered in a very select group of ads in 24 cities and towns. Akamai’s “average connection speed” measures download speeds seen within complex interactions between web browsers and servers that aren’t representative of network speeds; “average peak connection speed” is more appropriate for network measurement. This is explained in Akamai’s reports:

So you wouldn’t compare Akamai’s average connection speed to the advertised rate of a broadband connection if you had read the Akamai report; that’s apples and oranges. You also wouldn’t compare the highest speed offered in an advertising survey to the performance of the connections that people actually buy; very few people buy the highest speed available. But what would you find if you compared the Akamai average connection speeds of all the countries New America surveyed?

It turns out you can’t do that because NAF doesn’t survey whole countries, only carefully selected international cities and US cities and towns. Despite the narrow basis of their survey, they’re quite happy making sweeping generalization about whole nations, however; the comparisons between San Francisco, Bristol, VA and Chattanooga, TN with Seoul and Hong Kong are offered up as indictments of American broadband.

It’s difficult to compare Chattanooga with Hong Kong, and it’s even more difficult to compare the entire US to specific nations in Europe and Asia that are scarcely larger than America’s San Bernardino County, California. So to make an apples-to-apples comparison between American broadband speeds and those in the rest of the world using Akamai’s data, let’s compare US states with the overseas nations and administrative regions (Hong Kong is part of China, not a nation) and see where the chips fall. When you do that, you get this:

This isn’t very shocking, is it? Five of the 10 fastest regions are American states, six of the 11 fastest regions are states, and 10 of the 17 fastest regions are American states. I’m not inventing data here, I’m simply extracting data from a report cited by two the analysts who accept the New America’s work as perfectly valid.

I’m happy for analysts to read and cite Akamai, I just wish they’d do it in a more consistent and thorough way. And yes, it is true that Akamai’s measurement of average connection speed for the entire US is only 11.4 Mbps in the most recent report. That puts us in 14th place, between Norway and Belgium, on a global basis. Given that Akamai measures 200 nations and regions, most of which are smaller than the US and some of which are richer, 14th place is not too shabby; we’ve been as low as 22nd in the past.

The point I’m making is simple: you can’t judge a nation’s Internet service on the basis of an advertising survey, and both New America and the journalists who uncritically accept their claims are being less than forthright when they claim otherwise.

The second claim that New America makes concerns prices, and I’ll deal with that in a subsequent post. Hint: they’re misleading on this point as well, but in a more subtle way.