There is much to be learned from the growing body of international comparisons, including the Berkman Center Broadband Report. Unfortunately, for Canada, many will put too much emphasis on a few select rankings from the Report without taking the time to consider how these were derived.

The Berkman Center relied extensively on OECD data for a number of its comparisons. It used OECD data on broadband penetration per 100 population as a key input. However, as noted in Lagging or Leading, this measure is biased against countries such as Canada. The Berkman Center Broadband Report also included penetration per household but relied on older OECD data where Canada ranked 8th even though more recent international comparisons where Canada ranked as high as 5th (see Lagging or Leading, Table 4.2.2).

The Berkman Center incorporated into its ranking data on the penetration of 3G mobile services. Canada was reported to have a very low level of 3G subscribers per 100 population – less than 10%. There are two concerns with this data point. First, Canada and the United States do not compare well on measures of mobile penetration per 100 population because subscribers in these countries generally do not have multiple accounts (or SIMs). The Berkman Center Broadband Report acknowledged this difference yet did not make any adjustments. Second, Canada’s very low ranking on 3G penetration is not consistent with the fact that the service is available to 91% of the population, as noted in the CRTC Communications Monitoring Report. Since the Berkman Center’s ranking assigns a 30% weight to this indicator, Canada’s overall ranking on penetration is pulled down.

The Berkman Center next turned to measures of broadband service speeds where OECD data once again played a significant role, accounting for one-half of the country rankings for this indicator. As noted previously on this website, the OECD rankings on speeds are not reliable. In the case of Canada, the OECD relied on only 16 observations and, among those, significantly under represented higher service levels and over represented lower service levels. The OECD relied on many more observations for most other countries, with some countries' lists including duplicate services for these higher speeds.

The Berkman Center made some effort to address the multiple and duplicate offerings for its analysis of service prices among the 30 countries. It supplemented the OECD observations with its own analysis of advertised service prices. Some of these included higher speed services available in Canada, as noted in Figure 4.15 of its Broadband Report. Yet, none of these observations were included for the speed comparisons.

The other components of the Berkman Center Broadband Report rankings on speed were drawn from results of user-generated tests of achieved throughput speeds and latency conducted using Speedtest.net. It was acknowledged that this data has “several confounding factors that [require] we interpret the data with caution,” and the results on latency “produced very counterintuitive results”. Yet, the resulting rankings from this data accounted for the other half of the countries’ rankings on speed. Concerns respecting the use of data from Speedtest.net were discussed previously on this website.

Prices composed the third aspect of the Berkman Center Broadband Report on international comparisons. The OECD data on prices figured prominently here as well, accounting for one-half of the overall price ranking. As noted, the Berkman Center took additional measures to update and expand the OECD pricing data to derive new rankings based on the price across different levels of service. In each case, Canada’s ranking improved by four to six places.

While the steps taken to improve the pricing data were helpful, it is curious why the Berkman Center decided to create its overall ranking on price based on both the OECD price observations and a combination of the OECD and its own updated and expanded data. The latter would, for the most part, encompass the former observations so there is no reason to use both. In the case of Canada, using both sets of price rankings pulls Canada down in the rankings.

The other significant factor affecting Canada is the exclusion of very high speed services (download speeds of at least 35 Mbps), notwithstanding the fact that Figure 4.15 of the Berkman Center Broadband Report clearly indicates that it had price data from Canada for such services. Because this data was ignored, Canada was assigned a nominal ranking of 30 out of 30 for this category. And because the OECD also failed to include any such services from Canada, this very poor ranking was double-counted.

The Berkman Center Broadband Report does not do justice to Canada’s broadband performance because of these missteps. Compounding this is the Report’s heavy reliance on OECD data that have serious methodological problems and biases that weigh against Canada. Since the Report is only in draft form and is being review by the Federal Communications Commission, perhaps some of these concerns will be addressed.

A report released today by Mark H. Goldberg & Associates and Giganomics Consulting Inc. tackles the issue of the state of Canada’s broadband infrastructure. A copy of the report is available through this link.

This extensive report (tipping the scales at 100+ pages) provides a wealth of information on the country’s broadband infrastructure, how it developed and where it stands relative to other countries.

Highlights of the numerous statistics include:

Some 70% of Canadian households have adopted broadband services, an 8th place ranking among OECD countries;

94% of Canadian households can access broadband services based on wireline – with a high degree of overlap between DSL and cable facilities, while virtually all households can subscribe to wireless (e.g., satellite) broadband;

Canada has among the least expensive broadband entry-level service offers – second only to the United States according to the International Telecommunications Union (ITU);

The proportion of Canada’s internet connections of 5 Mbps or greater reached 27% in the second quarter of 2009, according to Akamai Technologies Inc.; putting Canada in 10th place among the more than 200 countries studied;

Canada achieved 7th place on the LECG/Nokia Siemens Networks Connectivity Scorecard and 9th place on the Economist/IBM E-Readiness index, measures that combine multiple indicators of internet infrastructure and use.

Frequently cited OECD results on the relative price and speed of broadband services are not reliable because of methodological problems discussed previously on this site and as noted in the report.

The report also notes:

"To be constructive in our own analysis, we have focused on the broader issues rather than any particular piece of research. In that regard, to help simplify the overwhelming amount of research that is available, we want to focus on answering a few simple, yet key questions:

1. Is broadband available to Canadians? If not, why not?

2. If it is available, do Canadians adopt the technology? If not, why not?

International comparisons can help us understand whether we lag or lead in terms of availability or adoption, but it is these questions that should be the focus of debate."

This new report provides a strong foundation on which to move the debate forward.

The Said Business School at Oxford University, with the support of Cisco Networks announced the findings of its second annual report on Broadband Quality Score (BQS). This has some interesting results but the published findings lack in supporting details on sources and methodology.

Key inputs to each country’s BQS are the download and upload speeds achieved by internet users. The download speed accounts for 55% of the score, while the upload accounts for 23% and latency accounts for the rest. Scores for 66 countries were compiled based on results from individual users who applied the Speedtest.net online speed test. The Said/Oxford report highlighted the fact that the results were based on 24 million tests between May and July 2009.

While this seems like a lot, it is only a fraction of the more than 440 million broadband subscribers worldwide, according to Point-Topic. [free with registration] The Said/Oxford report doesn't indicate how many of the 24 million test results were repeat versus unique users, or how they were distributed across countries, time of day, distance to server test site, and other variables that could affect the test.

The BQS results reported for a country will also differ depending on several aspects of the individuals participating in the Speedtest.net site during the sample period. The report does not indicate what, if any, steps were taken to normalize the results for these differences.

According to the report, the results of the tests of download speeds among Canadian users indicated an average speed of just under 5 Mbps. When I ran the test (today and earlier in July), the test results indicated download speeds almost double that. The test results would be different for users who have different service levels, chose a different test server site, time of day, and so on. If the speed test results used in the Said/Oxford analysis were taken mainly from users who were encountering congestion or dissatisfied with their speeds, then the results would tend to be biased downwards.

A more objective check on the results of Said/Oxford report can be found in another publicly available source of information on broadband service download published by Akamai Technologies Inc. Each quarter Akamai’s State of the Internet includes findings on the average download speeds achieved by users in dozens of countries. The results are based on data that Akamai collects from its globally distributed network of servers with connections to more than 400 million unique IP addresses.

Both sources put the same three countries – South Korea, Japan, Sweden - at or near the top. However, the results for many other countries were quite different. For example, Akamai’s report for the second quarter of 2009 found Hong Kong had an average achieved download speed of almost 7 Mbps but the report from the Said/Oxford report put this country’s speed at just under 5. The Netherlands did well according to Said/Oxford at about 12 Mbps but according to Akamai the speed was only 5.1 Mbps. Denmark was similarly boosted according to the Said/Oxford results. These examples demonstrate the dramatic differences that can occur when service speeds are based on different methodologies. In the case of the Said/Oxford report, the results are based on user-selected testing where the selection of participants was not random or adjusted to be representative of the universe. Compare this to results from Akamai's data which encompassed almost 425 million of the 440 million unique IP addresses worldwide.

As a footnote to the above, Akamai’s report for the second quarter of 2009 provides some positive news for Canada in terms of higher speed broadband service. The percentage of connections achieving speeds of 5 Mbps or greater has reached 27%, representing a 50% increase over the previous year and moving the country into 10th place overall, as shown in the following table from that report.

Once again, the OECD’s prices for broadband internet services have been cited in the press with little regard to how these prices were measured.

The Globe and Mail has an article under the header “Canada: Land of high-priced web access” which states that the price of high-speed internet access in Canada is $87.32 per month – second most expensive among the 30 OECD countries.

International price comparisons can be informative, but in this case, they are highly misleading. The price for high-speed internet service in Canada is more affordable than the OECD data indicates.

The chart shown in the article was taken from the OECD’s Communications Outlook report for 2009. Specifically, Figure 7.18 which is a listing of high-speed connections offering advertised download speeds of between 12 and 32 Mbps.

As noted on this website in the observation for August 20, 2009, the OECD’s methodology for calculating the price of broadband internet services does not provide reliable results for Canada. The OECD’s reported average prices for Canada are based on only a few price points for services and calculated a simple average without weighting the services according to actual subscription levels.

In the case of the quoted $87.32 for high-speed, the OECD relied on just two service prices: Bell’s Total Internet Max and Shaw’s Nitro. The price information was gathered more than a year ago and does not reflect current prices nor the range of services in the Canadian market that offer speeds of between 12 and 32 Mbps.

The current information on prices for services with those speeds is shown in the chart below, after converting these to U.S. purchasing power parity values as done by the OECD. The most recent values are available from the OECD.

The average price for service is $50.51, based on the advertised offers of the six major service providers in Canada with services in the range of 12 to 32 Mbps downloads. This is a simple average of six offers since the companies do not publish the subscription levels for these specific classes of high-speed internet services. If each company’s price was weighted based on its share of the six companies' internet subscribers, the average price would only increase by 4%.

Compared to the other OECD countries, an average price of $50 would rank Canada in 10th place, not 29th as the OECD chart suggests, for the lowest priced internet service capable of providing 12 to 32 Mbps.

Earlier this spring, the OECD released some statistics comparing 30 countries on several indicators of broadband internet performance. In addition to the oft-cited broadband per 100 persons statistic, in which Canada placed 10th, the OECD indicated that Canada's average price of a broadband service per megabyte (Mbps) was $26.11 (converted to U.S. purchasing power parity), making it the third most expensive. That price seemed high to some observers but, because of the source, was cited by several as an indication that the Canadian internet market is failing consumers.

On August 11, the OECD released its Communications Outlook for 2009. Table 7.14 of this publication includes the underlying data used to derive the $26.11 per Mbps price of broadband in Canada. How this was calculated was actually quite simple, and perhaps a bit too simple. It is based on the advertised price and speed of 16 services offered by three ISPs in Canada. The price was divided by the speed for each service and then a simple unweighted average was calculated. A 'Lite' service that costs about $25 to $30 per month and provides speeds of between 256 kbps and 1 Mbps works out to a price per Mbps of $50 to $100. (All dollar amounts in US PPP.) These types of services were given the same weight as the services that cost $40 to $50 per month and deliver speeds of 5 to 10 Mbps, which equates to a price per Mbps of $5 or $6. Also included in the 16 offers considered for Canada were three Wimax services with a calculated price per Mbps of $17 to $74. With all 16 offers given the same weight, the average price per Mbps equates to $26.11.

Statistics published by the CRTC in its Communications Monitoring Report 2009 (Table 5.3.3) indicate that almost 60% of residential broadband users subscribe to services that provide speeds of at least 5 Mbps, and fewer than one-quarter rely on 'Lite' services. A simple average of 16 offers does not correspond to these usage levels.

Canada's apparently poor performance as third most expensive among OECD countries is an artifact of this approach that appears to underrepesent the higher service levels and over represent the lower service levels, resulting in a skewed average price per Mbps.

Conversely, the OECD's analysis for most other countries included two or three times as many advertised offers. In a number of cases, the listed offers are heavily weighted to higher speeds, with some countries' lists including duplicate services for these higher speeds. The list for France includes two identical offers of "La fibre" service from Orange offering 100 Mbps that are priced the same. The list for Australia includes multiple offers from Bigpond with speeds of 20 and 30 Mbps.

Because the OECD's methodology did not apply any weights to the listed offers, a country's ranking in price per Mbps was determined to a large degree by how many and which offers were included in the calculations.

If the OECD's data for Canada had included more of the higher-speed services, the average price per Mbps would have been much lower. For example, Videotron's Ultimate Internet Speed 50 is advertised at $79.95 per month ($89.95 without a contract) and provides up to 50 Mbps, for an average price per Mbps of $1.60 (Canadian $) or about $1.32 in US PPP. There is a growing number of broadband internet service offers in Canada that have a price per Mbps of $5 or less (US PPP).

The CRTC's Communications Monitoring Report's Table 5.3.3 indicates the average revenue per residential internet subscriber was $37.44 (Canadian $) and the average downstream speed was 4.9 Mbps, based on actual reported subscription levels. A weighted average price per Mbps using this data would work out to $6.28 (US PPP), less than one-quarter the price derived by the OECD.

At this price per Mbps, Canada would rank among the ten least expensive countries in terms of price per Mbps. However, this ranking cannot be confirmed since the price per Mbps of the other countries reported to have lower-priced services is still based on simple averages that may not reflect the subscription levels in each country. In other words, GIGO (garbage in-garbage out).