technorati

Is Clearwire a Net Neutrality Violator? Does It Matter?

Reader Bill reports that Wireless ISP Clearwire blocked ports in the service provided to him in a way which prevented him from running audio and video chat applications.Bill was commenting on my post last week which included the Federal Trade Commission claim that only one small case of this type of anticompetitive behavior had been documented.I should have done a little more research – at least on Google – before repeating this claim uncritically.

Clearwire is a Wireless ISP offering a pre-WiMAX service in selected markets in the US and several other countries. The company was founded by Craig McCaw of Cellular One fame and claims 258,000 customers as of March 2007; it offers its own VoIP over its network connections. Turns out Clearwire was involved in a port-blocking flap with Vonage back in 2005.

First, here’s part of what Bill has to say:

“My service provider (Clearwire) purposely blocks specific ports that had prevented me from using my Internet connection for legitimate use, audio and video chat. My troubleshooting did not suspect this at first since, according to everything I read, no companies are engaging in anti-competitive behavior. The first place I found out about this is on the Internet where some unhappy customers’ discuss the specific ports Clearwire blocks. It turns out; those were the ports the iChat uses for audio/video chat. I called the company to ask them to fix this and they acted like there was nothing that could be done and that these "VoIP" ports (as per the CSR) were always blocked by Clearwire. I told them they were violating the good faith of our contract by preventing me from using my Internet connection for legitimate purposes and threatened to drop them. It was then that they escalated my claim (to Las Vegas) and opened those ports up for me within a couple days….”

In 2005, according to a story on CRN, a Vonage customer reported that he couldn’t use that service over a Clearwire connection and was told by a Clearwire customer service rep that he couldn’t use Vonage over Clearwire. Clearwire EVP Gerry Salemme is quoted as saying “Applications like Vonage's VoIP could hurt the other customers [on the Clearwire network], bring down the other customers.” I don’t know the context of this statement but, on its own, it’s technical nonsense. VoIP is hardly a bandwidth hungry application.

Vonage reported fixing the problem by changing the port number it uses for connections. [for my non-nerd readers, port numbers are arbitrary numbers included in IP packets which are used to say which application running on your computer or a web server should look at which packet. Given their purpose, they can easily be used to block packets destined for certain applications. Given their arbitrariness, they can easily be changed to defeat the blocking.]

The FCC didn’t get involved in the Clearwire case the way it did with Madison River, a rural telco that blocked VoIP, presumably because Clearwire is not a phone company subject to regulation (although more lately the FCC seems to feel that VoIP providers should at least be subject to the some of the obligations but not the rights of phone companies).The FTC did not cite this incident or current Clearwire incidents in its report on net neutrality.

I’ve claimed that a competitive market for Internet connectivity would make net neutrality a non-issue. This seems to be the case in countries like the UK where there is sufficient competition to allow consumer choice to be the regulator. Clearwire, with an interesting service, which I’ll write up later is the kind of competitor to the entrenched telco/cableco duopoly that we need to have emerge. But they are reportedly blocking – and in a way which advantages their own VoIP service.Given a choice I wouldn’t buy their service because of this policy.

Should the wrath of the FTC or the FCC descend on them?No, I don’t think so; they are not a huge monopoly; they serve (according to their FAQ and their map) largely places where there are DSL and cable alternatives.Their customers have a choice.The only regulation they need to be subject to is truth-in-advertising. They should be clear abut what they block and what they don’t – even for their own sake. It is quite possible that their CSRs are misquoting policy to callers.

Users like Bill who make their experience public are an invaluable part of any competitive landscape. As competition in broadband Internet access emerges (I’m an optimist), we all have a role in helping to keep it honest (which doesn’t obviate the need for truth-in-advertising).

BTW, here’s an interesting post by David Isenberg – no more a fan of Verizon than I am – favorably comparing Verizon’s terms of service to those of Clearwire. Competition cuts both ways.

Hope for Internet Quality (and Neutrality) Measurement

Fellow My Way blogger Eric Hernaez recently posted the important news that Cisco and Brix are proposing an extension to the SIP standard which allows devices to be put in a loopback mode in order to measure the quality of their connection to an Internet service.Standards are usually pretty dull stuff but this proposal could be a very big deal – even if the IETF, to which it was submitted, takes its usual forever to adopt it.

Although Cisco and Brix propose the extension in order to give VoIP, real time text, and video over IP providers a way to diagnose customer problems, the potential is much greater than just these very worthwhile applications.This technology, if widely deployed, can be the answer to two vexing problems: 1) how do consumers cut through the hype to compare the quality of broadband providers; and 2) how do we know if ISPs are violating the principals of network neutrality (these principals are not enshrined in any law or regulation), perhaps to advantage their own applications over those provided by competitors.

As Eric points out:

“The Cisco collaboration is a major coup for Brix. With the industry leader adopting this method of link testing, it will become a de facto standard whatever happens in the IETF. I would be surprised if other ATA vendors [nb. besides Cisco] did not quickly fall in line. Look for softclients and web-based tools to implement the draft in the near future.”

The devices which implement SIP are in almost every home or office which uses VoIP.SIP is built into routers which have VoIP capability and into cheap little VoIP boxes called ATAs (what connects your phone to the Internet if you have a service like Vonage).SIP devices are literally everywhere and spreading.SIP is also implemented as software in many IP-based VoIP applications.A new generation of mobile phones uses SIP when connected to a WiFi hotspot.

“Determining the speeds consumers are actually getting is tough to measure. Cable speeds can vary if many people in one neighborhood are online at the same time, like after dinner. Access over phone lines can be slower if the customer is far from the switching office, where the Internet signal originates.”

The article also says:

“In many cases, consumer advocates and industry analysts said, customers do not get the maximum promised speed, or anywhere near it, from their cable and digital subscriber line connections. Instead, the phrase ‘up to’ refers to speeds attainable under ideal conditions, like when a D.S.L. user is near the phone company’s central switching office.”

Both of these assertions are true.There are many other factors which also affect the bandwidth (aka speed), latency, and jitter experienced by a particular user at a particular place.Up until now (despite the valiant efforts of some online connection test sites) it’s been impractical to gather enough comparable data to assign grades to the various providers of broadband service.

OK.Fast forward.Now we’ve got millions of devices distributed throughout the country and the world which can cooperate in automated measurement of bandwidth, latency, and jitter between themselves and some host service somewhere. Let’s assume that the owners of these devices (you and me and all our friends) consent to have a trusted organization put the devices in feedback mode every once in a while for a few seconds for the purpose of gathering some data.

The cure for lots of variables is lots of data points.What is almost impossible to judge anecdotally becomes very tractable to measurement if it’s measured enough times in enough places.“Individual performance may vary” but it will become very easy to know which broadband providers perform how well where and doing what.It’ll be much easier to know if you want to buy your broadband access from the cableco, telco, or the new WISP (wireless ISP) down the street.When someone advertises “speeds up to..”, it’ll be possible to know how often that speed or 95% of it is actually obtained.With enough data we can distinguish between websites which are slow to respond and networks which are slow to deliver the response.Most packets traverse multiple networks; millions of triangulations on packet delivery performed by millions of devices should show which networks slow packets down or lose them and which speed them on their way.

I proposed a citizen journalism project which started with the development of software which could be run by millions of volunteers to determine both network performance and possible violations of Internet Neutrality.Maybe software developed to this new SIP extension fills the bill.I’ll be interested to learn more.

If it does do all I hope and if this capability ends up standard in routers and ATAs, it solves at least part of the development problem and much of the distribution problem for the citizen journalism exercise.For technical reasons, it is much better to run the monitoring software on a router connected directly to the broadband provider’s modem than on a PC – especially if that PC has a WiFi connection.

SIP updates can often be accomplished by a firmware download so it’s conceivable that this capability can be added to existing routers and ATAs in a routine maintenance release or upgrade (note:I haven’t done the technical work to know whether there may be some obstacle to this).It’s also possible that SIP capability will be added to non-VoIP routers and devices specifically to enable this loopback capability.

We may be able to compare networks AND look for non-neutral network connections sooner than I thought.

The Wall Street Journal Is Wrong: The US Doesn’t Have Competitive Telecom

“How much competition is there in U.S. telecommunications? So much that even California regulators have finally noticed. Last week the state's Public Utilities Commission voted 5-0 to lift decades-old price controls on land-line phone companies.”

I don’t think we need to regulate telephone rates but the telecommunications marketplace in the United States is becoming steadily less competitive.at&t CEO Ed Whitacre is well on the way to reconstituting the old AT&T whose demise was supposed to mean the end of monopoly.This time the monolith includes now crucial Internet and wireless services which didn’t exist in the 1980s.

Further exhibiting its cluelessness, The Journal (with which I usually agree) goes on to opine: “The good news is that these state regulators have finally acknowledged that … holding residential phone bills below actual cost is silly in today's brave new telecom world.” Apparently The Journal thinks that the price of providing telephone service has gone up in the last several decades.In fact it has plummeted to a tiny fraction of what it used to be along with all other data services.

Moreover, the value to the owner of the last mile connection to subscriber’s residence is much more than just the revenue from outgoing calls and the monthly line rental fee.Local phone companies get paid for all incoming “long distance” minutes to a phone ANDget the opportunity to sell enhanced services and DSL on a favored basis.Hell, they even get the chance to charge you for NOT listing your number.

Unfortunately more indicative of the state of telecom competition in the US is the lead from this story today in the New York Times:

“When the government’s multibillion-dollar auction of radio spectrum licenses began two weeks ago, it looked as if newcomers might get the chance to buy their way into the mobile phone business, leading to more choices for consumers.

“But now the country’s biggest cellular providers appear poised to win many of the 1,122 licenses up for auction, allowing them to expand their reach and reducing the chance that a new entrant might bring down prices.”

And who owns these cellular providers which have succeeded in “reducing the chance that a new entrant might bring down prices”? Traditional telephone companies is the answer. Hmm…

The greatest threat from a new entrant is that it would use these frequencies to provide voice as just one more data service just as Skype does with Internet-based phone service.That would collapse the margins in wireless which is where phone companies now get a huge share of their profits.

It is true that cable is providing some competition to traditional telcos in both voice and data service. This is a good thing and may be why we don’t need price regulation.But a duopoly does not mean we get all the benefits we could expect were there true competition.

Here are some signs of insufficient competition in the United States:

The relative slow speed and high price of broadband here as compared to most other developed countries.

A regulator (the FCC) which seems more intent on using regulation to slow competition (antiVoIP rulings, for example) than to enable it.

The reconstitution of AT&T.Latest step underway is the proposed at&t acquisition of BellSouth.Following the SBC acquisition of Ma Bell herself.

The monopoly on lobbying that the former “baby bells” have now that their lobbyists are no longer opposed by lobbyists from AT&T and MCI (who were also very good).

The arrogance of network providers in saying they should be paid twice for use of the Internet – once for connections (fair enough) and again for use of those connections (as in charging Google not just for its connection but for sending us stuff over its connection and ours).There would be no need to worry about enforcing net neutrality against such nonsense IF there were true competition in broadband in the US.Net Neutrality is NOT an issue in the UK, for example, where real competition and deregulation have made the issue moot.

Reader Comment on Net Neutrality (and My Rebuttal)

Reader Richard Bennett, whose own blog often articulates the telco point of view on Internet Neutrality more coherently than most telcos, posted a challenging comment to my post announcing that I’m speaking at the Berkman Center tomorrow.His questions deserve answers which I’ll give in this post and also tomorrow when I speak.Richard’s whole comment is quoted below but I’ve taken the liberty of interspersing my answers.

Richard: “You say: I very much believe we need an Internet which remains essentially neutral and application agnostic.

“Perhaps you could explain why you believe that and what it means to you. For example,

“* Do you believe that all applications have the same requirements from the network?”

Answer: The reason we need an Internet which is network application agnostic is because only an application-unaware network allows the kind of incredible innovation which has occurred on the Internet.David Isenberg (who Richard mislabels on his blog as a marketing person – David’s background is Bell Labs) explains why this is so brilliantly in The Rise of The Stupid Network).

My simplistic explanation is that a network which is built to be aware of specific applications may work well for those applications that the network designers had in mind but will inhibit if not absolutely prevent applications the network designers didn’t think of as well as the evolution of existing applications.The traditional phone network is a great example of that.Handsets on that network don’t even have the capability of cell phones because the network is optimized for voice and voice alone.I posted more on this here.

To answer the starred question: I KNOW that different applications rely on the network in different ways.We get to the implications of this in Richard’s next question.

Richard: “* Do you believe that applications that require tons of bandwidth, such as movie downloads, should have the same service from the network in terms of average packet delay as applications such as VoIP with modest bandwidth requirements but more stringent jitter requirements? (note: "jitter" is the variation in packet delay.)”

Answer: Yes, I do.I do know what jitter is since I founded and ran the world’s largest wholesale carrier of international VoIP and am a nerd but I do appreciate the explanation.Richard is right to say that VoIP has modest bandwidth requirements and is sensitive to jitter (because the human ear is very sensitive to breaks in speech).In fact, this is a wonderful example of both why a neutral network is needed and why it succeeds.

When Mary and I founded ITXC in 1997, the common wisdom was that voice over the public Internet couldn’t work because the Internet is a chaotic place and does not promise quality of service in any dimension including packet delivery times, jitter, or even reliable delivery of packets.In fact, Internet routers throw packets away when they are overburdened.Moreover, the Internet certainly wasn’t designed with voice in mind. But it IS application agnostic.

Guess what?We found that we could do exactly what David Isenberg said: put intelligence at the edge of the network – actually the interface between the public switched telephone network (PSTN) and the Internet.Some of this intelligence – jitter buffers for handling jitter, for example – was designed by gateway vendors like pioneering VocalTec and later Cisco and others.Part of it was home grown: technologies for discovering which routes over the Internet were currently giving us the best quality and value and using them.

It worked.When ITXC was sold in 2004, it was the seventh largest carrier of international minutes in the world (measured in minutes carried) – this includes all the traditional carriers.Note that virtually all of these minutes both originated and terminated on ordinary phones.The callers did not know that their carriers – most of the major carriers of the world and many new ones – were handing the calls off to us for delivery to the called-country via the Internet where our affiliates then fed them back into the phone network.We would not have been successful if we could not deliver the quality of service which these callers and their carriers expected.

The point is that we succeeded BECAUSE the Internet didn’t try to help us.We were able to build what we needed at the edge of an application-agnostic but open network.File sharing services, video distributors, and other Internet services all do their own customization of the network AT THE EDGE.That’s why the network does NOT have to try to figure out what services each application needs and why revolutionary new applications can be developed.

There would be no VoIP today if we were waiting for the owners of Internet pipes to optimize them for our application.

Richard: “* Do you believe that it would be a good thing for Americans to have fiber-optic tubes running between their homes and the public Internet? If so, do you believe that the companies who install the tubes should be allowed to make a profit from their operation?”

Answer:That’s easy.Yes and Yes.More bandwidth costs more.I would happily pay more for more bandwidth if it were available in Vermont.

Richard: “* Do you believe that it's good for content producers to profit from the advertising they move down the tubes, but no one else?”

Answer: I don’t accept the premise that only the content producers benefit from the advertising.Internet users benefit because they receive the content which is paid for by the advertising.The advertisersbenefit or they wouldn’t advertise.And the Internet Service providers benefit because they are paid for faster connections both by consumers who want the content and by the content providers. We wouldn’t need megabits of speed if we only did email.In fact, in a recent deal, Verizon agreed to pay for content in order to make their newest network more attractive to the end users who pay a premium (properly) to connect to it.

Richard: “* Do you believe that triple play is legitimate or illegitimate? And does your answer depend on whether triple play segregates traffic by frequency or by time?”

Answer: By triple play, I believe Richard means the offering of phone service, Internet access, and entertainment in a bundle.Yes, I do think it is legitimate for one company to offer all three services no matter how they’re delivered technically. As a technician, I believe it’s a bad idea to segregate by frequency rather than time on a new network (the Internet segregates packets and applications by time) but that doesn’t mean I’d outlaw frequency segregation.Both DSL and today’s cable-provided Internet access are examples of segregating by frequency which is OK as a transition technology.

What I object to is the ability of the telco and cableco duopoly in the US to use their duopoly position to advantage their own phone and entertainment services when they are also the underlying provider of capability.This is exactly analogous to the dominant provider of operating system software using the operating system to advantage its own applications.If there were a competitive market for Internet access in the US,I would think that discriminating would be bad business for an access provider and that the market would sort that out – assuming that discrimination was known which is what I’m speaking about at Berkman.

Richard: “* Do you believe that companies who invest in infrastructure are entitle (sic) to recoup their investment in part by selling services on the infrastructure that are advantaged technically over services sold on the same infrastructure by companies who have no infrastructure investment at risk?”

Answer: Yes.But, if the companies that you’re talking about are part of a duopoly, then I believe that they must make it equally possible for their competitors to technically advantage their applications whether they own infrastructure or not.The advantage can’t come from duopoly control of the infrastructure.See the Windows analogy above.Microsoft has to make its APIs available to application developers who have NOT invested in the infrastructure of an application.

Richard: “* And finally, do you believe meaningful measurements can be reported by people who don't understand what they're measuring or why?”

Answer: Yes.So long as they have the right software.That’s exactly what I think and what I’m recommending.Note I have NOT called from Net Neutrality regulation or legislation despite my fears of the duopoly.I am calling for information so that the marketplace and, if necessary as a last resort, the political body can act.

Richard: “I won't be listening to your talk at Berkman, but will most likely check whatever podcast is available after the fact.”

Answer:Good.Thanks for helping me plan the talk (sincerely) and I look forward to your comments after.This is an issue which needs more light and less heat.

Speaking on Net Neutrality

Although I’m not currently an advocate of Net Neutrality regulation or legislation, I very much believe we need an Internet which remains essentially neutral and application agnostic.It’s very important to know whether Internet access or backbone providers are blocking or disadvantaging applications which compete with their own services – especially since some CEOs like at&t’s Ed Whitacre have threatened to do exactly that.

One very good way to know if we have a problem is to have a large group of Internet users monitor their ISPs and access to various sites and applications.It would be very helpful to have unobtrusive tools we net monitors could run whenever we suspect that a particular application – VoIP, for example – is being blocked by someone in the Internet food chain.

This monitoring is very much in line with what Dan Gillmor calls “citizen journalism” and Jeff Jarvis prefers to call “networked journalism”.It also might fit with Jay Rosen’s newly announced NewAssignment.net.

Even the known existence of a corps of citizen monitors might discourage anti-competitive behavior.On the other hand, if such behavior is detected, it might be possible to use existing antitrust law to stop it.If not, maybe the monitoring proves that I am wrong and that new regulation or legislation – as dangerous as that might be – IS needed.

This won’t be easy to set up.We need monitoring software – preferably open source so we can make sure that it is not biased itself.We need testers of the monitoring software.Many volunteers to run it.Technically adept volunteers to verify results and to probe deeper into apparent instances of net discrimination.And bloggers – and/or traditional media – to publicize the result.

So far this is just an idea – not a reality.I will be as much looking for answers as suggesting them at the talk.Hope you can participate.I’ll post what I learn. BTW, I’m using the tag “blocknot” for this effort so feel free to do the same if contributing.

Some of my posts on Net Neutrality are here and here, a post on Ed Whitacre and “his” pipes is here, and an earlier post on a citizen effort to detect blocking is here.

Skype and WildBlue – A Case for Citizen (Network) Journalists

Suddenly I can log on to Skype and even connect calls over WildBlue’s satellite-delivered Internet access service; Skype chat works fine.However, the quality on both Skype-Skype and SkypeOut calls is unusable – much worse than would be expected just because of the long time it takes packets to get to a geostationary satellite and back (minimum of .45 seconds).

The sudden failure of Skype over WildBlue on May 15 and the recent sudden recovery may be a good case for citizen journalists.It MIGHT have implications for the Net Neutrality debate.

Users on the WildBlue Uncensored! Forumreport that, starting two weeks ago, they regained the ability to connect to Skype and complete calls.Some of them also report usable call quality. As I posted previously, users say they had generally acceptable VoIP and Skype performance over WildBlue prior to May 15.I wasn’t using WB then so have no firsthand knowledge.

Why did Skype suddenly stop working over WB?Why did it suddenly start again?Did WB block or deprioritize Skype or VoIP packets?Or did a Skype update loose the ability to deal with the extreme latency (delay) expected when a satellite is used?Conspiracy theorists point to the coincidence between the early May announcement of a deal between WB and at&t, Skype’s announcement of free calls to ordinary phones within the US and Canada, and the sudden inability of WB users to access Skype.WB denies blocking any packets of any kind and points to a disclaimer in its sales literature which says it does not support VoIP when declining to investigate this problem.

My VoIP experience leads me to believe that the current poor call quality I have is a result of Skype jitter buffer management not coping with long and highly variable latency.I don’t have the right tools to know whether VoIP packets are getting worse treatment than other packets which would exacerbate the problem.

Bloggers Dan Gillmor and Jeff Jarvis both write convincingly about the reality and potential of citizen journalism now that the Internet has us all so well-connected to each other and to other sources of information.Jeff prefers the term “networked journalism.” There is now both a need and an opportunity for citizen journalists to do some network sleuthing.

The broad and important question is: are there actual violations of Net Neutrality happening today?Are ISPs favoring services they offer by disadvantaging competitors on the ISPs’ network?If violations were to occur, how would they be detected and verified?Even those of us who believe that marketplace should regulate Net Neutrality (which requires more than a duopoly offering broadband access) know that a market can only operate efficiently when accurate and timely information is available.

We citizens at the end of our Internet connections are well placed to gather the data needed; but we don’t yet have the right tools to do so. I’m no longer technically competent to write these tools but I’m sure a lot of you are. It is also possible that both ISPs and vendors of services which might be blocked will make tools available.

It’s a good guess that, if any service is blocked, it will be VoIP because the cablecos and telcos who control so many of our pipes have their own VoIP services to offer, because VoIP threatens the legacy revenue of the telcos, and because VoIP is much more quality sensitive than email or even web browsing. So tools for detecting VoIP blocking or deprioritization would be very useful. SIP is a VoIP standard to which tools could be written with huge applicability – although SIP tools probably would not help determine what’s going on with WildBlue and Skype since Skype communication protocols are proprietary.

Once tools have been offered by helpful citizens and/or vendors, we need more volunteers to test and validate the tools.Finally, we need people to run the tools whenever they suspect that blocking may be occurring.Most of the time the tools will probably be useful just for troubleshooting a problem that is not blocking. If blocking is never a problem, that will be great. Perhaps the existence of detection tools will help assure against unfair treatment of packets by ISPs.

But, if there violations of Net Neutrality, then it ought to be possible to develop clear and unassailable evidence.There is no law mandating Net Neutrality today;However, evidence of packet discrimination by members of the duopoly might well be grounds for action under existing antitrust law (see previous post on antitrust).

If you know of available tools or would like to volunteer a tool or make some other suggestion, feel free to post a comment to this post.I’m using the tag “Blocknot” to describe this effort and have set up a Technorati watchlist and RSS feed on Blocknot and a del.icio.us tag and corresponding RSS feed. If you blog about this subject, it’ll help pull the effort together if you use this tag.If you’re just lurking, check these lists for possible action.

Andy Kessler: GrumbyBet you didn't know that Grumbys caused the flash crash. Andy Kessler's new book is a great fictional description of the world of super-programmers, hacking, viral success and disaster, and flash crashes.

C. J. Sansom: Winter in MadridSansom shifts his historical fiction from the reformation to Spain under Franco in the miserable winter of 1940. At least as good - maybe better - than his Shardlake series.