The carriers are not your customers: the Windows Phone 7 update mess

Microsoft has admitted that the Windows Phone 7 copy-and-paste update is going …

A month ago at Mobile World Congress, Microsoft announced that the Windows Phone 7 copy and paste update, known as "NoDo", would ship some time in the first half of March. That's not going to happen now, as the company has formally announced in a blog post that the update is delayed until the second half of the month.

The reason given is so that Microsoft can take a little extra time to ensure that the update "meets [Microsoft's] standards, your standards, and the standards of [Microsoft's] partners." The statement assured users that the updates scheduled for later this year won't be impacted by this delay, and that multitasking, Twitter integration, and an Internet Explorer 9-powered browser are all going to ship as expected.

The announcement also outlined the way that updates are signed off, first by the hardware companies, then by the carriers. This is a complex process: Microsoft's update delivery channel is used not only for software updates for Windows Phone 7 itself, but also for hardware-specific updates such as new drivers or, I believe, the low-level firmware used to control the cellular radio. This is a complexity that a company like Apple doesn't have—for Apple, all the hardware is its own. On top of that, Microsoft has, regrettably, given carriers the ability to block updates.

It's good to talk

It's good that Microsoft has finally spoken a little about what's going on. An official confirmation that there will be delays is better than rumor and innuendo, and the claim that the delays won't have any impact on future updates is also a little reassuring. But the response to the blog post in both the comments and around the Web has been largely hostile, and it's not hard to see why. What we wanted to see was some straight-talking. Something taking ownership of the problems, something to provide confidence that the problems had been solved, something to assure us that these issues were isolated—ultimately, something that would show us that Microsoft would deliver on the promise of its platform. What we got instead was PR-laden equivocation.

A big problem is that the announcement wasn't really telling us anything we didn't already know. A statement made on an official French blog said that NoDo would not roll out until the second half of March. The new blog post just makes clear that this is the case across the board.

On top of this, it offers no real explanation either for the past problems or for the new delays. This makes it hard to believe that the future is going to be any better. Though Microsoft has fixed the problems experienced by many owners of Samsung handsets for most users, there are still some who are having trouble installing the initial February update unless they completely wipe their phones, and since that update is a prerequisite for NoDo, those troubles are unlikely to disappear.

The partners aren't the important ones

The major issue, however, is not so much these details as it is a systemic lack of effective communication. Microsoft was slow to act in response to the initial Samsung problems, and the news about the delay was similarly circulating for several days before this official confirmation. Microsoft is still treating the OEMs and carriers as if they were its customers. While they're the ones Microsoft is dealing with directly, treating them as the customer will kill the platform. The early-adopting end users—the people who actually bought the platform, are buying the applications, and are encouraging their friends and families to follow suit—have to be treated as king. But they're not, and they're nothappy.

Microsoft, instead, is covering for its partners. It covered for Yahoo when a Yahoo bug caused sky-high data usage. And it's still covering. Instead of useful information such as "which carriers have blocked the February update," or "which carriers will block NoDo when it eventually arrives," we get vague comments about "working closely" with the carriers. Even though we already know that some carriers can actually treat their customers with utter contempt and block updates, Microsoft still skirts around actually admitting it—a fine example of PR standing in the way of actually relating to the public. Joe Belfiore already acknowledged that they could, so there's really no need to dance around the issue.

Obviously this is information that might be considered "sensitive," but putting the carriers' needs over the users' needs serves only to alienate the users, and it's Microsoft who looks bad as a result.

Nor does the post do anything to actually explain why the updates are taking so long to deliver. It talks about the OEM involvement and the carrier involvement. But none of it really makes sense.

The OEM issue is certainly the more sticky one. The fact that OEM-specific updates can be pushed out means that there is necessarily communication and testing between the manufacturers and Microsoft. This is a complexity no other phone platform has, and while it will be good in the long run, it's somewhat understandable that there are issues in the short term. However, the blog post doesn't paint a very pretty picture of the process. It may just be an oversimplification, but it sounds like Microsoft basically has to take it on trust that the OEMs have done their job properly. As is obvious from the Samsung issue, that trust isn't earned.

In its previous post about the February update, the company proudly claimed that the update worked for 90 percent of users, with a further 5 percent failing due to trivially diagnosable conditions (lack of free space) that the software didn't actually bother to warn about, preferring instead to try to install and then fail. If a Windows Service Pack only worked for 90 percent of users, leaving 5 percent perplexed with useless errors and another 5 percent with some mystery malady that risked leaving their machines useless, it would be a dismal failure.

The PC space is far more diverse than any cellphone ecosystem, and yet Microsoft makes patching that a routine and highly successful experience. Not flawless, but not far off. That it can't do so for nine different phone models, each using damn near the same hardware, is astonishing.

What would be good to see is an acknowledgement that this process broke down. Instead, we we're told that laboratory testing "can simulate—but never quite equal—the experience of delivering software to thousands of real phones. So basically, we're getting a shrug of the shoulders, and it's probably going to happen again.

The carriers can only ever make things worse

As for carriers, the blog post also made clear that carrier involvement was a part of any phone's testing. This is probably true, but it's completely missing the point in a couple of ways.

First and foremost, carriers really don't need to get in the way here. The testing they do just can't be all that important or significant. How do we know this? Because they'll let you stick any phone on their network (at least in the world of GSM networks). Even in the US, with its frequency-based barriers to entry, T-Mobile and AT&T will sell you a SIM card with no phone, for you to slap into any old device and use as you see fit, and this is repeated around the world. Some operators (mostly MVNOs who depend on another network's hardware) don't offer any handsets at all; only SIM cards and the expectation that you'll provide your own handset.

If the carriers can do this safely—and overwhelmingly, they acknowledge that they can—then they can also allow firmware updates through without interfering. If it's safe to use a firmware update on an unbranded handset on a network, it's safe to use a firmware update on a carrier-branded but otherwise identical handset. Plainly an untested firmware can't screw things up too much, or else the only phones they'd allow on their networks would be carefully vetted.

The carrier's involvement with the firmware should end at branding and perhaps a little configuration: the custom boot screen, the custom highlight color, a handful of other settings that carriers are allowed to touch, and the preinstalled applications. And that's it. There doesn't need to be extensive testing (they're not testing every phone on their networks anyway), and there certainly shouldn't ever be reason to allow carriers to block or delay updates. It should take mere minutes for a carrier to produce an appropriately branded firmware and give it the thumbs up.

And if they really can't do that? Microsoft (technically, if not contractually) can put out minimally branded firmwares of its own. That'll lose the preinstalled applications, but so what? They're all downloadable from Marketplace anyway! If the carriers continue to ruin the experience, that's got to be an option.

And that gives way to the second point: we've already seen updating done better, and that's the standard Microsoft needs to live up to. We've already seen the iPhone. Apple already works with dozens of carriers (to whatever extent that it's necessary to do so—Apple prohibits carrier branding), and yet in the same timeframe since Windows Phone 7's launch, Apple has managed to push out four releases containing fixes and/or new features (plus a fifth release for CDMA support). And while the number of different Windows Phone 7 models might create more work for Microsoft, they shouldn't make much difference to carriers, since most carriers only offer one or two models anyway, just as they do with the iPhone.

Plainly, regular updates without carrier blocks are feasible. Not just feasible: they are the standard that smartphone users should be demanding. If Microsoft can offer an iPhone-like update experience with an Android-like selection of handsets, that is undoubtedly a good thing. It is one of the promises of the entire platform. A high-quality update experience is one of the major things that can set this platform apart from Android, just as the delightful user interface already does.

So instead of straight talking or useful facts, we're left with a delay to NoDo so that the company can "learn all [it] can from the February update," and, uh, basically nothing to reassure us that it won't happen again.

"Innovation plans"

Microsoft knows that regular updates are valuable. As the blog post says in its unique PR-laden way, "Delivering regular updates to your phone is a key part of our innovation plans." And we also have quite a bit of evidence that copy and paste has been done for months. A build with copy and paste was shipped to developer handsets on December 5th last year. That build was number 7338. New and presumed-to-be-final builds that have leaked for certain HTC handsets have build numbers of 7339. The emulator that developers use to test their software on NoDo also has build 7339.

That strongly suggests that not too much has happened since the 7338 test build; that even then, the feature was all but done. The company also described it as the "January update" in a webpage (though the original page has long since disappeared into the memory hole, and now no longer mentions January), and even before Windows Phone 7 launched described it as coming in "early 2011"; the tail-end of March hardly qualifies as such. So we're looking at a delay of more than three months.

All this trouble, and for something that at the end of the day isn't even a major update—just copy and paste, some performance tweaks, and some small improvements to Marketplace. And yet Microsoft and its partners are struggling to deliver that much.

One has to wonder what's going to happen if an actual important update needs to ship. A gaping security hole in the browser, for example, that allows rooting the phone and malicious access to personal data. Something where a three-month turnaround is simply not acceptable. Are carriers and OEMs still going to be able to drag their heels and screw up the updating process? Or will this finally give them the spur they so badly need to swing into action?

Not fatal... yet

It's still early days in the life of Windows Phone 7, and none of these update troubles are in themselves fatal. But the communication has been abysmal. It has been consistently reactive, patronizing, and so laden with PR wording that it tells you next to nothing anyway. It's better than silence, but only just, and it's no wonder that the community feels short-changed by it.

Ideally Microsoft would be releasing updates in a timely manner, every month or two, adding new features incrementally and justifying the commitment made by the early adopters. Let the updates flow as fast as they can be developed. We know it's possible.

But if that doesn't happen, at least have the courtesy to tell people why. Tell people that the update is going to be delayed beforehand. Don't just keep silent and claim, "Well we never actually meant it to go out in January, that was all a misunderstanding." The userbase isn't that stupid. Tell people which carriers are doing what. We know carriers can block things, so don't fob us off with vague claims that we'll get the updates eventually—make sure that if someone is denied an update then they know who to blame.

The current strategy looks bad. It is causing a loss of confidence among early adopters, and that hurts. These are the people who will fall in love with the platform and who will do the real job of selling it to their peers. They have a reach and influence that no marketing budget can buy, and if they feel that the platform is lost, and that Microsoft can't deliver on its promises, and if they no longer have the faith that it will develop into the platform that it deserves to be, they will defect to other operating systems, taking their influence with them. Both the customers and Windows Phone deserve better.

You want to know what's even worse? According to Paul Thurott, the actual code for the update itself was done in NOVEMBER. OF LAST YEAR. It's seems pathetic that not the actual coding of the update but how it's being "tested" and delivered is the reason for this enormous slowdown.

Simply put, if there were a MAJOR bug in Windows Phone 7 that all of a sudden caused your phone to fail, Microsoft has shown that it wouldn't be able to fix it in any respectable amount of time. And that should be concerning for any potential customer of their phones.

I think you're being a little hard on MS. They're doing something that hasn't been done before -- and may not even be possible. They're trying to push out simultaneous updates across multiple carries, multiple OEMS, and multiple phone models (and apparently multiple firmware versions within those models). Frankly, I'm amazed they were even able to get 90% on first try. Take away user error and flaky Samsung, and it would be a 100% successful update.

Google doesn't update at all except their own Nexus phones.

Apple has complete control over their hardware, so they only have to deal with carriers. So, their update process is much less complex. Also, Apple at this point has carriers begging them to allow the iPhone on their network. MS is situation is reversed, and as a result, they lose a lot of clout about forcing updates through the carriers' approval process.

In the end, I'm not entirely convinced if their update plans are even possible. It'll take a TON of work on MS's part to verify the update across all the OEMs/models/carriers since apparently they'll have to verify everything the OEM's and carriers report. This will make updates even slower to arrive, so you'll probably be lucky to get 2 updates a year. We'll see.

This is just a different view to the same problem as Android is having, namely, the handset manufacturers aren't organized for or capable of deploying regular phone updates. In the dumbphone days they only had to update a phone maybe once or twice with a small bugfix patch, and they did that at a glacial pace. Now they need to push a new OS image every couple of months, and they just don't have the development, QA, and infrastructure to do that. Their old school shit just won't fly anymore. I have a feeling that the first OEM that really catches on and modernizes will end up taking over a huge portion of the market.

Whichever smartphone manufacture that can totally wrestle control from the carriers is the top player in my mind. The only one that's the closest to achieving that is Apple.

My Nexus One has had two major version updates, plus two or three incremental, pushed out OTA over the last year without any interference from Rogers. I don't think any other Android handset can lay claim to that on any carrier.

The problem is subsidies. I didn't mind shelling out the buy the phone direct from Google, but most people would opt to have the carrier subsidize that price in exchange for locking into a contract. The customers want subsidies, the manufacturers need the carriers to subsidize, so the carriers are holding the cards right now.

Apple managed to break that model, but you can bet that the carriers hate that fact. Not only do the carriers have to leave Apple in control of the platform (tethering not withstanding), but they have to offer deeper subsidies than for comparable phones. Apple has the momentum right now, but if they lose that and the phone starts to lose significant ground, you can bet that the carriers will pull back and force Apple into concessions, because Apple also needs the carriers to subsidize their phones.

Sadly, I can't see a particular Android or Win Mob model being able to generate that degree of market traction to force the carriers to relinquish control. The major manufacturers are too dependent on the carriers to risk rocking the boat too violently.

As long as Google and Microsoft can convince (or contract) manufacturers to provide vanilla-devices free from carrier interference, albeit at non-subsidized retail prices, there will at least remain an option.

Microsoft, instead, is covering for its partners. It covered for Yahoo when a Yahoo bug caused sky-high data usage. And it's still covering. Instead of useful information such as "which carriers have blocked the February update," or "which carriers will block NoDo when it eventually arrives," we get vague comments about "working closely" with the carriers.

In launching a new product, Microsoft needs to have its OS on as many phones and networks as possible. As a result, they had to cede some control to the carriers and manufacturers, such as the ability to block updates, branding, and so on. So, do you think it makes sense to alienate the carriers/manufacturers and point blame? Even if it is their fault? This only makes it less likely for that carrier/manufacturer to continue to support the new OS, especially in the wake of the Nokia deal. Making the carriers/manufacturers look bad is not good business for MS at this point.

I'd LOVE to see the carriers get out of the way of the phones, and it's unfortunate that Apple has been the only company to negotiate this. It seems with the strict customization requirements that Apple has put on the handset manufacturers, this is precisely the problem they were trying to avoid.

None of this bodes well for Nokia. They're going to be late to the party where most of the guests have already left.

Making the carriers/manufacturers look bad is not good business for MS at this point.

Its a question of what is worse, annoying the customers or the carriers. They already offer carriers less abilities to change the OS than Android does. They have tight control over the possible hardware configurations. Why stop at something as important as updates? This OS is based on Silverlight on .NET so its highly abstracted away from the hardware. So different hardware configs shouldn't be the huge problem as well.

Let's face it this could have been a crucial advantage over Android and they blew it.

I think you're being a little hard on MS. They're doing something that hasn't been done before -- and may not even be possible. They're trying to push out simultaneous updates across multiple carries, multiple OEMS, and multiple phone models (and apparently multiple firmware versions within those models). Frankly, I'm amazed they were even able to get 90% on first try. Take away user error and flaky Samsung, and it would be a 100% successful update.

Google doesn't update at all except their own Nexus phones.

Apple has complete control over their hardware, so they only have to deal with carriers. So, their update process is much less complex. Also, Apple at this point has carriers begging them to allow the iPhone on their network. MS is situation is reversed, and as a result, they lose a lot of clout about forcing updates through the carriers' approval process.

In the end, I'm not entirely convinced if their update plans are even possible. It'll take a TON of work on MS's part to verify the update across all the OEMs/models/carriers since apparently they'll have to verify everything the OEM's and carriers report. This will make updates even slower to arrive, so you'll probably be lucky to get 2 updates a year. We'll see.

Everything you said while sticking up for Microsoft ... just made me want to go buy an iPhone. lol

While a fledgling competitior, coming from a negatively perceived industry position, Micrososft will have to treat partners like partners/customers. It can't vilify them.

If they can tell Yahoo to fix something in private first, instead of immeditely putting them on blast, isn't it more likely they'll have continued support from Yahoo (good news for when we need Flickr in the Hubs)?

If AT&T is willing to promote three handsets while Verizon is discounting WP7 entirely, then Microsoft is not going to bully AT&T. The iPhone grasped the mind->marketshare necessary to bully carriers, MS hasn't yet.

Windows Phone deserves better but with your ongoing attack of MS, looks like they won't even have a chance to sort out the issues.

Yes, because Ars has that amount of power.

By all accounts, WP7 is a nice platform, and it was brought to market successfully in a very aggressive timeframe. That looks like good work on a software and engineering level, and even management of the launch cycle.

What's going wrong now appears to be a management issue, and as much as people want to say, "oh it's so hard, it can't be done, so many OEMs, so many carriers, oh the humanity..." what it comes down to in the end is that Microsoft chose this business model of their own free will. If it isn't working, Microsoft carries the can. Someone C-level should be shown the door for the way this is dragging out, and I think the right person is Ballmer.

Edit:

digger1985 wrote:

And Peter, where is the front page article about Yahoo fixing their IMAP implementation? I remember all the blame going to MS before. At least you could do something positive.

I think you're being a little hard on MS. They're doing something that hasn't been done before -- and may not even be possible. They're trying to push out simultaneous updates across multiple carries, multiple OEMS, and multiple phone models (and apparently multiple firmware versions within those models). Frankly, I'm amazed they were even able to get 90% on first try. Take away user error and flaky Samsung, and it would be a 100% successful update.

Google doesn't update at all except their own Nexus phones.

Apple has complete control over their hardware, so they only have to deal with carriers. So, their update process is much less complex. Also, Apple at this point has carriers begging them to allow the iPhone on their network. MS is situation is reversed, and as a result, they lose a lot of clout about forcing updates through the carriers' approval process.

In the end, I'm not entirely convinced if their update plans are even possible. It'll take a TON of work on MS's part to verify the update across all the OEMs/models/carriers since apparently they'll have to verify everything the OEM's and carriers report. This will make updates even slower to arrive, so you'll probably be lucky to get 2 updates a year. We'll see.

Everything you said while sticking up for Microsoft ... just made me want to go buy an iPhone. lol

*shrug* So go buy an iPhone if you don't already have one - which I bet you do. Hell, my next phone may very well be a jailbroken iPhone if Android and WP7 don't get their heads out of their asses. But if there's something specific that you think I'm off base about, I'd love to hear it.

Blue Adept wrote:

"I think you're being a little hard on MS."

The situation is lousy. If doing this work was too hard, they shouldn't have worked themselves into this situation. That's business at its most basic.

Agreed. But let's wait to see what happens. I can accept a somewhat botched initial update limited to exactly one phone manufacturer. I wouldn't accept a second though.

I think you're being a little hard on MS. They're doing something that hasn't been done before -- and may not even be possible. They're trying to push out simultaneous updates across multiple carries, multiple OEMS, and multiple phone models (and apparently multiple firmware versions within those models). Frankly, I'm amazed they were even able to get 90% on first try. Take away user error and flaky Samsung, and it would be a 100% successful update.

I think the company will get it right, eventually. But I don't think it has handled things appropriately in the meantime. It needs to be open and forthcoming to actually convince us that it is on top of the situation.

NoDo has been ready since at least early December, and it was probably built some days or weeks before that. We know that, because we've seen build 7338 on the development handsets. If it had shipped within, say, a month of actually being complete, nobody would be complaining. That'd be fantastic! I think it's what Microsoft wanted to do, too, hence the "January update" naming.

But for whatever reason, it hasn't managed it. And rather than coming clean, we are being fobbed off.

Why can't the OEMs just worry about plugging in the phone specific pieces (i.e., drivers, radios, etc.), and let Microsoft distribute OS updates directly? This is exactly how PCs and Windows work. The other option would be to continue baking things all into one image, but delivering that rapidly with little to no customization, similar to the Nexus phones or CyanogenMod for Android. Neither of these are being done. The model is broken and not geared towards consumers. I guess we've taken the situation with PCs over the last 20 years for granted.

wrote:<quote>I think you're being a little hard on MS. They're doing something that hasn't been done before -- and may not even be possible. They're trying to push out simultaneous updates across multiple carries, multiple OEMS, and multiple phone models (and apparently multiple firmware versions within those models). Frankly, I'm amazed they were even able to get 90% on first try. Take away user error and flaky Samsung, and it would be a 100% successful update. </quote>

I think this is exactly right. But I'd fault Microsoft for embracing a model that may just be over-complicated. "Good Enough Software" has been their credo for many many years. I find these results an unsurprising manifestation of their corporate culture.

X-Box stands out in contrast to other MS products - and demonstrates (to me) that MS is capable of creating a rock-solid product.

X-Box is an integrated approach though. And it would appear that an integrated approach is necessary - at least at this time - to have predictable results.

I hope MS can manage their way to a satisfying result. The market and consumers will benefit from their contrasting vision.

Could it be that the phone manufacturers are discovering that providing OS updates to the current generation of smart phones *greatly* reduces how frequently people buy a new phone, and therefore reduces how much money they're going to make? I see the same thing happening on the Android phones, where there are a lot of phones out there will 2.1 and 2.2 (and even some brand new phones and tablets are shipping with 2.2), despite updates being available.

Regardless of what many people think, Microsoft has a strong track record of providing system installs and system updates that that worked with little problem. The ability to easily install Windows and have it work with a very wide range of peripheral devices has been a large part of the reason for Windows' success. Over the last thirteen years I have installed pretty much everything Microsoft shipped without any real worry about problems. But, unless my problems with Windows 7 Service Pack 1 are a result of some unusual issue with my particular machine, Microsoft has severely lost its edge. Since Microsoft has provided very little explanation of what is in the Windows 7 Service Pack and has adopted the practice of rolling out endless Windows updates with very little explanation of what they contain, evaluating my problems is difficult. But they seem to have to do with changes that Microsoft has made in how devices are configured and in particular in how the USB bus works. It is not entirely clear whether my system instability has been do to problems with its bios, mistakes in Microsoft's function, difficulties in the process of reconfiguring devices, or intended changes of Microsoft's that don't work the way I expect. But the net is that a system that was working without problem became unstable after installing the service pack. I could only recover by totally reinstalling Windows from scratch. Even after reinstalling, it has taken me several rounds of trying different updates and restoring my system image from a backup to get back to a system that appears to stably support all the devices that I have been in the habit of using with my system. If any significant number of business users have an experience like mine, it will destroy the reputation of Windows 7. Microsoft managed to minimize the system stability problems in installing Windows 7. But the problems I have seen with its service pack are the worst that I have ever seen from any Microsoft install and those problems come with an update that has zero known value.

I think the company will get it right, eventually. But I don't think it has handled things appropriately in the meantime. It needs to be open and forthcoming to actually convince us that it is on top of the situation.

I'm waiting for the day that phone OEM's are like PC OEMS. Just sell me the damn phone and let me put whatever I want on it -- not likely anytime soon, but maybe with LTE? And, honestly, you've got a bit more faith in MS than I do about pulling this off. The number of manufacturers and phone models is only going to increase from this point on out. I'll keep my fingers crossed, because I want to see their 3 screens vision become a reality.

@jackifusPersonally, my crystal ball is saying that at some point MS will use Nokia as their primary platform, and those are the phones that MS will update directly. And that's where the vertical integration will manifest. The other manufacturers will push out the updates on their own.

In launching a new product, Microsoft needs to have its OS on as many phones and networks as possible. As a result, they had to cede some control to the carriers and manufacturers, such as the ability to block updates, branding, and so on.

I don't disagree with this, but it's amazing to think about what the situation is now. Microsoft is in a position that it basically has to grovel to the carriers, taking it on the PR chin as a result, all so they don't get dropped. Meanwhile, Apple is in a position to tell the carriers that it's going to do updates when it wants, and there's nothing they can do to stop it.

Looking back 10 years, I don't think you could describe a situation where one company is big enough to do whatever it wants while the other has to concede huge control over its platform to keep itself in the marketplace, and say that Apple was the one with the huge control.