If there is one complaint we hear from Android users more than any other, it's the speed at which software updates arrive. Or don't arrive. It's especially tough on tech enthusiasts who read about all the advantages of the new update, but can't get it on their own devices for six months, if at all. Even brand new devices typically fail to launch with the latest version of Android. The Verizon Galaxy Nexus—a marquee Android flagship if there ever was one—only got its new Jelly Bean (Android 4.2) update today.

This has been going on for years. So what the hell? Why hasn't the problem been fixed? And who's to blame here? We asked Android manufacturers, carriers, and Google what the hold-up was. And, what a tangled web we found.

What Goes Into an Update?

When Google releases a new version of the Android software, there are, essentially, three steps that must happen before the update will show up on your phone. First, the chip-makers must provide new "hooks," or code that allows the operating system to communicate with (and thus control) the hardware components. Because there are many different chipmakers within the Android ecosystem of devices (Qualcomm, Texas Instruments, Nvidia, and Samsung, among others), and each company has different chips that it makes (eg. Qualcomm's Snapdragon S3, S4, and S4 Pro), each one takes a different amount of time to develop. Typically, though, the chipmakers are able to deliver the new hooks within a month or two.

Then the software stack moves on to the manufacturers. Because each device is built with slightly different components, the new software must be custom-tailored for each phone or tablet. In other words, Samsung can't just apply its TouchWiz UI to Jelly Bean and then push it to all of its devices. Plus, each wireless carrier has its own unique set of software requirements.. That may include base-level functionality, and it may include carrier-specific apps (*cough* bloatware *cough*). That's in addition to whatever customization the handset manufacturers are doing in terms of their third-party user interfaces. According to Samsung's Nick DiCarlo, it takes about six to eight weeks, on average, from when the company get the OS update from Google to when it can deliver the finished version to the carrier. Small bug-fix updates will be much shorter. Bigger updates could be way longer.

Manufacturers just can't leave Android alone. When they build Android devices, they've…
Read more Read more

The manufacturers' third-party UIs ("skins") get blamed more than anything for upgrade slowness, and it's easy to see why. After all, they are visually prominent, and seem to be the only tangible difference from a Google Nexus phone, which typically launches with the latest Android version. But most of the work is actually fitting Google's new software to the hardware componets. "It's not as simple as, if we didn't do customization, just downloading a ROM from Google. That wouldn't work," says HTC's Drew Bamford. "So, even if we did no customization, I'm not sure that the process would be much faster, to be honest."

So if not skins, what's the major delay? Don't look at the manufacturers.

The Big Hold-Up

Welcome to the wonderful world of carrier testing. The wireless carriers have to test not only every single new phone they plan to offer, but also every software update to every phone that they are already carrying. Simply put, they have to be certain that the phone will work on their network as advertized. How hard is that? Try mind-bogglingly.

"They've got limited resources, people, time, equipment," says Samsung's DiCarlo. "The test scopes for these, as the networks get more complex with CDMA, GSM, LTE, multiple bands, now getting into VoLTE next year, different regions of the network are made with different network providers, so they have to test in all the regions. So the network testing complexity is extraordinary."

Each carrier has a validation team. They do everything from drop tests for the hardware to benchmark tests against usability metrics. They take software through automated experiences to see if there is a slowdown somewhere. When they finally give TA (Technical Acceptance) they want to be sure that they're maintaining their standards. "We try to do capacity planning," says T-Mobile's Jason Young. "We look ahead in the year and we are setting projected TA (or Technical Acceptance) dates for devices 6-12 months in advance. Then we work backwards from there." When they anticipate many device updates coming near each other they ask, "What device is more important for us to bring to market?" This prioritization is a sticky subject. According to DiCarlo:

"If you are a carrier and you're running a lab and you're supporting 30 or 40 phones at a time—and from their view they're supporting hundreds of phones. Two years of contracts over many years, right?— Do they want to spend time testing the new hotness that's coming out at the beginning of Q4, or an OS update for a phone from two years ago?"

This is a no-brainer. The carriers, after all, are in the business of selling you new devices to keep you hooked into their services. For the devices already sold, it makes sense to focus on the most popular devices first in order to keep the most people happy with the least amount of effort. It's simple economics: they get more bang for their buck that way. So how long does this take?

"I can tell you that when we release a new product to carriers, we can have it running in our labs for six months before it's released by the carrier," says HTC's Bamford. "It can take a long time." T-Mobile's Young confirmed that it is typically three to six months from the time they get the new software until it goes live. Simple addition, then, will tell you that it may be as much as nine months for that new software to make it to your device, and that's only if the manufacturers and carriers agree that it's worth devoting the time and resources to update it at all.

But Apple Is So Much Faster!

This is one of the greatest illusions in the tech game. Apple announces the new version of iOS, and wham! You can download it on your phone within a day or two. How did Apple manage to skip that exhaustive carrier-testing that everyone else had to go through? It didn't. It had to jump through the same hoops as the Android manufacturers; the only difference is that Apple jumped through them before it announced the update.

Sprint's Ryan Sullivan explains:

"I don't think that Apple is necessarily any faster, I think it just appears faster because when they're announcing the OS release, they're launching it. A lot of that is just because they control the platform stack, but they have the same group of people who are working continuously on the network integration pieces for their 250 wireless carriers across the world. So, while Google is announcing the software when it's done at the platform-level only, and then they allow OEMs access to it so that they can build that network integration layer. It appears that Google and the Android process takes much longer. It doesn't. Google just takes it half way, and then it's up to the manufacturers and carriers to take it the rest of the way so that it will work on a network.... I think the overall cycle from start to finish is relatively similar between iOS and Android, it's just the point at which the platform is announced."

Apple also has an edge here because it not only makes the operating system but also builds the hardware. The narrower portfolio of device components that are used means adapting the OS to the existing hardware (all of which is done in-house) is a faster, more seamless process. This is also why Nexus devices are first in line; Google works closely with its chosen hardware manufacturer to go with the new version of Android, and then it works with the wireless carriers to make sure everything is tested and ready to go ahead of the announcement. That only works at launch, though; older Nexus devices will still have to go through carrier certifications just like everything else when a new OS update comes in.

The Boogeyman

A lot of Android conspiracy theorists have come to the conclusion that manufacturers and/or carriers deliberately delay software upgrades to older devices in order to sell new ones. Of course, not a single person we spoke to would admit to that, despite our prodding. But what's actually happening isn't quite so cut and dried.

Again, it's all about prioritizing resources. Manufacturers have only so many employees, and they have to decide how best to use them. If setting them to work on applying a new update to older hardware makes them look good, they'll do it, but of course priority is given to new devices—the devices which are just about to launch, or which have recently launched and on which advertizing dollars are still being spent. And because network testing is so exhaustive, of course the carriers must prioritize, too, but different carriers will prioritize in different ways, depending on their current device lineup and what they have coming down the pipe.

According to Motorola's Punit Soni: "Some carriers say, ‘This update is really important to us, so as soon as you get it to us we're going to put it into the lab and devote all our resources to it'. Other say, ‘This is actually third or fourth in our queue, so we're going to have to wait a little bit until we can put it through our labs.'"

How Can It Get Better?

There are really only two ways this situation can improve. On the business end, manufacturers can scale back the number of devices they launch. It sounds crazy, but we're already seeing it happen with Motorola and HTC. At one point the market was flooded with so many different Android phones that it was impossible for consumers to make any sense of it. By consolidating their device profiles, HTC and Motorola are able to focus on phones that will matter. Then, in theory, since they have fewer devices, their resources won't be spread as thin when working on upgrades. Fewer devices for the carriers to test would help speed things up, too, though it will always be up to them how they prioritize, which will continue to be frustrating for users.

Consumers aren't completely powerless, though. Almost every phone on the market today can be rooted. As we recently noted, one of the best reasons to root your Android phone is that you can almost always download the latest updates months before carriers push them out. Now, there is some risk involved, as you may be putting software on your phone that isn't entirely ready for primetime. The best you can do is know your sources and read reviews for the build you're putting on (and make sure it's built for your exact phone/carrier). That said, it's a great way to sidestep the bureaucratic mess that is the official Android update process.