Technology Lab —

Google boosts open video by funding ARM Theora codec

Google has providing funding to the TheorARM project, an open source Theora …

In a move that will boost support for open video on the mobile Web, Google has provided funding to TheorARM—a project that produces an ARM-optimized implementation of the Ogg Theora video codec. Google's support for the project could be a signal that the search giant is significantly warming up to open video.

Although HTML5 delivers open standards for Web video playback, browser vendors have not been able to reach a consensus on the codec. Some parties favor Ogg Theora, a royalty-free codec that can be freely redistributed because it is thought to be unencumbered by patents. Others favor H.264, a codec that offers technically superior compression but is burdened with costly licensing fees.

Although Google's Chrome Web browser has support for both codecs, the company has declined to provide Theora-compatible video streams on YouTube, which uses H.264 for its HTML5 video player. During a discussion about codecs that took place last year on W3C's mailing list, a Google representative expressed extreme skepticism about Theora's suitability for the Web and strongly indicated that Google would not consider reencoding YouTube's massive video collection to accommodate users of Theora-only browsers. Google's newfound commitment to TheorARM might be a sign that the company is warming up to the possibility of more broadly supporting the royalty-free codec.

One of the chief criticisms of Theora is the lack of broad support on mobile devices—an area where Web video playback has become increasingly critical. Significant technical challenges make it difficult to get acceptable Theora playback on ARM-based devices. Developer Robin Watts, of Pink Noise Productions, has been working on a solution. In the TheorARM documentation, he described some of the biggest hurdles.

"The standard Theora decoder, as supplied, currently contains no ARM code whatsoever. Furthermore, it relies on various support libraries including libogg and libvorbis to do ogg bitstream handling and vorbis decoding. Unfortunately, libvorbis relies on floating point operation, which makes it a non-starter on the ARM platform," Watts wrote.

For his TheorARM project, Watts undertook a massive optimization effort in order to achieve good Theora performance on ARM. This partly involved rewriting performance-sensitive sections of the decoder in native ARM code. To deal with the floating point dependency in libvorbis, he built his own variant of Tremor, an integer-only Vorbis decoder.

Watts was initially releasing TheorARM under GNU's General Public License (GPL), which precluded his improvements from being adopted by the upstream BSD-licensed Theora implementation. (Watts may have chosen the GPL in order to sell commercial licenses to hardware vendors who wanted to use his decoder in their products under more permissive terms.) Google's funding has made it possible for Watts to relicense TheorARM and distribute it under the BSD license, which means that it can now be merged back into upstream Theora and freely integrated into proprietary applications.

By supporting Watts and funding his project, Google is making a significant contribution to open video. The ARM-optimized Theora decoder implementation supplied by Watts could make it possible for Theora to gain broader support on mobile devices. In a post that was published in Google's open source blog, Watts discusses some of the implications of TheorARM and offers insight into Google's reasons for supporting the project.

"We need a baseline to work from—one standard format that (if all else fails) everything can fall back to. This doesn't need to be the most complex format, or the most advertised format, or even the format with the most companies involved in its creation. All it needs to do is to be available, everywhere," he wrote. "Theora was designed and is maintained with the overriding goal of avoiding patents. No other codec can come even close to claiming to be as patent or royalty free as Theora can, whilst still holding a candle to the alternatives."

Indeed, Theora's lack of patent encumbrances makes it conducive to serving as a universal baseline codec. Google's financial support of TheorARM is obviously significant in the sense that it will enable broader adoption of the codec, but the message in the blog entry may be even more important. By allowing Watts to promote Theora as a universal baseline for Web video in a statement in the official Google open source blog, the company is endorsing the idea of making Theora ubiquitous. This represents a significant reversal from the search giant's previous sentiments.

Although this move will be enthusiastically welcomed by open video advocates, its timing seems somewhat peculiar. Google recently acquired video company On2, the company that developed the original technology behind Theora. On2's more recent codecs, specifically VP8, are said to be fully competitive with H.264. Many people in the industry assumed that Google acquired On2 so that it could open VP8 and make it a royalty-free standard for open video. The fact that Google still hasn't confirmed that theory, and is now more broadly committed to Theora, remains puzzling.

It takes some serious cajones to back an inferior codec just to make a point.

Unless something has changed about Theora, doesn't it provide much poorer video quality than competing standards?

I'm thinking it's both a poke at Apple and an insurance policy against harsh fee's for h.264 later. I think it's free until 2014 (??) but will be re-evaluated then. At that point if there's a free fallback for those who don't like the pricing structure it'll be a lot more difficult to over-charge.

How is Theora NOT a timebomb? Just because the project tried to avoid patents doesn't mean some troll isn't going to sue a huge company like Google. They can afford to risk it which is partly why they might be hedging their bets here AND going with h.264 at the same time.

FoO, while the MPEG LA can choose to increase rates in 2014, the maximum allowed increase is already known, so this is something companies can plan for.

Regardless, optimizing away floating point operation? Huh? What kind of ARM cores do they expect they'll actually be decoding video on? Practically all reasonable handsets today have native FP support. For example, when existing mobile game developers stated working with the iPhone, one of the first things they were told was "Please remove your fixed-point math libraries when porting your games. You're not in the dark ages any more".

Perhaps I'm being too trusting here, but could this not just be an attempt to bring some form of Theora up to scratch so that HTML5 video will catch on a little more? The switch to an open codec would allow other browsers such as Firefox to join in the game, and Google usually seem far more interested in making sure that a new technology is adopted than they are in trying to win market share by planting a flagpole in it and shouting "We got here first!" (although obviously, they could do that as well).

It seems a particularly relevant idea in the case of HTML5 video, because the average YouTube visitor isn't going to know or care what kind of video they are watching, and so won't be switching to Chrome for the video any time soon.

Theora's big advantage in the mobile space is that it's cheap to decode. I'd guess that battery is the limiting resource more than bandwidth, and that this will become even more true over the next few years.

How is Theora NOT a timebomb? Just because the project tried to avoid patents doesn't mean some troll isn't going to sue a huge company like Google. They can afford to risk it which is partly why they might be hedging their bets here AND going with h.264 at the same time.

Theora has patent trolls.H264 has patent trolls. AND the MPEG LA,

If the entire industry backs H264, and it becomes the ONLY codecs, the MPEG LA is given absolute monopoly power, and no one can do crap. If Theora becomes a reasonable fallback that's also supported, MPEG LA knows that if they piss people off, Google and the like will give them the finger, so they'll have to think twice about the decisions they make.

If there is a viable alternative than the H.264 codec, than when the codec will be kept at a reasonable cost if any at all. If there is no competition than there could be a substantial cost to people who use it. A small cost now for long term cost control is IMO what Google is doing.

ogg bundled with android and chromeos (latter more likely then former i guess, as the management of the two projects seems to have various differences) and a google app bundle (that includes android market) thats not tied to some NDA'd requirement sheet.

once those are in place, apple can kiss interest in their walled garder bye bye.

heck, what would be really interesting is if google unspun the android market into a non-profit independent of the google android project. But given the apple like leadership of said project at this time, i suspect pigs will fly before that happens.

This is all well and good that Google is putting money into this project to develop an ARM decoder for Theora, but what this story underscores is that Theora still requires a *software* decoder.

Every modern ARM SoC worth a damn for video playback comes with a hardware video accelerator with support for H.264. These have been slowly getting better for years, and are crucial for high performance and lower power video playback on ARM platforms.

Theora has a long way to go if we're still talking about someone writing a decent software decoder for ARM.

Also, this part raises some questions :

Quote:

"The standard Theora decoder, as supplied, currently contains no ARM code whatsoever. Furthermore, it relies on various support libraries including libogg and libvorbis to do ogg bitstream handling and vorbis decoding. Unfortunately, libvorbis relies on floating point operation, which makes it a non-starter on the ARM platform," Watts wrote.

For his TheorARM project, Watts undertook a massive optimization effort in order to achieve good Theora performance on ARM. This partly involved rewriting performance-sensitive sections of the decoder in native ARM code. To deal with the floating point dependency in libvorbis, he built his own variant of Tremor, an integer-only Vorbis decoder.

It was my understanding that ARMv6 and ARMv7 based processors do have acceptable floating point support in the form of hardware VFP and NEON acceleration.

The article states that H.264 has "costly licensing fees." Any qualification to that? Costly compared to what? They're certainly not costly while their free, and everything regarding what the fees will be in the future is speculation. I understand the need for caution--no one wants to just hand over the keys to the city, er, create a monopoly. But did the industry get suckered into paying for .gif was under attack? No, they changed to .jpg when that issue came up. Theora would be better off trying to start from scratch to compete. than to keep building on such a rocky foundation.

Every modern ARM SoC worth a damn for video playback comes with a hardware video accelerator with support for H.264. These have been slowly getting better for years, and are crucial for high performance and lower power video playback on ARM platforms.

Most sane ARM SoC do this by using a DSP. But then again, they cost a bit more.

also, H264 need hardware help, as it can drive a high end X86 to cry uncle. Heck, more and more the decoding is happening as "shaders" running on a GPU even on a desktop PC.

@anonymousmonk: H.264 isn't currently free. It costs money to encode (i.e. Youtube like sites pay), it costs money to decode (so browsers have to pay), it costs money to deliver subscription TV on the internet, it costs money to broadcast old fashioned TV etc.

What is currently free, and what they've non-bindingly promised to be free for the next five years is web video provided free to end users. So after you've paid to encode the video, and your customers have paid to decode it, you don't have to pay just for sending a file over the internet (well not yet).

Personally for me, despite all the furor on the web about how H.264 is unstoppable and a done deal, when companies give up hundreds of millions of dollars in licencing revenue I assume they're doing it because charging for web streaming would kill uptake stone dead. So it can't be *that* good.

And for reference, if anyone describes Theora as "poorer video quality" they really mean "similar quality to Apple's H.264 encoder" or "better than Adobe's H.264 encoder". Things get a bit confusing because there happens to be an GPL'd H.264 encoder called x264 which is better than all of them.

thanks for the great explanation, economics! I'd add that the owners of h.264 haven't made any promises about keeping prices down in the future (unlike mpeg-la) so not only might the price rise, it might rise quite a bit, which is why some describe it as a "timebomb."

Even if you have theora as a standard fall back it does not mean you can't serve h264 to those who have a decoder in place. Youtube already has a bunch of version of each video in different quality. Having the low quality as theora would not even increase storage needs significantly in these cases. Then you would not need to use theora where h264 is significantly superior (high res high quality, I guess).

Plus, everyone that does not want to or can pay the fee can stick to just theora.

Media codecs is such a messed up (mine) field .

On an article related note, I really like googles way of funding open source project, both like this and like summer of code.

The licensing fees on H.264 always puzzles me - if there's a cost involved, and it requires licensing, how come there are free, open-source H.264 encoders like x264?

Because software patents don't apply everywhere

So essentially, the root cause of all these problems lie with the broken US patent system? (yet again)

FFS!!!!!!!

Actually, it's more that the patent system in the US actually works and doesn't work in other places (more that there's nothing to protect MPEG LA's patents). The patents they hold are valid enough, it's just that they're unlikely to go after anyone personally using x264 (especially when showing a sue happy stance now would probably kill the inclusion of their format in HTML5).

Something I don't get, their promise of keeping H.264 "free" until 2014. What's to stop them releasing H.265 20 minutes after it's full adopted and stopping development of H.264? It'd be theoretically easier to update than to switch to an entire new format at that point.

Theora is a huge step backward, but H.264 is a huge timebomb. I cannot fathom Google's silence over VP8.

It isn't an "either one or the other" choice between Theora and VP8. There's nothing at all detrimental in having a broad choice of open, royalty-free video codecs. There is a variety of open image formats, a variety of open audio formats, and video need not be any different. Yes, the design goals of Theora and VP8 are similar, but Theora is in use in the here and now.

There's a quite a bit of Theora content on the Web and you're going to want to play that in Firefox, Opera, and Chrome on MeeGo, Android, Windows, OS X, Linux, and whatever other OS platforms become attractive. If nothing else, having more than one well supported open, royalty-free video codec diminishes the risk of open video in the long term as there will be functional alternatives available if any one open codec falls foul of submarine patents.

I envisage the triumvirate of open video codecs with broad support will be Theora, (eventually) Dirac, and (maybe) VP8. Probably all in the Ogg container. Used in combination with Vorbis, Speex, and CELT it will give all browsers a strong video and audio stack with which to support open multimedia on the Web.

So your contention is that there's been no growth or innovation in web browser development? I would suggest that the next crop of browser releases will be better than browsers ever have been with pervasive hardware accelerated rendering, far better performing JavaScript runtimes, 3D with WebGL, far better standards compliance, and better semantic markup.

Closed formats are the exception on the Web. Open formats result in low barriers to entry to software development which leads to more competition and more innovation in web software on more platforms, which, I further suggest, is a boon for everyone on the Web.

It takes some serious cajones to back an inferior codec just to make a point.

Unless something has changed about Theora, doesn't it provide much poorer video quality than competing standards?

Theora 1.1 (Thusnelda) was already a massive improvement over 1.0 and with becoming Ptalarbvorm it's going to get very close to h.264 if not on par at least with the stuff that most sites stream and of course decoders are backwards compatible. All in all this is very good news indeed.

Every modern ARM SoC worth a damn for video playback comes with a hardware video accelerator with support for H.264. These have been slowly getting better for years, and are crucial for high performance and lower power video playback on ARM platforms.

Most sane ARM SoC do this by using a DSP. But then again, they cost a bit more.

also, H264 need hardware help, as it can drive a high end X86 to cry uncle. Heck, more and more the decoding is happening as "shaders" running on a GPU even on a desktop PC.

I agree that H.264 is more complex, but that's not to say that for mobile applications, software only is good enough for a simpler codec like Theora.

General purpose CPU cycles are always at a premium on ARM, and hardware acceleration needs to be there in order to handle higher complexity video workloads, and to reduce CPU utilization on less complex workloads to improve battery life.

So your contention is that there's been no growth or innovation in web browser development? I would suggest that the next crop of browser releases will be better than browsers ever have been with pervasive hardware accelerated rendering, far better performing JavaScript runtimes, 3D with WebGL, far better standards compliance, and better semantic markup. Closed formats are the exception on the Web. Open formats result in low barriers to entry to software development which leads to more competition and more innovation in web software on more platforms, which, I further suggest, is a boon for everyone on the Web.

Open formats result in high barriers to entry due to awful ecosystems and badly designed frameworks.

Am I the only one that thinks that Google might believe that VP8 is patent encumbered by other companies' patents? Or that they might not be abiding completely by their slogan of "do no evil". come to think of it, we can't trust any company to advance the interest of the Internet and the consumer. it's only by accident that this happened. maybe buying on2 and shelving VP8 was sorta like giving the finger to apple.

Interesting move, definitely sending a message to ... well, to someone. Let's all just hold our breath waiting for GooTube to reencode all their videos. Or not, since they've said they won't do that. This is not a tide turner until that starts happening in some big way. Personally, I think that Theora is destined to fail, but who knows, perhaps this sort of threat of Big Backing will change things. It will not, however,magically make Theora a better codec.

Open formats result in high barriers to entry due to awful ecosystems and badly designed frameworks.

Really? See the Internet to see this is, once again, utter horseshit.

Quote:

Browsers aren't unfunded, they're supported by search provider deals.

So first it's licensing fees and when that is shown to be absolute nonsense it's well ok, but the only reason open source browsers aren't stagnant is because of funding. No one has denied or implied that funding doesn't help tremendously, but you said no licensing fees = stagnant growth. This is simply not the case.

So first it's licensing fees and when that is shown to be absolute nonsense it's well ok, but the only reason open source browsers aren't stagnant is because of funding. No one has denied or implied that funding doesn't help tremendously, but you said no licensing fees = stagnant growth. This is simply not the case.

Even if you have theora as a standard fall back it does not mean you can't serve h264 to those who have a decoder in place. Youtube already has a bunch of version of each video in different quality. Having the low quality as theora would not even increase storage needs significantly in these cases. Then you would not need to use theora where h264 is significantly superior (high res high quality, I guess).

and funny enough, the html5 video tag allows for this, as you can have multiple source elements inside it. Then the browser provides some way for the user to pick his preferred source before playback.