Decoding the HTML 5 video codec debate

The HTML 5 video element has the potential to liberate streaming Internet …

The increasingly competitive browser market has at last created an environment in which emerging Web standards can flourish. One of the harbingers of the open Web renaissance is HTML 5, the next major version of the W3C's ubiquitous HTML standard. Although HTML 5 is still in the draft stage, several of its features have already been widely adopted by browsers like Safari, Chrome, and Firefox. Among the most compelling is the "video" element, which has the potential to free Web video from its plugin prison and make video content a native first-class citizen on the Web—if codec disagreements don't stand in the way.

In an article last month, we explored the challenges and opportunities associated with the HTML 5 video element. One of the most significant of these challenges is the lack of consensus around a standard media codec, a contentious issue that has rapidly escalated into a major controversy. The debate has now stalled without a clear resolution in sight.

The HTML 5 working group is split between supporters of Ogg Theora and H.264. Their inability to find a compromise that is acceptable to all stakeholders has compelled HTML 5 spec editor Ian Hickson to "admit defeat" and give up on the effort to define specific codecs and media formats in the standard itself. This is problematic because the lack of uniform codec availability will make it impossible for content creators to publish their videos in a single format that will be viewable through the HTML 5 video element in all browsers.

In an e-mail posted to the WHATWG mailing list, Hickson outlined the positions of each major browser vendor and explained how the present impasse will influence the HTML 5 standard. Apple and Google favor H.264 while Mozilla and Opera favor Ogg Theora. Google intends to ship its browser with support for both codecs, which means that Apple is the only vendor that will not be supporting Ogg.

"After an inordinate amount of discussions, both in public and privately, on the situation regarding codecs for <video> and <audio> in HTML5, I have reluctantly come to the conclusion that there is no suitable codec that all vendors are willing to implement and ship," Hickson wrote. "I have therefore removed the two subsections in the HTML5 spec in which codecs would have been required, and have instead left the matter undefined."

Ogg Theora is an open format that is thought to be unencumbered by patents. The primary reference implementation is distributed under an open source license and it is being developed by the non-profit Xiph.org with funding from Mozilla. Ogg is strongly preferred by the open source software community because it can be freely redistributed without requiring licensing fees.

H.264 is a high-performance codec that is maintained by the ISO Moving Picture Experts Group (MPEG) as part of the MPEG-4 family. It is emerging as the dominant codec for both streaming video and optical media, as it is said to deliver the visual quality of MPEG-2 (used on DVDs) at roughly half the bitrate. The MPEG LA consortium manages licensing of the underlying patents that cover H.264 compression algorithms and other software methods needed to implement the codec. In order to use the format, adopters have to pay licensing fees to MPEG LA.

Patent problems

Patent encumbrance is one of the driving forces behind the HTML 5 video codec controversy. The patent licensing requirements mean that H.264 codecs can't be freely redistributed, making the format a non-starter for Mozilla and most other open source browser vendors. Opera also objects, saying that the licensing fees are too high. Mozilla and Opera strongly advocate Ogg Theora as an alternative because its freedom from known patents could ensure that there are no licensing barriers that prevent ubiquitous adoption.

Apple objects to Ogg Theora, claiming that the lack of known patents on Theora doesn't rule out the threat of submarine patents that could eventually be used against adopters. Apple is also concerned about the lack of widespread support for hardware-based Theora decoding, a factor that diminishes the format's viability on mobile devices. Google shares Apple's skepticism about the potential of Theora in the marketplace. The search giant claims that Theora's lack of quality relative to H.264 will make it an impractical choice for large-scale streaming video services such as YouTube.

Obtaining a license for H.264 from MPEG LA doesn't guarantee complete immunity from patent infringement liability, though. Although it is generally assumed that MPEG LA controls all of the relevant intellectual property pertaining to H.264 implementations, there is still the possibility that a third-party which is not a member of the consortium has a broad patent covering related compression technology that it can independently enforce against MPEG LA licensees.

Although Theora is not known to infringe any patents, critics fear that enhancing it to make it competitive with the most modern and efficient codecs will greatly increase its exposure to infringement risks. Some critics even contend that it's not possible to advance Theora without inevitably hitting a patent wall.

Another licensing issue that is often overlooked is the ambiguity of MPEG LA's future patent royalty collection plans. MPEG LA has established broadcast fees that licensees will be required to pay for distributing free (or ad-supported) streaming video content on the Internet. These fees will not be instated until the end of 2010, when the second H.264 licensing period goes into effect. The language used in the current license treats Internet streaming just like over-the-air television, implying that the licensees will have to pay broadcast fees per-region. That could prove to be extremely costly for Internet video providers who make their content available around the world.

MPEG LA has provided no guidance, clarification, or insight into what the broadcast licensing fees will look like. When asked directly about the issue, MPEG LA representatives say that they haven't even decided yet themselves. The worry is that H.264 licensing for content distributors could potentially become too costly to sustain widespread use for streaming Internet video.

The compression efficiency debate

The viability of Theora for large-scale streaming video web sites is Google's primary concern. Google is committed to shipping both Ogg and H.264 support in its browser, but intends to use the latter to power its popular YouTube video site. The direction that YouTube goes for streaming video will be enormously influential and could, by itself, play a very significant role in determining the outcome of the codec issue.

The extent to which Theora lags behind H.264 is often overstated and the codec is, in actuality, in better shape than is generally thought by many of its critics. Google open source programs manager Chris DiBona is skeptical, however, and articulated the search giant's concerns about Theora's compression efficiency during the debate on the WHATWG mailing list.

"If [YouTube] were to switch to Theora and maintain even a semblance of the current YouTube quality it would take up most available bandwidth across the internet," DiBona said. "The most recent public number was just over 1 billion video streams a day, and I've seen what we've had to do to make that happen, and it is a staggering amount of bandwidth."

DiBona's quality claim was broadly disputed by Theora supporters on the mailing list. Mozilla's Mike Shaver encouraged DiBona to examine the most recent Theora developments, suggesting that the latest improvements have helped to significantly close the gap in compression efficiency.

"I don't think the bandwidth delta is very much with recent (and format-compatible) improvements to the Theora encoders," he wrote. "[Codec improvements] are a big part of what we've been funding, and the results have been great already. I'd like to demonstrate them to you, because I suspect that you'd be a better-armed advocate within Google for unencumbered video if you could see what it's really capable of now."

Xiph's Gregory Maxwell responded to DiBona's mailing list post by publishing a comparison that aims to demonstrate Theora's efficacy relative to H.264 in the context of YouTube-quality streaming video.

"Using a simple test case I show that Theora is competitive and even superior to some of the files that Google is distributing today on YouTube," he wrote. "Theora isn't the most efficient video codec available right now. But it is by no means bad, and it is substantially better than many other widely used options. By conventional criteria Theora is competitive. It also has the substantial advantage of being unencumbered, reasonable in computational complexity, and entirely open source. People are often confused by the correct observation that Theora doesn't provide the state of the art in bitrate vs quality, and take that to mean that Theora does poorly when in reality it does quite well."

Ogg has several high-profile supporters, including popular video streaming site DailyMotion and the Wikimedia Foundation, the organization behind the popular Wikipedia Internet encyclopedia. DailyMotion recently began the process of converting its video library to Ogg which it plans to deliver through the HTML 5 video element. DailyMotion has acknowledged that its Ogg streams have some technical deficiencies compared to its current Flash-based video streaming solution, but is confident that it's the best approach in the long run.

The Wikimedia Foundation, which is a strong supporter of open technology and unencumbered accessibility to information, was already committed to Ogg even before the HTML 5 video element gained multiple browser implementations. The organization is collaborating with Mozilla in the effort to boost Theora quality. In an e-mail on the WHATWG list, Wikimedia Foundation volunteer media contact David Gerard said that the organization is also interested in helping Mozilla to raise general awareness of the advantages that unencumbered video would bring to the Internet.

"I'd also point out that Wikimedia has vast publicity abilities in this direction," he wrote. "And we're watching the progress of Theora and Dirac on a day-by-day basis, for obvious reasons. So if you need large charitable organisations to help you with making this the obvious publicity choice for a happy Internet with cute fluffy kitties, I can tell you we'll be right there!"

A Fluffy Kitty summary of the codec debate

The following images demonstrate what the future of the web might look like, depending on potential outcomes of the codec debate.

The Web, with an unencumbered video codec

The Web, with many competing patent-encumbered codecs

I, for one, welcome our fluffy kitty overlords.

The undesirable middle-ground

A solution that seems logical on the surface is to simply expose each platform's underlying media playback engine through the HTML 5 video element—DirectShow on Windows, GStreamer on Linux, and QTKit on Mac OS X. This would make it possible for the browser to play any video formats that are supported natively on the user's computer.

From a purely technical perspective, this is not an impossible problem to solve as there are already existing libraries that do this and provide a cohesive abstraction layer on top. One prominent option is Nokia's Phonon library. It could also possibly be done by using the Quicktime and DirectShow plugins for GStreamer.

Mozilla strongly opposes this approach because it would heighten the risk of fragmentation. Allowing content providers to use any codec that is available on the user's computer might undermine the advantages of the HTML 5 media element because there would be no consistency guarantee and content would not be able to work everywhere. That is, however, arguably the situation that already exists as a result of the impasse in the codec debate.

Conclusion

Hickson has clearly grown tired of the debate and has no interest in allowing the divisive issue to continue distracting HTML 5 stakeholders from their efforts to push forward the standard. He takes the view that documenting a codec in the standard will achieve nothing unless the browser vendors are willing to conform with what the standard says. Microsoft, which has no plans to implement the HTML 5 video element at all, is also still an impediment to bringing open Internet video to the masses.

It's unfortunate that this debate is threatening to derail the adoption of standards-based Internet video solutions. The dominant video solution today is Flash, a proprietary technology that is controlled by a single vendor and doesn't perform well on Linux or Mac OS X. There is a clear need for an open alternative, but the codec controversy could make it difficult. My inner pessimist suspects that Microsoft will finally get around to implementing HTML 5 video at the same time that the H.264 patents expire, in roughly 2025.

154 Reader Comments

Creating de jure standards like HTML5 becomes a meaningless exercise if it contradicts the de facto standards. H.264 is clearly the de facto standard and so either you get all the big content vendors (Google, Apple, etc) on board with a different standard like Ogg/Theora, or you stick to de facto.

It *would* be better if you could get Apple/Google to both support Ogg/Theora, but they didn't agree to. So now, as a practical matter, the only way to help the HTML5 standard at this point is to support H.264 -- or drop it altogether. I personally feel that the standard would be better with a <video> tag than none at all, so I'm disappointed with the decision.

Originally posted by drag:There is ZERO advantage of H.264 over Theora when it comes to submarine patents.

Actually there is, in terms of presence. H264 is out there being shipped in millions of devices in hardware form today, and for it to have gotten to that point various things were done at the MPEG. The MPEG being an arm of the ISO mind you, they've got a bit of experience with this sort of thing. Its a known factor, and never underestimate how much that means to a company's lawyers.

H264's techniques have been searched for exhaustively in patents and patent applications, by more organizations than there are individual developers on the Theora project (think basically every CE company on earth that builds products around H264, like say BluRay players, AVCHD players in China, etc). The same is not true of Theora, it has not had even remotely as exhaustive a search through patents as H264, and as far as patent liability goes, with Theora you're effectively taking one organization's word that the codec is patent free (Xiph).

MPEG LA is fairly careful about making sure they have the rights for the IP they're claiming to sell. That and the codec's well established history give it a huge advantage over Theora as far as submarine patents go. There aren't that many openings for a submarine patent in H264 because of how thoroughly it's been gone over and because of the age of many of the patents that cover it. Theora on the other hand has the potential to be a minefield. The VP3 codec on which its based had patents on it, which were all granted in perpetuity to Xiph IIRC by On Technologies, but that doesn't rule out advancements since VP3 having infringed on patents that nobody at Xiph knew about.

quote:

* There is absolutely no chance that MPEG will pay your legal fees or licensing fess if a third party comes after you.

That's not the point, the point is how likely it is that a patent troll ends up holding a valid patent over either codec. The companies aren't that concerned over legal fees defeating garbage patents, they're concerned about genuine ones, or fairly close to genuine ones, that predate their own patents. An example of such a patent would be say the one that the CSIRO held on 802.11g that the IEEE ignored, that lead to injunctions against a few hardware makers. H264's patented to hell, and is very well known legally. Theora is not patented other than in the original VP3 patents, and investigations into its current behavior have been inadequate compared to those of H264. The two situations are not equal in the slightest as far as how unknown each codec is legally.

For someone to have grounds to launch a submarine patent attack on H264 they'd have to have applied for the patent back in the mid 90s, because that's when most of this tech was developed. Any patent troll showing up in even the rocket docket with some junk patent written in the 2000s over things in H264 would be tossed as soon as the prior patents used for H264 are shown. OTOH if they show up and claim they have a patent that applies to techniques added to Theora since it diverged from the original VP3 codec, they'd likely have a much better case.

Creating de jure standards like HTML5 becomes a meaningless exercise if it contradicts the de facto standards. H.264 is clearly the de facto standard and so either you get all the big content vendors (Google, Apple, etc) on board with a different standard like Ogg/Theora, or you stick to de facto.

So how exactly do you expect Mozilla/Opera to ship a free product that contains code that you have to pay for?

quote:

I'm with Apple. What's the point in implementing an inferior codec that's useless to most devices? Even – especially – if ogg takes off in desktop browsers, it perpetuates the current situation we have with Flash et al: mobile video isn't possible because everything is done on the CPU.

This is what I don't understand. Theora is a video codec like H.264. That means that hardware acceleration, outside of hardware built specifically for that purpose will be done with GPUs and shaders. Shaders are just GPU programs. Theora is a codec. Shaders can operate on codecs.

Unless there is something so unusual about Theora that it cannot be decoded on shaders, I don't see the problem. It's simply a matter of writing the proper shader algorithm.

Originally posted by darkowl:Wait, we're still letting Opera think they're relevant with their miniscule market share? They've gone through many years and many versions and they still haven't gone anywhere significant. Who cares what they think? They'll be dead in five years, ten if you want to be conservative. The market is now between MS, Apple, Google and Mozilla.

Every wii uses Opera, every DS does and every mobile phone for AU/KDDI uses it too. I would not say they have a miniscule market share.

Anyway - why all the scorn directed at Apple, Ogg, Opera, etc? The real problem here is Microsoft and Adobe. We're ultimately talking about the adoption of HTML5, right? Without that, the codec issue is completely moot.

* Support both codecs, with a fair note about the goods and bads of either.

* Let the fools who broadcast H.264 content and then get charged for it suffer in 2010, tell them "we told you so" and leave the door wide open for them in ogg land.

* Drop H.264 support because the masses of small content providers will have spoken. By this point, VideoBay will have killed YouTube. Then move to wmv because Microsoft will have screwed us all with IE9 anyway.

I didn't think the piece was biased. Some of it seemed to be quotes that weren't attributed however... maybe missing some quotation marks.

Having looked at the samples on Hicksons site I have this to say. Low-bit rate Ogg looks like crap, but it's watchable, whereas low-bit rate H264 is lower than crap and totally unwatchable. The high bitrate samples look about the same to me but are still a little painful to watch. Hence I don't think bitrate (bandwidth) is really the issue as much as decoding is. You need hardware accelerated decoding so you can make use of high bandwidth without killing one's CPU and battery.

A couple comments: -It seems to me that Apple has entirely valid reasons to support H.264 over Ogg, which is that it already has invested in it with both hardware and software.

- I think everyone agrees that, as I pointed out above, the lack of hardware acceleration for OGG is a monumental problem. I have a question though, couldn't hardware be made that could accelerate both?

- I'd vote for the HTML 5 spec including OGG AND the ability to utilize external codecs. I don't see any browser maker being able to leverage hardware acceleration directly.

Originally posted by d_jedi:H.264 is a technically superior format to theora. Therefore, as far as I'm concerned, it is the one that should be used. If that requires paying licensing fees.. well, that's too fucking bad.

Current royalties are 20 cents for the first 100k, 10 cents and are capped after awhile. Not much for Google to pay, but it does screw up the licensing on Mozilla.

Of course, the MPEG-LA front could just end it all and ask for royalties on only the content being sold or encoded, and not on end user decoders, but they lump the encoder and decoder together.

Then there is MP3 and how they didn't care about LAME. And then there was Microsoft getting shafted in court about MP3 because they didn't a license from one of the holders of patents.

quote:

Apple objects to Ogg Theora, claiming that the lack of known patents on Theora doesn't rule out the threat of submarine patents that could eventually be used against adopters. Apple is also concerned about the lack of widespread support for hardware-based Theora decoding, a factor that diminishes the format's viability on mobile devices.

Seeing as how On2 owns the VP3 patents and put them into the open domain, I fail to see how Apple is worried.

OTOH...

quote:

Although Theora is not known to infringe any patents, critics fear that enhancing it to make it competitive with the most modern and efficient codecs will greatly increase its exposure to infringement risks. Some critics even contend that it's not possible to advance Theora without inevitably hitting a patent wall.

is more like it.

And, of course, the biggest bull shit of it all is this: We don't sue the people who made the infringing technology, because that would bring in no $$$. No, let's sue Google and Apple instead if they distribute it.

Originally posted by jb510:Having looked at the samples on Hicksons site I have this to say. Low-bit rate Ogg looks like crap, but it's watchable, whereas low-bit rate H264 is lower than crap and totally unwatchable. The high bitrate samples look about the same to me but are still a little painful to watch. Hence I don't think bitrate (bandwidth) is really the issue as much as decoding is. You need hardware accelerated decoding so you can make use of high bandwidth without killing one's CPU and battery.

If you see anything like this, then it was encoded very badly (either x264 default options or, worse, ffmpeg default options - these are getting fixed in a few days). H.264 has so many insurmountable coding improvements over Theora that it could never approach it in quality, especially when even Thusnelda still only has 1pass ABR as a ratecontrol method.

(I suspect Theora would be difficult to decode in hardware due to not using macroblock coding order, but it is simple even for the early-90s era technology it is, so maybe it's possible)

I think h.264 is the way to go largely because it is technically superior and has the larger installed based.

Would it not be possible for Firefox and Mozilla to use an underlying service (like Quicktime's) to do h.264 video? Most Macs and PCs already have QT installed. I can't see the problem there. You download Firefox, go to a website that references the HTML 5 video tag and you either have QT installed or are directed to a list of sites where you can get it. You head over to Apple.com, and get QT.

Along the way, you've installed Firefox, QT, and very likely iTunes as a result. Seems win-win for me. It also seems a lot easier than hoping that someone has a Xiph plugin installed.

1. I fail to understand why a Web Standard should define ONE video codec to use for the Internet. Like almost everything ( if not everything ) in life, No one things will suit us all. ( I understand their intention. But the world / universe doesn't work like that. And they should have known better )

2. I also fail to understand why HTML 5 should define a Video Codec. They should concentrate on Core Markup languages itself. No wonder why HTML 5 took nearly 10 years and it is still not yet well defined.

3. Mobile Devices are the future's PC. In fact they already are. People either dont understand or neglect the problem with Mobile Devices. Hardware Acceleration is a MUST. It will not scale like PC because of battery limitation. There are already 100S of MILLIONS of devices with H.264 Hardware Acceleration supported. Many different chips to choose from. And more planned chipset and devices coming in the near future. Theroa? ZERO.

4. The latest Theora 1.1 Aplha 3 has HUGE improvement over the original 1.0 or VP3. And that is a fact. But H.264 is still better. And properly will be forever by technical terms. While the PSNR graph of the latest Theora update looks good. The Clips i downloaded tells a completely different story. Unless you like ringing, and muggy colours, my personal preference is H.264. I dont know if the clip was using Maxwell edition, I will need to check. But given the H.264 version was simply using the default settings in X264 encoder gives much confidence to how good the best H.264 implementation X264 is.

5. Mozilla wants the video Decoder inside their Browsers. And Theora is an important steps because H.264 require license fees for distribution. Mozilla wants to build Firefox as a Platform. The less it rely on the OS the better. I believe it is politics more then anything else.

Anyway, I still believe we should all use H.264. But given the web is about chooses, the simple solution is to offer Theora AND H.264 depending on Browsers. Hardware is Cheap, Software is expensive"With today's enormous HDD, adding a few more TB of space shouldn't cost that much.

Why doesn't MPEG just grant a cheaper implementation license for H264. I can understand why Mozilla and Opera object to expensive non-open source video codecs, but MPEG4/H264 is a far better option than Theora.

H264 having far superior quality at lower bit-rates than Theora is a secondary issue in this discussion.

Surely the most important issue relates to non-desktop applications.Due to their limited CPUs, devices like cheap laptops, netbooks, tablets, UMPCs, and especially smartphones *cannot* playback high-quality/HD video that is being decoded in software (I.E. using solely the CPU). In order for these devices to play high quality video at decent framerates, they have to use dedicated video decoding logic that is built-in to their graphics chipsets or added on system-on-a-chip platforms. The architecture of these dedicated video decode engines varies, but most of them are fixed function and are optimized for specific formats, mostly MPEG4/H264 and VC1, with some offering support for MPEG2. NONE of them offer native support for Theora.

Even if mobile device processors were powerful enough to use software decoding, it is a major waste of power. Dedicated video decoding logic can decode HD video using a tiny fraction of processing power and energy use of a software decoder.

If HTML5 video wants to remain at all relevant to mobile devices like the highly succssful iPhone+iPod Touch family and the enormous netbook market, they have to adopt H264 as the standard.

Originally posted by ramuman:I think h.264 is the way to go largely because it is technically superior and has the larger installed based.

Would it not be possible for Firefox and Mozilla to use an underlying service (like Quicktime's) to do h.264 video? Most Macs and PCs already have QT installed. I can't see the problem there. You download Firefox, go to a website that references the HTML 5 video tag and you either have QT installed or are directed to a list of sites where you can get it. You head over to Apple.com, and get QT.

Along the way, you've installed Firefox, QT, and very likely iTunes as a result. Seems win-win for me. It also seems a lot easier than hoping that someone has a Xiph plugin installed.

They could use mplayer, but that solves nothing. You would be back to a plugin again, and I doubt many companies here in the US and abroad would appreciate mplayer and its ability to decode any format, especially Dolby Digital (Dolby sent nasty letters to the university hosting VLC about how it can decode DD - wait, maybe it was DTS - you get the idea).

The whole point of this is simple: Install a browser and you are done. No plugins, no add-ons, no dependencies on the OS, and no patent minefields. Checking the license of FireFox shows nothing that requires a patent license or royalty.

Also, the whole point of HTML 5 video is that you can manipulate it as well.

Personally, I think it is unnecessary to reinvent the wheel by trying to popularize Theora. H.264 is widely used, has existing hardware acceleration support in both mobile devices, computers, and living-room electronics, and is now fairly well known to the public.

In my mind the ideal solution is for Apple, Google, and Microsoft to try to negotiate reduced licensing fees for H.264 for use in internet browsers and/or pay for H.264 licensing fees for open-source browsers. Perhaps 1 fixed bulk H.264 licensing fee paid for by Apple, Google, and Microsoft that will cover all present and future open-source browsers that will implement H.264 HTML5 support. Why would they agree to sponsor the open source community? For Google and Apple, if they really want to stop the use for Flash for internet video and get H.264 to be the standard then they should be willing to pay for it if the cost isn't too unreasonable. In Microsoft's case, even though they haven't commented on the issue, I wouldn't be surprised if they would prefer H.264 over Theora too. For Microsoft, being seen to sponsor the open source community and promote internet standards would generate positive press against all those anti-trust and monopoly allegations that crop up, especially with the EU. And it's not like Apple, Google or Microsoft don't have the profits to invest somewhere.

I believe the standard should automatically be OGG/Theora and in addition the people who wish to use the H.264 can pay for their own licensing of the crap through MPEG LA, as need be.

That way the license is carried over to the users, on a self serve basis.

When most people refuse to pay to use it, it'll just prove that the H.264 and MPEG are unpopular.

And hopefully it has disclaimer of PURE HATE ( a rant against Microsoft, Adobe, Nokia, Apple, Google, etc) to better tell why these companies were so lazy to say yes and instead insist to make the users lives more difficult.

Due to their limited CPUs, devices like cheap laptops, netbooks, tablets, UMPCs, and especially smartphones *cannot* playback high-quality/HD video that is being decoded in software (I.E. using solely the CPU). In order for these devices to play high quality video at decent framerates, they have to use dedicated video decoding logic that is built-in to their graphics chipsets or added on system-on-a-chip platforms. The architecture of these dedicated video decode engines varies, but most of them are fixed function and are optimized for specific formats, mostly MPEG4/H264 and VC1, with some offering support for MPEG2. NONE of them offer native support for Theora.

Even if mobile device processors were powerful enough to use software decoding, it is a major waste of power. Dedicated video decoding logic can decode HD video using a tiny fraction of processing power and energy use of a software decoder.

If HTML5 video wants to remain at all relevant to mobile devices like the highly succssful iPhone+iPod Touch family and the enormous netbook market, they have to adopt H264 as the standard.

Most of the problems decoding H.264 are in the HP level and are at 1920x1080 resolutions. Even a lowly dual core atom could play a 720 HP stream with no issues. And handheld devices do with only the SP level, which is a "breeze" to decode in comparision. Besides, who puts a 10GB 1080p movie on a 480x320 handheld to watch anyhow?

And even then, CoreCodec put to shame all other software decoders and basically said, "It's the code, stupid. We don't need no stinkin video card decoder." Even the new multi-threaded compile of mplayer/ffmpeg gets rid of 1080p player issues regarless of what video card you have. I didn't get a chance to use it on the Atom though.

Consider this perspective however: Most web pages serve up free content. The text and pictures are perhaps copyrighted, but nothing prevents you from copying and pasting content. Bring video into the picture, and usual it takes much more effort and money to produce than simple text and pictures.

So, companies either use DRM such as NetFlix, obfuscation via Flash or Silverlight, or other means to prevent ripping or saving the content as you would text. However, other people could care less. So let's define the bare minimum to allow easy access to free video in the HTML 5 spec, and for those who need to make a dime, can keep using Flash or other pointless methods of DRM via plugins (more like programs).

Obviously, Theora isn't trying to be the be all end all solution, but just a standard one. I doubt anyone will have issues decoding YouTube Theora videos on an ipod or any other device. My old Palm LifeDrive could play Xvid/Vorbis video in an ogg container just fine using TCPMP, even if the video was larger than 640x480. This was also on a device with an old Intel XScale 416mhz CPU.

And if H.264 wins? Do what already Safari does with Theora; get a plugin. But at least a plugin is much better than installing a program like QT or Real Player or Flash ever was.

Originally posted by ltcommander.data:Personally, I think it is unnecessary to reinvent the wheel by trying to popularize Theora. H.264 is widely used, has existing hardware acceleration support in both mobile devices, computers, and living-room electronics, and is now fairly well known to the public.

In my mind the ideal solution is for Apple, Google, and Microsoft to try to negotiate reduced licensing fees for H.264 for use in internet browsers and/or pay for H.264 licensing fees for open-source browsers. Perhaps 1 fixed bulk H.264 licensing fee paid for by Apple, Google, and Microsoft that will cover all present and future open-source browsers that will implement H.264 HTML5 support. Why would they agree to sponsor the open source community? For Google and Apple, if they really want to stop the use for Flash for internet video and get H.264 to be the standard then they should be willing to pay for it if the cost isn't too unreasonable. In Microsoft's case, even though they haven't commented on the issue, I wouldn't be surprised if they would prefer H.264 over Theora too. For Microsoft, being seen to sponsor the open source community and promote internet standards would generate positive press against all those anti-trust and monopoly allegations that crop up, especially with the EU. And it's not like Apple, Google or Microsoft don't have the profits to invest somewhere.

You said it better than I could have. This is the only "best" solution I can see. Plus plus.

This is awesome news. It only further cements Flash Platform as the solution with the most ubiquity across platforms and devices. Flash player plus h.264 video is awesome and it's viewable by almost everyone... today and in the future. Far better quality than the Ogg BS. Nobody really cares about web video being open source except for about 0.5% of the market.

One that is unencumbered by patents, and open source, allows writing of third-party applications that let you save screenshots from video streams, and that let you save any video stream you view in your browser to your hard disk.

One that is governed by patents, and which can only be implemented in accordance with license restrictions, can be guaranteed only to be implemented in ways that enforce any existing DRM on video and audio content.

Perhaps the former is more appropriate to the <video> tag, while the latter can be left to plug-ins, though.

Originally posted by drag:Its just as likely that H.264 is violating unknown patents as Theora.

This has been argued before, but since it keeps coming up, let me try to explain it in a slightly different way:

Let's say that I have a video-related patent and the willingness to sue. If that patent were related to H.264, I would have sued in about 2007 or so: I could have sued Apple, MS, Google, Sony...just about every IT company, really. If it were related to Theora, I couldn't do that. I'd have to sit around and wait and hope that Theora is picked up by someone that as enough money to sue (and, God forbid, maybe argue that it should be picked up in standards á la Rambus). That is the difference between H.264 and Theora from a patent perspective.

Of course, the situation is even more stark for Apple and Google: If anyone can sue for H.264, they will get sued anyway. Putting H.264 in Safari and Chrome increase their exposure by exactly nothing.

Originally posted by Drazick:No one mentioned the Audio tag?Is everybody agree on Vorbis?

Frauenhofer's patents on MP3 will expire by the end of 2012, about 1 year after the final draft of HTML 5. It's likely that most browser vendors will add MP3 support for the <audio> tag then. It's also likely that Mozilla, Google and Opera will support Vorbis earlier than that.

Originally posted by drag:Its just as likely that H.264 is violating unknown patents as Theora.

This has been argued before, but since it keeps coming up, let me try to explain it in a slightly different way:

Let's say that I have a video-related patent and the willingness to sue. If that patent were related to H.264, I would have sued in about 2007 or so: I could have sued Apple, MS, Google, Sony...just about every IT company, really. If it were related to Theora, I couldn't do that. I'd have to sit around and wait and hope that Theora is picked up by someone that as enough money to sue (and, God forbid, maybe argue that it should be picked up in standards á la Rambus). That is the difference between H.264 and Theora from a patent perspective.

Of course, the situation is even more stark for Apple and Google: If anyone can sue for H.264, they will get sued anyway. Putting H.264 in Safari and Chrome increase their exposure by exactly nothing.

Additionally, any company who's part of the H.264 patent pool obviously can't sue a H.264 licensee. And that includes all the big playres with lots of compression patents, and who know the prior art well.

So, licensing H.264 gives you a quite long list of companies you know you can't be sued by, and who have a big stake in making sure others aren't.

Originally posted by jb510:Having looked at the samples on Hicksons site I have this to say. Low-bit rate Ogg looks like crap, but it's watchable, whereas low-bit rate H264 is lower than crap and totally unwatchable. The high bitrate samples look about the same to me but are still a little painful to watch.

YouTube's H.264 encoding is meh at best. They use Main Profile not High, and aren't using CABAC entropy coding.

I made some samples of what good H.264 should look like with the same source. x264 at 640x360 can beat Theora at 400x224 in per-pixel qualty. In general, the best H.264 encodes can hit the same quality level as Theora at about half to a third the bitrate. Check 'em out.

The Ogg files were made by Xiph, the others are mine.

Anyway, so as not to include too many posts in a row, let me also point folks to the MPEGLA license summaries which outline what the H.264 licenses look like. They're really not that expensive, and have pretty high minimums below which nothing is due, and then caps for maximum annual payments for most categories.

In general, I think anyone doing big enough business as to trigger a license fee is going to be distributign enough content that the compression efficiency advantages of H.264 over Theora would save them more in bandwidth costs than they'd pay in the license fees.

quote:

Originally posted by Alphonse:This is what I don't understand. Theora is a video codec like H.264. That means that hardware acceleration, outside of hardware built specifically for that purpose will be done with GPUs and shaders. Shaders are just GPU programs. Theora is a codec. Shaders can operate on codecs.

Unless there is something so unusual about Theora that it cannot be decoded on shaders, I don't see the problem. It's simply a matter of writing the proper shader algorithm.

All the GPU video decoding I've seen so far has been handled by dedicated hardware in the GPU. I've heard theorizing about doing shader-based decoding, but never a real demo of it. Even getting YUV to RGB conversion to work consistantly on the existing GPUs as part of a presentation layer is more of a hassle than it's worth.

Beyond that, these are getting displayed in the browser, and the browser can overlay things over the video like controls. Pushing video decode to the GPU would only make sense if the whole browser rendering model is being implemented in OpenGL or DirectX. Which would all kinds of awesome, but not something on anyone's short term roadmap AFAIK.

quote:

Originally posted by beoba:Does anyone have a link to Microsoft making an official statement on this, or is it just their de-facto status at the moment?

Not that I'm aware of. IE8's still pretty new, and I'd imagine a feature the scope of video would be IE9 at the earliest.

Given the complexity of media pipelines, it'd probably be a lot easier to implement good tag support in Silverlight than it would be integrating it into the Mozilla rendering model . Chrome is sucking in ffmpeg wholesale, which should be fine, and Safari just calls QuickTime, which should also be fine. But trying to get reliable A/V sync, scrubbing, pipelines, etcetera is HARD and can take years of refinement. Mozilla's going to have a fun learning experience if they think getting a codec integrated is even 5% of the work to build good media playback. And they're even pushing a codec without B-frames, which means they can't drop a single frame on decode without losing all frames until the next I-frame.

It'll be interesting to see how playback holds up with bigger video on slower machines, and with multiple video clips playing at once. There's a whole lot of pipelining and scheduling required to make sure that the frame you need is available by its display time.

quote:

Originally posted by deltatux:Sure, OGV may not be the best codec out there, but it's still scalable to H.264 and I believe by the time HTML 5 comes out OGV will be as good. So, I'm casting my vote for OGG on HTML 5 if they let us vote lol.

H.264 has improved by more than Theora has since the Theora bitstream was locked down. Xiph's done heroic work refactoring On2's...interesting encoder design, but there's only so much that can be done within the bitstream design. And it's old. No B-frames! There's certainly room for them to squeeze out further efficiency, but H.264's not a moving target. And H.265 isn't that far away, and will have another big leap in compression efficiency.

Given what the Theora bitstream syntax is capable of, it can probably match MPEG-4 Part 2 simple profile once Xiph has addressed the limitations of the VP3 codebase. If they're brilliant and lucky, maybe ASP (divx/xvid) efficiency could be reached in a few years. But compared to H.264 High Profile, Theora doesn't have anything like CABAC entropy encoding, variable block sizes, the well-tuned in-loop deblocking filter, multiple reference frames, etcetera.

The one wildcard for Theora is adaptive quantization tables. I could see that doing something surprising (again, if they're brilliant and lucky).

Originally posted by darkowl:Wait, we're still letting Opera think they're relevant with their miniscule market share? They've gone through many years and many versions and they still haven't gone anywhere significant. Who cares what they think? They'll be dead in five years, ten if you want to be conservative. The market is now between MS, Apple, Google and Mozilla.

Opera may well have a small desktop market share but it is important for 2 very good reasons. 1) it is arguably the best browser out there. 2) it has a giant user base on Wii, DS and many many many mobile phones.

In my opinion they need to stay well away from requiring users to install codecs as this completely negates the whole point of the video element in the first place. What is the point in getting rid of the need for plug-ins by creating a new need for codecs? I am also dismayed that the patent system is so badly broken that 2 codecs that have been around for some time still have no idea of whether they are infringing patents or not! Surely if they were then the patent holders should know by now and have done something about it? It re-enforces my idea that patent holders should only have a certain amount of time to challenge patents once the infringing item is in the public domain in order to stop patent trolls waiting until something has become popular. Patents were designed so that you inventors could make money from their creations not so that they could lock up an idea and then make money from others.

They should refrain from using h.264 as the licensing need makes free software much harder. If someone wants to create a browser in their bedroom and then give it away for free then they will be needing a free codec not an expensive one! If apple want ogg to be better then they are free to help improve it. Also once it is a standard then hardware manufacturers will add support for it, problem solved!

Several commenters have claimed that no system-on-a-chip supports Theora. However, at least one SoC I've worked with, TI's OMAP3430 (used in the BeagleBoard and Gumstix single board computers), can play 720p HD video at 2W total board power consumption (excluding display), yet doesn't include direct support for any video codecs. It has a built-in DSP, plus highly efficient SIMD instructions, so all playback is done in "software." You can get codec packs from TI for the DSP, and there's no reason someone couldn't write a Theora decoder for the DSP, or implement it on an FPGA (which implementation could later be turned into an ASIC -- in fact, Elphel security cameras already have a hardware-accelerated Theora ENcoder).

It seems likely that what most people are calling "hardware" decoding is actually some code running on a general purpose DSP, and anyone can implement Theora for those DSPs. Custom ASICs are so last decade.

As usual, patents are screwing everything up. MPEG-LA isn't helping its case either, with the coupling of encoder and decoder in licensing and with the upcoming changes.

Really, Theora isn't so bad. We need some least-common-denominator codec which must be freely licensable in full (including any known patents) to all and must be implemented by all compliant implementations of HTML5; and if Theora is not that codec, perhaps Dirac can be it.

EDIT: Oh, and what the previous poster said about the situation with hardware acceleration. If your hardware is a one-off, non-upgradeable decoder, you've done it wrong.

There's one fairly important point that no-one has mentioned, afaik ever. With HTML 5 we're moving towards full featured web apps. Not just run-of-the-mill video playback, but video editors, VFX, multiple layers of processed video. Sure not yet, but eventually. There's a reason VJs use older codecs for performances, because H.264 may be the most technically advanced codec out there but that comes at the cost of decoding power. Sure you've got hardware acceleration but can that handle multiple streams at once? How about on a mobile device? Older codecs, while not as efficient can be run on older hardware, or on new hardware many times over. That counts for a lot.

YouTube's H.264 encoding is meh at best. They use Main Profile not High, and aren't using CABAC entropy coding.

I made some samples of what good H.264 should look like with the same source. x264 at 640x360 can beat Theora at 400x224 in per-pixel qualty. In general, the best H.264 encodes can hit the same quality level as Theora at about half to a third the bitrate. Check 'em out.

that comparison was just to rebut the claim that using theora for Youtube would exhaust the net bandwidth.

quote:

Originally posted by benwaggoner:Anyway, so as not to include too many posts in a row, let me also point folks to the MPEGLA license summaries which outline what the H.264 licenses look like. They're really not that expensive, and have pretty high minimums below which nothing is due, and then caps for maximum annual payments for most categories.

Yep, but that is current license. The point is that they can change them as they wish after they expire.

quote:

Originally posted by benwaggoner:All the GPU video decoding I've seen so far has been handled by dedicated hardware in the GPU. I've heard theorizing about doing shader-based decoding, but never a real demo of it.

It seems to me that the main complaint against Theora is the lack of hardware-support for it. But I bet that if it became the standard for HTML5-video, we would very quickly get a metric assload of chips that have hardware-support for Theora. Yes, it would be problematic for older hardware, but things change.

Ben Waggoner raises an important point. H.265 is coming soon, with a whole new range of patents. I know there will be new patents, because that is indeed the point of the cartel who are designing the H.265 codec. Just as GM was "an underperforming pension fund that happens to manufacture cars", MPEG is a patent cartel that happens to produce codecs.

Now is the time to get off the upgrade treadmill, before all our precious H.264 hardware is rendered obsolete. Instead we can choose Theora, which as a less power hungry codec, doesn't even need bespoke hardware, and therefore can be upgraded (for free, assuming your device makers lets you) in the future.

The Theora devs have stated that they have ideas for improvements that would require breaking the spec (i.e. the equivalent of H.264->H.265) but there's no point as there's still performance to be wrung out of the current spec (which they inherited and are acutely aware of limitations in). But obviously these improvements would in the grand tradition of open source software only be produced when and if necessary. There's no big marketing reason to hold ideas (or patents) back for the next big version when you have a marketing push and get everyone hand over a bunch of cash and to junk their old computers that aren't Vista/H.264 capable.