Benefits of GPU-powered HTML5

At MIX 10 we showed how we’re building on new Windows technologies like Direct2D, DirectWrite and XPS to enable Internet Explorer 9 to render all standards-based web content – text, images, video and SVG – using the power of the GPU.

In this blog post we’ll review the major improvements for web developers and users that come from building on these Windows technologies. For more detailed information on Direct2D technologies, see this excellent PDC2008 talk.

Performance, performance, performance

The benefit of building on Direct2D technologies is that the browser makes the most of the underlying PC hardware that is optimized for rendering rich graphics. This results in faster web applications and a higher quality browsing experience for users and web developers.

As IE9 does more work using the GPU, there is less CPU load, enabling other browser subsystems to do more, as well as enabling higher frame rates for smooth animation and video playback.

The GPU is a much better choice for some graphical operations – for example, the GPU executes alpha blending and bilinear image scaling much faster than the CPU, and uses pixel shaders to perform complex per pixel calculations.

Super-fast zoomed browsing

IE9 uses the GPU to scale images and other content, making zoomed browsing very fast – this is what makes the map zooming demo on ietestdrive.com so fast.

Windows is still the only broadly used operating system that allows the users to change the size of all UI elements on the screen to improve readability and legibility on new high DPI displays on laptops and desktop computers. IE9 builds on the work done in Internet Explorer 8 (the first browser to zoom Web page content by default) to ensure that users can easily read the Web on high DPI displays.

Hardware accelerated HTML5 video using Windows Media Foundation

IE9 makes the most of your graphics hardware by using the Windows Media Foundation system to play HTML5 video, using the CPU or the hardware video decoder if available.

The reduction in CPU usage on machines with hardware video decoders greatly improves battery life – for example, in our MIX demo we played two 720p HD videos, using barely 30% CPU on a $400 netbook. (versus 100% CPU usage in other browsers, playing only one HD video while dropping frames.)

The IE video engine decodes video directly with and into the GPU. Once the video frames are decoded, they can be treated like any other bitmap in the graphics pipeline with full compositing and integration into the rendering system.

In addition to being up to 30% faster than IE8 decoders, the new WIC decoders understand embedded color profiles in images, making IE9 a color-managed browser that understands ICC v2 and v4 profiles.

Text quality and performance

IE9 uses the GPU (via DirectWrite) to do text output – up to twice as fast as IE8, and with higher quality. Text can be smoothly animated in IE9, and sub-pixel positioning is a more faithful representation of the Web (and font) designer’s intent.

(We’ve also heard your feedback about some fonts showing up ‘blurry’ on some systems; we are working on addressing this.)

High quality graphics printing

To do high quality printing of HTML5, you need a high quality print subsystem. Internet Explorer 9 directly converts web content into XPS format when sending output to the printing system.

XPS is a more modern print system with native support for features such as opacity and complex paths, which results in increased fidelity and quality when printing modern web content.

What if my PC doesn’t have a GPU?

Because IE9 is built on Direct2D, it has full software fallback support for every feature. The goal is to improve graphics performance and fidelity even for the small number of machines in the world without GPUs, via high quality software emulation.

New HTML5 experiences – with the power of the GPU

Taken together, all of these GPU-powered capabilities in IE9 make it easier than ever before to create amazing new class of Web applications using same markup. For example, it took just an hour to create the final MIX demo – a rotating carousel of four translucent videos (at the 28:00 mark in the video). The page uses simple HTML and JavaScript - the same markup that runs in other browsers, but with higher quality and performance made possible by harnessing the power of the GPU.

More to come

We are working with our GPU hardware partners to ensure that every Windows user, using any hardware, has a dramatically improved experience when browsing the Web. We want to enable web developers to create a new class of GPU-powered rich web applications that go far beyond what is possible with today’s browsers.

So IE 9 can use JPEG XR: Yay! JPEG XR is software patented (eugh), unusable in open source products (double eugh) due to the implementation’s license specificly preventing modification of the implementation’s code, including redistribution in software that can be modified, and typical uses don’t show much improvement over ‘classic’ JPEG (Yet Another ‘eugh). No yay after all.

Do you think users and other browser makers will risk the same hell that Compuserve created with GIF?

And on top of that, IE 9 won’t support Ogg, nor Vorbis. Patented technologies FAIL.

So given that IE9 makes use of the Windows Media Foundation framework, I take this to mean that IE9 will playback HTML5 video with whatever codec is installed on the system. Will IE9 also bundle Theora and Vorbis to make open video and audio support simpler for the end user?

That doesn’t matter, Frank Olivier is bragging about it on the very post.

Interesting enough reading this part:

"What if my PC doesn’t have a GPU?

Because IE9 is built on Direct2D, it has full software fallback support for every feature. The goal is to improve graphics performance and fidelity even for the small number of machines in the world without GPUs, via high quality software emulation."

Note that it has "full software fallback", you can do it for D2D but can’t use GDI for supporting 55%+ of the world (XP users). I couldn’t even test Ie9 yet, because I don’t have either Vista or 7. And won’t be able to have one, costing about R$600 in my currency. (around US$300)

"ryan: Incorrect. You should watch that video too. They support the MP4 container with h264 video only. For audio, AAC and MP3. They don’t support ogg, vorbis, theora, quicktime, real, WMV, WMA, etc."

I watched the video. During the question and answer session there was no emphatic ruling out of Theora support beyond an "I don’t believe so", and prior to that there was some considerable confusion from the program manager about which container formats were in fact supported. The question of whether IE9’s use of the Windows Media Foundation means any installed codec can be used in conjunction with HTML5 video or audio in IE9 was not resolved by the video.

What do you think the acid3 test actually means? Most of the real web devs say very little, and the respected columnists do too (e.g. http://blogs.zdnet.com/Bott/?p=1896&tag=col1;post-1896 — "controversial acid3"). There are other browsers with higher acid3 scores that can’t do borders right 🙁

Speaking of high DPI displays (which BTW aren’t that new, e.g. Dell built them into premium laptops for years), are there any plans on moving away from the 96-DPI rule? Making pt or cm relative to the DPI (and therefore absolute to the real world)?

I may have found a possible bug: on Windows 7 32 bit, when Microsoft Security Essential’s real-time protection is enabled, an orphaned iexplore.exe process keeps running in the background after closing every IE8 window for a variable amount of time (usually about a minute). This process is small in size (usually 3-7 MB)

Instead, every iexplore.exe process is closed immediately (as it should be) if Security Essential’s real-time protection is disabled.

I have an ION based system (my media center) and a none ION Intel Atom netbook. I can confirm that IE9 runs a lot faster with ION.

Great work, not only does IE9 catch up, it sets a new standard.

As enterprise system administrator, I know IE is the only enterprise able webbrowser. (try packaging the latest official release of firefox or chrome using App-V or ThinApp…, or lockdown via GPO’s/GPP, and patching via WSUS)

Keep up the good work, and hopefully release an IE9 version that includes a user interface soon.

p.s.

One tip, please do not include default favorites in IE9, it’s quite difficult to remove the default ones in IE8 for new users.

Roaming favorites (e.g. via Mesh) would also be very appreciated for home users

Not sure what you are asking here – Windows 7 setup will look up the native resolution and physical size of your display, and set the appropriate DPI; IE8 and IE9 will resize all web page content to this setting. All of this zoom scaling is done without the web page author having to do anything, but you can (if you want to ) look it up with http://msdn.microsoft.com/en-us/library/ms533721(VS.85).aspx

@Johm Piddlemeyer – re: "I’m sure Microsoft would be happy to support a truly ‘free’ patent-safe good quality codec – if one existed."

I would bet thousands that Microsoft would never ever support a trully ‘free’ patent-safe codec!

It goes totally against their business plan.

Users want free and open codecs so that they don’t have to suffer the wrath of DRM that doesn’t work – but Microsoft wants to sell their software on their OS/devices so that their business customers can abuse DRM to provide higher profit margins at the expense of usability and happy customers.

I can’t speak for everyone – but I will have absolutely no part in any codec that enforces DRM in any way shape or form whatsoever. It goes totally against my ethics and code of conduct.

@stanley Huh? There is no DRM in IE9, h.264 or HTML5. Microsoft showed IE9 playing youtube.com html5 video (h.264 format) at the MIX conference – no DRM there – just plain old html5 video.

@Ryan By default, dailymotion serves up video using (drum roll..) FLASH! Hardly a good argument for Theora. How many visitors to dailymotion get Theora?

Good for Wikipedia for trying to serve up only ‘open’ video – but I bet that their time and their/your money is better spent on using h.264 instead of Theora (h.264 uses less bandwidth; has vastly better tools; works on mobile phones; etc etc)

Millions and millions of iPhones and iPod support h.264. Any Google is only now starting work on making Theora work on ARM chips? (Note that this is not even a hardware implementation…how many more years will it take before anything more than a tiny slice of the general population can watch a Theora video on their mobile phone.)

There are plenty of reasons to not support Theora – it doesn’t work on mobile. No industry support (no AV tools export to Theora). No consumer cameras produce Theora (my cheapo camera produces h.264 files…)

Don’t like Microsoft’s decision? Use Firefox and don’t bother the rest of us.

@Piddlemeyer: h.264 costs 5 million bucks (and I think it’s PER YEAR) for a content distributor license. I’m not sure Wikipedia has this sort of cash on hand.

How long would it take for the 20% less efficient Theora encoded content to reach that extra cost in bandwidth costs?

Reading the release note for TheorARM 0.0.3 (the Theora fork intended for ARM tuning pending reintegration upstream), it is already capable of fully decoding 320×240 video at more than full speed on a 400MHz ARMv4 device with no video acceleration whatsoever (55% of the CPU load is spent on YUV to RGB conversion on un-postprocessed video).

Your average mobile phone includes a 320×240 screen. If it can playback video, chances are that it includes a hardware YUV to RGB converter. The iPhone has a 480×320 screen, with YUV to RGB and downsampling done in hardware, for example.

@Wurst Sure, EOT fonts have some ‘DRM’ – type designers (just like most rational human beings) don’t want hundreds of hours of their work stolen. EOT DRM is hardly a burden on the average web browser user.

You’ve made great progress! I like that you are transparent about your Test Suites, even pitting IE9 agains IE8 itself, too.

I also like that you linked to 3rd party browser tests like CSS3 Test Site, ACID 3, and International Color Consortium. These public test pages should be the baselines for all web developers (NOT just ACID 3)

REQUESTS:

1. In your next Platform Preview, please include this HTML5 Testing page:

@hAl: The license that’s royalty free until 2015 seems to be regarding website hosted content. Reading some of the materials on the MPEG-LA website, it’s not clear if this actually permits a royalty-free _decoder_ until 2015. Can you provide citations showing that the royalty exemption applies to browser H.264 decoders?

All browsers passed at least 100 out of 160. Well except for IE8 that passed a pathetic 19. The good news is that IE9 Platform Preview with its GPU enhanced rendering passed a whopping 19 out of 160 tests — oh wait that’s an improvement of zero percent on an originally horrible score.

Thanks for trying out for the Future of the Web team! – don’t call us, we’ll call you.

@Nick, as mentioned during the keynote, IE9 Preview #1 does not support the Video or Audio tags; they’re coming in a future preview build.

Others on the team are more expert in the exact demands of HTML5, but as far as I know, the spec requires no particular codec to be supported, suggesting that testing for particular codecs leads to a misleading score.

I would kindly request that a little thing that’s been annoying me about the scroll bar be fixed. I use to browse the interwebs with my IE8 window maximized, and often, in a reflexive manner, i tend to swing my pointer all the way to the right to grab hold of the scroll bar and scroll. The problem is that the scroll bar itself ends just short of the rightmost part of IE’s window, and so I have to move the pointer a couple of pixels to the left before I can start scrolling.

It’s not the whole world, but I sure would appreciate it if this was fixed. Oh and good work on IE9 so far! *thumbs up*

@hAL: and in 2015, Wikipedia will have to re-encode all their videos, because I betcha the MPEG won’t set such a golden goose slide.

For a precedent, see Compuserve and GIF compression (that caused the development of the PNG format). Also, Fraunhofer IIS/Thomson and the MPEG 1 layer 3 audio compression algorithm (that caused the development of the Vorbis codec – along with mp3’s low quality).

@hAL: wavelet-based compression has been under investigation for quite a while now – and it isn’t much more advanced than, say, Theora. It does have much better potential (lossless), but it seems the power required for both encoding and decoding makes it unsuitable for many uses. Moreover, the main sponsor decided to improve the specification through a new version that contains extensions.

In comparison, Theora saw huge improvements in quality recently without any need to tweak the decoder – which is one strength of that particular codec : since they froze the data stream, it doesn’t matter if you use the last CVS extract or an early beta of the decompressor, you’ll be able to read any file that conforms to the Theora format. For Wikipedia’s use (encode once, use many many many times for a long long long time), that makes more sense than Dirac.

On other elements (such as streaming access to the BBC’s library, which is more than probably stored as lossless high definition content), Dirac is indeed far more sensible in the long run.

The situation also has to be compared with MPEG-4 ASP, of which NO implementation was able to read another implementation’s output without artifacts (if at all) for quite a long time (until DivX and Xvid actually got to become the unofficial reference implementations). MPEG-4 AVS mitigated that somewhat, by starting directly with a single reference implementation that also got used ‘in the wild’: x264.

The problem with patents is that, depending on the state of the law at the time (it can change from place to place too), they may actually last longer than planned. It is thus a good idea to make use of the oldest public domain algorithm as possible, as the effect of submarine patents on it will last less – for example, with Theora’s specification having been completed on June 2004, according to US law, no patent filed after June 2005 could be enforced against it. At best, patents filed before may be valid ‘only’ until 2024-2025.

On the other hand, patents on the Dirac format, due to the format still being extended, will remain a risk until 2030 and later.

Remember, Theora is based on VP3.2 (and is capable of playing VP3-encoded content). VP3.2 was released in late 2000, so any submarine patents for VP3.2 would actually expire in 2020, and that only applies to algorithms that weren’t on On2’s previous codecs. Only patents that involve the changes to VP3 added by Theora would last beyond 2020. So, as time goes on, Theora becomes a harder target to hit, patentwise.

I’ve noticed a new test on the Preview demo site: ICCv2 and ICCv4. I find it confusing:

– current IE9 preview fails the ICC test: the top and bottom of the mesa don’t have the same alpha levels. If I look at the reference image (or enable ICC support in Firefox 3.6), the top and bottom should have identical levels. Will that be fixed in the next preview?

– the sea slug photo pages are structured differently for IE 9 preview and other browsers: I can’t find for the life of me the actual reference image.

Will ICC profiles be enabled only in IE 9 mode, or will they enabled in all modes (5,7,8,9)? It’s been disabled in Firefox because many websites display images with profiles while these are not intended to apply, so I guess the former, but can you confirm?

@Mitch74 There can be no patents on any format, there is just the possibility that there are patents on the implementation of a decoder/encoder. (BTW: before the patent on the lossless compression algorithm used in the GIF image format expired there existed ways of writing decoders that did use the patented technique)

@wurst: no, sorry: the patent can be on the format. Actually in GIF, the patent was on a specific compression technique: the Lempel-Ziv-Welch algorithm. You could implement the GIF format if you weren’t using that compression method, but that meant that your implementation couldn’t read (or create) LZW-compressed images.

On MPEG-4, the reason Xvid (and x264, and LAME, etc.) exists at all is that in most countries, it is considered that math equations can’t be patented (and source files are assimilated to math equations) but distribution and use of a compiled form are subject to patents…

As a matter of fact, you can’t find binary versions of these codecs on the projects’ websites (Xvid’s win32 binary is found on an external website, which is hosted out of the US).

Tangentially related to the question about adding support for additional image types before:

Will IE9 finally prevent Quicktime from usurping control of standalone PNG images?

I have Quicktime set to not handle .PNGs, and for all that the MIME Types button on the current Quicktime control panel is still broken and opens the wrong thing, for some reason opening the Quicktime control panel through the Quicktime Player instead gives me a version with a working MIME Types button and I have it set to not handle .PNGs in there as well. Yet when I open a PNG link in a new browser window, I have to wait around for Quicktime to load up and mess up the image handling.

If I disable the quicktime addon in IE then try opening the link it causes the page to come up as a broken image requiring quicktime addon to display!