Introducing the Azure project

Motivation

Firefox 4’s graphics performance is great, and some last-minute “low-hanging fruit” inefficiencies Robert O’Callahan found made us even faster on some demos in Firefox 5. We’re not content with “great”, though, and our investigations into how to make drawing even faster have revealed that some of our choices in Gecko’s graphics engine aren’t optimal for performance.

How Mozilla renders

To display a web page, Gecko’s layout engine looks at how that page is constructed, and divides it into independent layers. Each of those layers is drawn using Cairo, a 2D graphics API built on top of the operating system’s graphics APIs. Cairo’s drawing into layers can be hardware accelerated — this is known as “content acceleration” — or done in software. Right now, content acceleration is fastest on Windows Vista and 7, where we use Direct2D, but we want to be able to extend good acceleration to all operating systems, including mobile.

After drawing, layers are composited together using either the platform’s 3D APIs (OpenGL or Direct3D) or Cairo, and the result is displayed on the screen.

Format conversions

Cairo uses a fixed point representation of numbers. Unfortunately, this means that when Cairo calls get mapped to an API like Direct2D, which uses floating point numbers, two conversions are required: one from from your internal floating point representation to Cairo’s fixed point representation, in which it is initially stored, and then back to floating point again when Cairo makes Direct2D calls.

Statefulness

Cairo is a stateful API. This means that any given function call depends on the results of the previous function calls to determine its result. Unfortunately, statefulness strictly adds overhead to an intermediate graphics API: if it maps to another stateful API, Cairo has to track its own state separately from the underlying API (usually due to subtle differences between the two). Also, Cairo uses a stateless surface API internally; whether the underlying API is stateful or stateless, there’s always a conversion from one to the other. Further, the HTML5 Canvas element is a stateful API that’s built on top of Cairo, and it has to have its own state tracking too.

What makes this worse is that Gecko is better-suited by a stateless API; when we begin rendering any part of a web page, we need to reset most of Cairo’s stored attributes, since they’re determined by the element’s CSS.

What can we do about it?

We would very much like to remove these unnecessary conversions and state tracking. We’d also like to be able to hardware accelerate content drawing in an easy and cross-platform way.

This API needs to be close enough to Direct2D to be little more than a thin wrapper, but general enough to be implemented on OpenGL and Direct3D 9.

The Azure project

To accomplish this, the Mozilla graphics team is creating a new 2D graphics API. It’s going to be stateless, and significantly closer to Direct2D. We’re also going to develop new “3D” backends for drawing content in an accelerated way to OpenGL, Direct3D 9, and Direct3D 10.

The backends

Once complete, the new 2D API will be implemented by several different backends, depending on OS, hardware, and driver. We will also have a fallback implementation, using Cairo, that we’ll use when we don’t have a more specialized backend available.

Direct2D

We don’t make the best use of Direct2D right now. Part of it is the floating/fixed point impedance mismatch I described earlier, but part of it is capabilities Cairo exposes that Direct2D doesn’t implement. These need to be worked around or emulated using software. The API we create as part of the Azure project will be close enough to Direct2D to not require fallbacks.

Quartz / CoreGraphics

On OS X, Cairo uses its Quartz backend for all drawing. Quartz can be fast: Safari is faster than Firefox on some demos, even though we both use the same backend for drawing. We believe one reason this is the case is because Quartz is stateful: Cairo needs to convert from its stateful API to its internal stateless surface API, then back to Quartz’s stateful API. We’ve designed Azure’s stateless 2D API so that it can map from stateless to stateful in a more optimal, and optimizable, way.

Cairo

Cairo is our failsafe implementation. It’s the backend we’ll use before others are ready or when we explicitly choose not to use hardware acceleration, for example because of driver issues. We also need Cairo for printing support.

It’s possible that we will decide to modify our internal copy of Cairo to allow us to use the internal stateless surface API directly, since we think Cairo’s surface API is a pretty good match to Azure’s 2D API.

3D Backends

The goal of the “3D” backends is to implement Azure’s 2D API on top of the common 3D APIs (OpenGL and Direct3D) so we can have content acceleration on Windows XP, OS X, Linux, and mobile. Our intent with these 3D backends is to be fully cross-platform, including Windows 7. This lets us better control our performance characteristics and bugs, rather than spending a lot of time and energy working around Direct2D bugs.

Electrolysis and Azure

The Azure 2D API will make it possible for us to hardware accelerate content drawing, even in the distant future when we “sandbox” our content processes. This new 2D API is required for hardware accelerated sandboxed content processes, but isn’t required for the initial Electrolysis roll-out.

Timeline and checkpoints

Over the past few weeks, we created a proof-of-concept Direct2D backend to test our hypothesis that we could create a faster non-Cairo API. Our hypothesis was correct — our backend is significantly faster than the Direct2D Cairo backend on some commonly-used Canvas demos and benchmarks. This work is all available in the graphics project branch.

Our initial success led to our current plan: to design and test this new 2D API as a backend for the HTML5 canvas element. The graphics team has included getting the Azure Direct2D canvas backend enabled in mozilla-central as a Q2 goal.

We are also in the middle of writing similar canvas backends for Cairo and Quartz, both of which are already in the graphics project branch, but those are not yet operational. We haven’t defined specific time goals for those backends.

Postscript: Why “Azure”?

Jim Clark invented the first GPU at Stanford in 1979, called the “Geometry Engine”. He founded SGI based on this work.

SGI made many workstations and servers with “colour” names like Onyx, Indigo, and Crimson. They are also the original inventors of OpenGL.

After SGI, Clark founded a little-known software company called Netscape.

It seemed only fitting to pay homage to him in this small way!

This entry was posted on Tuesday, April 26th, 2011 at 1:39 pm and is filed under acceleration, development, firefox.
You can follow any comments to this entry through the RSS 2.0 feed.
You can skip to the end and leave a comment. Pinging is currently not allowed.

“This API needs to be close enough to Direct2D to be little more than a thin wrapper” – does that mean Windows will always have an advantage over OS X, Linux and Android as Azure on Direct2D will have the least amount of overhead?

Windows won’t have an advantage, except in that Direct2D already exists and we still have to write the 3D backends. We think the Direct2D API is a good one to emulate when we write our own 2D-on-3D rasterizer.

Reason is simple – supporting an open standard (OpenGL) instead of Direct3D. Mozilla usually has a strong stance on open standards. Understandably, there might be concerns of lacking or poor OpenGL drivers, vs. better supported Direct3D, but at least OpenGL should be available as an option, for those who prefer open standards.

Maybe it should be written [azyr] so that these english speaking folks manage to pronounce it (somewhat) right. This would distinguish it from the Microsoft thing. When I saw a commercial for Windows Azure I thought “What the h*** is this Windows Escher? Why does Microsoft name a product after a famous artist?”, until I realized that english people cannot pronounce french words at all. (Also I heard one pronounce d’Artagnan something like “Dartainion”. lol)

I agree that Azure is confusing. For the longest time I kept thinking Microsoft, Microsoft… until I saw this article. It may not violate any trademarks but its sure confusing!

Also I’m confused about Azure and Direct2D vs Azure and Direct3D. Wikipedia has only confused me. Why are your using Direct2D and not Direct3D? What are advantages / disadvantages of both approaches? What does IE9 use?

I agree with the uncertainty of naming this project Azure. I believe Microsoft is choosing that as their overall name for their “cloud initative” so that they will be naming some ten things “Azure (version of ____)”.

Checking above FF appears just to be doing a fun little “shout out to Jim Clark’s penchant for colors”, so any color name will do.

> distant future when we “sandbox” our content processes
How “distant” are we talking about here?

In any case, good stuff, though this is a massive undertaking. I don’t know how many rendering primitives you need, but subpixel-AA’d font rendering sounds like it’ll be quite a bit of work (moreso if you aim to match each platform’s rendering).

I have been wondering about this in a big way since my biz is tied to browser differences. How is this going to change cross browser testing? Meaning are the different backends going to render differently in terms of layout… I would almost think they would… depends on where the render is at exactly. Is the layout engine separate? What I am hoping will not happen is that ff5 in direct2d will looking different than ff5 in opengl….
Any thoughts on that?

Layout is, indeed, a separate module, and we have very extensive tests that make sure it stays the same, so except for some minor font rasterization differences, rendering should be exactly the same from one OS to the other. No need to worry!

[…] ready to move beyond the planning phase and begin coding. The new graphics core has been dubbed Azure, an homage to SGI innovator Jim Clark — who invented the first GPU in 1979. Azure’s aim is to […]

[…] it all at once. Mozilla also announced enhancements for the Gecko layout engine in a separate post by developer Joe Drew. Project Azure, as it is known, focuses on the creation of a new 2D graphics API for Firefox […]

Surely it would be better to add the features you need (stateless and floating point) to cairo?
Then it would not just be Mozilla benefiting, but all the users of cairo.
It would still be a large undertaking, but if cairo really wants to treat Windows as a first-class platform, they should be willing to accept such a change.

Isn’t it a bit short-sighted to *now* consider a cross-platform graphics API built around Direct*D when Windows installs are waning and the rest of the planet use OpenGL?
I think my issue is azure being close enough to direct2d to cause (even minor) performance hits on opengl.

We think that Direct2D is a good API for general hardware acceleration of 2D drawing, which is why we want the Azure 2D API to be reasonably close to it. Direct3D and OpenGL are close enough to not make a difference w.r.t. implementation of a 2D API on top of it.

Keep in mind that we eventually want to be able to choose between Direct2D and our Azure “3D” backends.

Oh please, pleae rename this project. Not only Microsoft has a project running with this name, but so do I.
I’m about to develop a game called “Azure: Infinite Skies” (http://azure.tuxfamily.org) which name gets more and more confused :-S

Do you plan to make the 3D backend configurable for Azure? I.e. let’s say on Windows, where Direct3D and OpenGL are available both, would user be able to use OpenGL as Azure’s backend as an option? (This at least would provide an option of not depending on proprietary APIs).

I quickly read your post and the comments so sorry if I missed a point but it seems to me you took the slippery road of rewriting yet another 2D API. Since you already relied on Cairo, wouldn’t it have been a huge + for Mozilla in terms of coding effort and a huge + for OSS in terms of code sharing and reuse, to help improve Cairo like preparing a future stateless version of Cairo’s API and contributing efficient backends… dunno, just wondering! Well, the Cairo folks can always look into Azure’s code later on anyway, can’t they?

[…] front, the huge 20% speed-up is due to the addition of a new graphics backend in Firefox 7 called Azure, which is a unified 2D graphics API that Firefox can use across every platform. At the moment, […]

[…] front, the huge 20% speed-up is due to the addition of a new graphics backend in Firefox 7 called Azure, which is a unified 2D graphics API that Firefox can use across every platform. At the moment, […]

[…] front, the huge 20% speed-up is due to the addition of a new graphics backend in Firefox 7 called Azure, which is a unified 2D graphics API that Firefox can use across every platform. At the moment, […]

[…] ability. With better 2D graphics, user will have more chances to experience high quality graphics. It’s going to be stateless, and significantly closer to Direct2D. We’re also going to develop new […]

[…] than before, which can, for example, make a difference when playing games in the browser. The technological basis for the improvement is the Azure API, which works with Direct2D under Windows and is expected to be […]