Project Silk

Editor’s Note:An earlier version of this post appeared on Mason Chang’s personal blog.

For the past few months, I’ve been working on Project Silk which improves smoothness across the browser. Very much like Project Butter for Android, part of it is finally live on Firefox OS. Silk does three things:

Align Painting with hardware vsync

Resample touch input events based on hardware vsync

Align composites with hardware vsync

What is vsync, why vsync, and why does it matter at all?

Vertical synchronization (vsync) occurs when the hardware display shows a new frame on the screen. This rate is set by specific hardware, but major displays in the US occur at a rate of 60 times a second, or every 16.6 ms (milliseconds). This is where you hear about 60 frames per second, one frame each time the hardware display refreshes. What this means in reality is that no matter how many frames are produced in software, the hardware display will still only show at most 60 unique frames per second.

Currently in Firefox, we mimic 60 frames per second and therefore vsync with a software timer that schedules rendering every 16.6 ms. However, the software scheduler has two problems: (a) it’s noisy and (b) it can be scheduled at bad times relative to vsync.

In regards to noise, software timers are much noisier than hardware timers. This creates micro-jank for a number of reasons. First, many animations are keyed off timestamps that are generated by the software scheduler to update the position of the animation. If you’ve ever used requestAnimationFrame, you get a timestamp from a software timer. If you want smooth animations, the timestamp provided to requestAnimationFrame should be uniform. Non-uniform timestamps will create non-uniform and janky animations. Here is a graph showing software versus hardware vsync timer uniformity:

Wow! Big improvement with a hardware timer. We get a much more uniform, and therefore smoother, timestamp to key animations off of. So that addresses problem (a), noisy timers in software versus hardware.

With part (b), software timers can be scheduled at bad times relative to vsync. Regardless of what the software does, the hardware display will refresh on its own clock. If our rendering pipeline finishes producing a frame before the next vsync, the display is updated with new content. If we fail to finish producing a frame before the next vsync, the previous frame will be displayed, causing jankiness. Some rendering functions can occur close to vsync and overflow until the next interval. Thus, we actually introduce more potential latency since the frame won’t be displayed on the screen anyway until the next vsync. Let’s look at this in graphic form:

At time 0, we start producing frames. For example, let’s say all frames take a constant time of 10 ms. Our frame budget is 16.6 ms because we only have to finish producing a frame before the next hardware vsync occurs. Since frame 1 is finished 6 ms before the next vsync (time t=16 ms), everything is successful and life is good. The frame is produced in time and the hardware display will be refreshed with the updated content.

Now let’s look at Frame 2. Since software timers are noisy, we start producing a frame 9 ms from the next vsync (time t=32). Since our frame takes 10 ms to produce, we actually finish producing this frame at 1 ms AFTER the next vsync. That means at vsync number 2 (t=32), there is no new frame to display, so the display still shows the previous frame. In addition, the frame just produced won’t be shown until vsync 3 (t=48), because that’s when the hardware updates itself. This creates jank since now the display will have skipped one frame and will try to catch up in the upcoming frames. This also produces one extra frame of latency, which is terrible for games.

Vsync addresses both of these problems since we get a much more uniform timer and the maximum amount of frame budget time to produce a new frame. Now that we know what vsync is, we can finally go on to what Project Silk is and how it helps create smooth experiences in Firefox.

The Rendering Pipeline

In super simplified terms, Gecko’s rendering pipeline does three things:

Paint / draw the new frame on the main thread.

Send the updated content to the Compositor via a LayerTransaction.

Composite the new content.

In an ideal world, we’d be able to do all three steps within 16.6 ms, but that’s not the case most of the time. Both steps (1) and (3) occur on independent software timers. Thus, there is no real synchronizing clock between the three steps, they are all ad hoc. They also have no relation to vsync, so the timing of the pipeline isn’t related to when the display actually updates the screen with content. With Silk, we replace both independent software timers with the hardware vsync timer. For our purposes, (2) doesn’t really affect the outcome, but is presented here for completeness.

Align Painting with Hardware Vsync

Aligning the timer used to tick the refresh driver with vsync creates smoothness in a couple of ways. First, many animations are still done on the main thread, which means any animation using timestamps to set the position of an animation should be smoother. This includes requestAnimationFrame animations! The other nice thing is that we now have a very strict ordering of when rendering is kicked off. Instead of (1) and (3), which occur at separate synched offsets, we start rendering at a specific time.

Resample Touch Input Events Based on Vsync

With Silk, we can enable touch resampling, which improves smoothness while tracking your finger. Since I’ve already blogged about touch resampling quite a bit, I’ll keep this short. With Silk, we can finally enable it!

Align Composites with Hardware Vsync

Finally, the last part of Silk is about aligning composites with hardware vsync. Compositing takes all the painted content and merges it together to create the single image you see on the display. With Silk, all composites start right after a hardware vsync occurs. This has actually produced a rather nice side benefit — the reduced composite times seen here:

Within the device driver on a Flame device, there’s a global lock that’s grabbed when close to vsync intervals. This lock can take 5-6 ms to get, greatly increasing the composite times. However, when we start a composite right after a vsync, there is little contention to grab the lock. Thus we can shave off the wait, therefore reducing composite times quite a bit. Not only do we get smoother animations, but also reduced composite times, and therefore better battery life. What a nice win!

With all three pieces, we now have a nice strict ordering of the rendering pipeline. We paint and send the updated content to the Compositor within 16.6 ms. At the next vsync, we composite the updated content. At the vsync after that, the frame should have gone through the rendering pipeline and will be displayed on the screen. Keeping this order reduces jank because we reduce the chance that the timers will schedule each step at a bad time. In a best-case scenario in the current implementation without Silk, a frame could be painted and composited within a single 16.6 ms frame. This is great. However, if the next frame takes 2 frames instead, we’ve just created extra jank, even though no stage in the pipeline was really slow. Aligning the whole pipeline to create a strict sequence of events reduces the chance that we mis-schedule a frame.

Here’s a picture of the rendering pipeline without Silk. We have Composites (3) at the bottom of this profile. We have painting (1) in the middle, where you see Styles, Reflow, Displaylist, and Rasterize. We have Vsync, represented by those small orange boxes at the top. Finally we have Layer Transactions (2) at the bottom. At first, when we start, compositing and painting are not aligned, so animations are at different positions depending on whether they are on the main thread or the compositor thread. Second, we see long composites because the compositor is waiting on a global lock in the device driver. Lastly, it’s difficult to read any ordering or see if there is a problem without deep knowledge of why / when things should be happening.

Here is a picture of the same pipeline with Silk. Composites are a little shorter, and the whole pipeline only starts at vsync intervals. Composite times are reduced because we start composites exactly at vsync intervals. There is a clear ordering of when things should happen. Both composites and painting are keyed off the same timestamp, ensuring smoother animations. Finally, there is a clear indicator that as long as everything finishes before the next Vsync, things will be smooth.

Ultimately, Silk aims to create a smoother experience across Firefox and the Web. Numerous people contributed to the project. Thanks to Jerry Shih, Boris Chou, Jeff Hwang, Mike Lee, Kartikaya Gupta, Benoit Girard, Michael Wu, Ben Turner, and Milan Sreckovic for their help in making Silk happen.

Well, it would be nice if this actually works — because right now, Firefox is hopelessly busted regarding VSYNC. There are even bugs reports showing that people are able to achieve VSYNC synchronization in JavaScript in Firefox, while Firefox can NOT achieve synchronization in native code.

The true VSYNC test to pass is http://www.vsynctester.com while running on hardware that is a couple of years old — Because the speed of new hardware masks tons of internal problems!

Why? Because just because you can precisely render aligned to VSYNC is meaningless — unless those rendered frames actually end up in *UNIQUE* VSYNC intervals on the screen.

I’ll answer the first two question first: Silk is coming to desktop on Windows and Linux and should arrive by the end of Q2 2015 on Nightly. 6 Weeks after that, it should arrive on the Firefox Developer edition.

In response to Scrambler. You are right, it will only work work if the frames end up in unique vsync intervals. Silk does that, we composite, which shows the actual frame, once per vsync interval. From bug 1080869, and tested on a 2 year old machine, we pass the vsynctester.com test with flying colors on Windows 7.

Mason, as documented at http://www.vsynctester.com, running vsynctester.com on newer hardware is meaningless (an invalid test), as newer hardware (even only two year old hardware) is so fast, the speed of the system (mostly the GPU) completely masks and hides internal web browser timing problems.

I see lots of FF vsync problems on older hardware, but I do not see vsync issues on newer hardware.

You need to test on hardware from 2011, which easily supports hundreds of fps, but where FF ACTUALLY HAS vsync problems. Then, run with Silk and test to see if VSYNC is fixed. If so, great, but if not, then the true underlying problem and cause of the vsync issues were not well understood.

I would offer my system for testing, but Mozilla in its draconian internal processes, rather than work around a very serious bug I found and reported, has instead decided to ‘fix’ my bug by blacklisting my video driver. FF is now in the Recycle Bin on that system. IE and Chrome work great on that system. I was told by a FF manager in a private email (as an excuse) that they were shocked by the bug I found because it never turned up in internal pre-release testing. But all that clearly points out is that NO ONE within FF is testing FF on older systems (only nice new systems).

The only true valid test that Silk is actually any better is to find a system where the ‘old way’ fails, but the new way (Silk), fixes the problem. Do you agree?

@Scrambler
while I can understand your frustration, and also hope that those vsync problems get fixed eventually, I don’t think such a passive agressive and condescending attitude is going to get us any close to that goal.
I like the work you are doing with vsynctester and Mozilla is doing with Project Sync. And ultimately you two have a similar goal, so wouldn’t working closer together be more beneficial?

@Mason
I was curious whether the current vsync problem also affects the playback of youtube videos using the html 5 player. I’m having a lot of vsync troubles on one of my machines, but I have no idea what the problem could be. Can I rule Firefox out?

@Yggdr5 – Hmm, interesting. What kind of vsync trouble, like tearing? Can you try on other browsers such as IE / Chrome and see if you see the same issues. If it is just a vsync issue and not a rendering problem where Firefox is too slow, Silk should fix your bug.

I read your description, and even on my new machine, without silk, Firefox fails the test pretty badly. The animation already has jankiness, the inter-frame time is always red, the render-time is always super spikey. We fail even on this two year old machine without silk. The inter-frame time isn’t really consistent. The purple late setTimeout like you said is very janky, even on a high end macbook pro. Like you have found, the test fails very badly on Firefox, even on high end hardware.

Like you said, the only true way to validate Silk is to see if it is better on a system where the current test fails. The current test fails on Firefox on pretty much every machine I’ve tried. It passes on the same machines with Silk enabled. Here is a picture of how we’re doing on master, without silk:

We still can improve some more, but this machine was pretty janky on Firefox without Silk but smooth on Chrome/IE. With Silk and Firefox, it was pretty smooth, so I do think it has improved quite a bit!

Mason, that is great (and very welcome) news. Thank you for your efforts — it will make a huge difference for a lot of people (especially in the gaming communities).

I guess my experience with FF is completely the opposite. I find it very hard to find a machine the causes FF to fail at the vsync test (because most machines I have access to are modern, and have a very fast GPU). But my personal notebook PC at home always fails (a ‘fast’ Dell XPS 17 from 2011). Sadly, I won’t be able to test Silk on that since FF now has my graphics driver blacklisted.

As an additional stress test, you may want to try the ‘double frame rate stress test’, newly documented at vsynctester.com/help.html. Chrome is able to pass that test, even on my old notebook from 2011. Hopefully FF+Silk can?

Yes, hopefully you will find a way so that when the vsync interval ‘fires’, that the JavaScript animation callback is called back ASAP. I can see in the green line that that is happening some of the time. It would be great to see that happen all of the time.

Thank you for you time and effort! I sure look forward to seeing Silk rolled out…

It depends on the OS and what the OS reports. On Windows, we use DwmComposition so I’m not really sure what the OS will report in those cases. Mac should generate a DisplayLink, which will probably correctly align to those refresh rates. There’s nothing specific about 60 FPS hard coded, but I don’t have a non-60 hz monitor to test.

So… I’ve been reading bug reports and seem to have discovered good news. Mr. Chang and others from the FX team have just finished Project Silk vsync work needed for Windows 7+ and Mac OS X. However, there are about 4 related bugs that need to be fixed before it can be enabled. I’d link bug numbers, but the devs don’t like random people commenting on bugs.

Optimistically, Project Silk on desktop will be enabled in current Nightly (38) or the very next one (39). Then we can follow it through Aurora -> Beta -> Release.

So, under Silk, the requestAnimationFrame (rAF) JavaScript will fire VSYNC aligned, but that rendered frame is NOT composited until the NEXT VSYNC interval, and under Windows with Aero enabled (always on under Win8, and on most of the time under Win7), will not reach the screen until the NEXT VSYNC interval because Aero is itself a compositing manager (anything sent to the display takes until the next interval to display) — meaning that there is a VSYNC+VSYNC delay (1/60+1/60, or 1/30 second) delay from the time rAF is called, until a rendered frame reaches the screen under Windows?

So Silk compositing kicking off VSYNC aligned is always compositing the prior frame? If not, please clarify the article about when compositing takes place (immediately after rAF returns?)