Reading out the end time in browser speed tests

A few weeks back I did some DOM speed tests on mobile browsers (results forthcoming). The most important result of these tests is not the actual values (although they’re interesting), but the fact that I could finally prove a theory that I’ve had in the back of my mind for at least two years now.

Basically, when setting up a speed test you should be very careful to allow the browser to render the result on screen before you close the test by reading out the second timestamp.

An example will clarify this point. A DOM speed test may look roughly as follows:

Although this script might seem fine at first sight, it’s not. The problem here is that the entire test, including the time measurements, are wrapped in one function, and that some browsers only applies the result of the test (i.e. the changes in the DOM you want to test) to the screen after the function has ended entirely. Thus the end timestamp is read before the browser has applied the result to the screen.

So what really happens here is the following:

Get start time

Perform DOM manipulation in browser memory

Get end time and calculate result

The function ends, and only now does the browser start to apply the results of the DOM manipulation to the actual DOM.

Not all browsers wait until the end of the function to apply the results, but some do, among which Safari for the iPhone 2.2. These browsers only report the time it takes them to build the new DOM tree in the browser memory, and disregard the time it takes to show it on the screen.

Any testing methodology must work in all browsers, so our test function must work around this problem.

The correct way of conducting this test is setting a timeout for reading out the end time. The function ends when the in-memory DOM manipulation has been done, which allows the browser to apply the changes.

Because the browser uses its complete capacity for applying the changes to the screen, executing the timeout is deferred until it is done, even if that takes far longer than the formal timeout time.

Define a function to get end time and calculate result and set a timeout

The function ends, and only now does the browser start to apply the results of the DOM manipulation to the actual DOM.

Once the DOM manipulation is finished, the browser is freed up to treat the timeout we set, and now the function that gets the end time and shows the result is executed.

In other words, this allows us to also measure the time the browser needs to show the results on screen.

This is what we really want to know. If the browser is lightning-fast in performing DOM manipulations in its memory, but sluggish in applying them to the screen, the net result for the end user is a relatively sluggish user experience.

This is not a random theoretical musing. I tried both methodologies on Safari iPhone 2.2 in a test that generated 5,000 list items. The first, incorrect, method yields a total time of about 3 seconds, while the second, correct, method yields a total time of about 14 seconds. That’s an 11 second difference.

(I’d be interested in the iPhone 3 results. I deliberately haven’t updated my iPhone yet, because there still are some tests I want to conduct on the 2.2 OS.)

In other words, the iPhone 2.2 takes 3 seconds to perform the necessary calculations in its memory, but subsequently takes 11 seconds to show the changes on screen. It’s slow, in other words. The first method does not reveal that; we need the second one for an accurate reading.

Therefore it’s important that whenever you do speed tests, the end time should be calculated only after the browser has finished applying the results to the screen.

@Andrea
> What if some browser renders directly before the timeout then?

The test result will be off by 15ms.

Considering with this test all mobiles are going to be far above 1s, that gives an error of at worst 1.5% (and usually far lower, on ppk's page the fastest listed device is at 5.4s, which gives us a potential error of 0.28%). Not exactly something worth worrying about, I don't think PPK considers these tests scientific, they're here to give people an idea of what to expect rather than hard numbers (I might be proven wrong if ppk whips out bar charts with error intervals and full-blown stat analysis of the results, but I doubt he's going to bother)

@Ionut maybe 10 ms for DOM tests are not a problem while I was thinking about a general purpose function but being DOM something "a part" I agree we should find a way to perform the layout linearly in the single callback. For ppk tests and purpose I guess this technique is almost perfect though

@ppk I guess there's a reason why you couldn't use a zero-timeout: setTimeout(fn, 0); any chance you can explain why?

Several people seem confused by the arbitrary 10ms delay - and you can't call it "the correct way" without explaining that the computation stage takes hundres of times longer, and that "setTimeout(fn, 0)" is flawed in some way (is it?)

I've been using a similar method to test rendering times, although I'm not comparing the results between browsers, just comparing different techniques on an individual browser.

Out of curiosity, why 10ms? I've used 0ms in the past to simply place the callback in the queue (after rendering). Have you encountered a browser that would call func setTimeout(func, 0) before redraw has completed?

@BARTgG: chrome's js engine might be multithreaded, but in a single page/tab, I'm willing to bet you won't actually get multiple threads, since there is no way to do that safely in clientside JS. the whole DOM and most of the BOM relies on a single threaded event model.

On the other hand, I don't think there's any hard /guarantee/ anywhere that the rendering engine will update the whole page just because you're returning from an event handler.

@PPK: doesn't reading an element's offsetWith and/or similar properties force an immediate rendering update? If that's reliable that seems to me to be a better way to doing this.

@Joost/BARTdG I seem to remember reading in the JavaScript spec that the JavaScript engine should be single threaded, or at least appear to be to the actual JS code, in order to make certain promises (I'm not sure which promises...and don't have time to look it up atm)

Whether a 10 msec wait time will introduce a substantial error in the measurement depends on 2 factors:
1) The granularity of the OS scheduler. On Windows PCs that's about 16 msec. On some mobiles it might be 1 msec. Other (non-phone) Linux ARM embedded devices I work on have 1 msec OS scheduler frequency.
2) The amount of time for the page rendering. If the page rendering time is very small then the 10 msec timeout will be too large a fraction of the total time. For a 1 second page rendering time the 10 msec will have an insignificant impact.

If you are causing a reflow then the the reflow calculations are suspended until you exit the current execution context.

If you can force an immediate reflow then you can still check the time immediately after. What will cause an immediate reflow varies from platform to platform. Checking the offsetWidth of an element is usually sufficient, calling getComputedStyle() sometimes works too. It's an art more than it is a science. :)