Using Chrome Traces to Automate Rendering Performance - aka how browser-perf works

Ten years ago, increasing the performance of a website usually meant
tweaking the server side code to spit out responses faster. Web
Performance engineering has come a long way since then. We have
discovered patterns and practices that make the (perceived) performance
of websites faster for users just by changing the way the front end code
is structured, or tweaking the order of elements on a HTML page.
Majority of the experiments and knowledge has been around delivering
content to the user as fast as possible.
Today, web sites have grown to become complex applications that offer
the same fidelity as applications installed on computers. Thus,
consumers have also started to compare the user experience of native
apps to the web applications. Providing a rich and fluid experience as
users navigate web applications has started to play a major role in the
success of the web.
Most modern browsers have excellent tools that help measure the
runtime performance of websites. The Chrome Devtools features a powerful
Timeline panel
that gives us the tracing information needed to diagnose performance
problems while interacting with a website. Metrics like frame rates,
paint and layout times present an aggregated state of the website’s
runtime performance from these logs.
In this article, we will explore ways to collect and analyze some of
those metrics using scripts. Automating this process can also help us
integrate these metrics into the dashboards that we use to monitor the
general health of our web properties. The process usually consists of
two phases – getting the tracing information, and analyzing it to get
the metrics we are interested in.

Collecting Tracing information

The first step to get the metrics would be to collect the performance trace from chrome while interacting with the website.

Manually recording a trace

The simplest way to record a trace would be to hit the “start
recording” button in the timeline panel, performing interactions like
scrolling the page or clicking buttons, and then finally hitting “stop
recording”. Chrome process all this information and shows graphs with
framerates. All this information can also be saved as a JSON file by
right-clicking (or Alt-clicking) on the graph and selecting the option
to save the timeline.

Using Selenium

While the above method is the most common way to collect tracing from
Chrome, doing these repeatedly over multiple deployments or for
different scenarios can be cumbersome. Selenium
is a popular tool that allows us to write scripts that perform
interactions like navigating webpages or clicking buttons and we could
leverage such scripts to capture trace information.
To start performance logging, we just need to ensure that certain parameters are added to the existing set of capabilities.
Selenium scripts have binding for multiple programming languages
and if you already have selenium tests, you could add the capabilities
in the following examples to also get performance logs. Since we are
only testing Chrome, running just Chromedriver instead of setting up Selenium and then executing the following node script gets the trace logs.

The script above tells selenium to start chrome with performance logs,
and enables specific trace event categories. The “devtools.timeline”
category is the one used by Chrome devtools for displaying its graphs
about Paints, Layouts, etc. The flags “enable-gpu-benchmarking” expose a
window.chrome.gpuBenchmarking object that has a method to smoothly scroll the page. Note that the scroll can be replaced by other selenium commands like clicking buttons, typing text, etc.

Using WebPageTest

WebPageTest also roughly relies on the performance log to display metrics like SpeedIndex. As shown in the image, the option to capture the timeline and trace json files has to be enabled. Custom actions
can also be added to perform actions in a WebPageTest run scenario.
Once WebPageTest run finishes, you can click download the trace files by
clicking the appropriate links. This method could not only be used in
WebPageTest, but also with tools like SpeedCurve that support custom metrics.

Analyzing the Trace Information

The trace file that we get using any of the above methods can be loaded in Chrome to view the page’s runtime performance. The trace event format defines that each record should contain a cat (category), a pid (processId), a name
and other parameters that are specific to an event type. These records
can be analyzed to arrive at individual metrics. Note that the data
obtained using the Selenium Script above has some additional metadata
for every record that may need to be scrubbed before it can be loaded in
Chrome.

Framerates

To calculate the FrameRates, we could look at events by the name DrawFrame.
This indicates that a frame was drawn on the screen and by calcuating
the number of frames drawn, divided by the time for the test, we could
arrive at the average time per frame. Since we also have benchmarking category enabled, we could look “BenchmarkInstrumentation:*” events that have timestamps associated with them. Chrome’s telemetry benchmarking system uses this data to calculate the average frame rates.

Paints, Styles, Layouts and other events

The events corresponding to ParseAuthorStyleSheet, UpdateLayoutTree and RecalculateStyles are usually attributed to the time spent in Styles. Similarly, the log also contains Paint and Layout events that could be useful. Javascript time can also be calculated using events like FunctionCall or EvaluateScript. We can also add the GC events to this list.

Network Metrics

The tracelogs also have information related to firstPaint etc. However, the ResourceTiming API and the NavigationTiming
APIs make this information available on the web page anyway. It may be
more accurate to collect these metrics directly from real user
deployments using tools like Boomerang, or from WebPageTest.

Considerations

There are a few things to consider while running the tests using the methods mentioned above.

Getting Consistent metrics

Chrome is very smart at trying to draw the page as fast as possible
and as a result, you could see variations in the trace logs that are
gathered when running tests for a scenario twice. When recording the
traces manually, human interaction may introduce differences in the way a
mouse is moved or a button is clicked. Even when automating, running
separate selenium scripts occurs over the network and executing these
scripts is also recorded in the logs.
To get consistent results, all this noise needs to be reduced. It
would be prudent to not combine scenarios when recording trace runs.
Additionally, running all automation scripts as a single unit would also
reduce any noise. Finally, comparing tests runs across multiple deploys
is even better since both these runs would have the same test script
overhead.

Trace file size

The trace file can take up a lot of space and could also fill up
Chrome’s memory. Requesting for the logs from ChromeDriver could in a
single, buffered request could also result in Node throwing out of
memory exceptions. The recommendation here would be to use libraries
like JSONStream
to parse the records as a stream. Since we are only interested in
aggregating the individual records, streaming can help consume scenarios
that are longer and hence take more memory.

Browser Support

The performance logs are available on the recent versions of the
Chrome browser, both on the desktop and on Android. I had worked on a commit to make similar trace logs available for mobile Safari. Hybrid applications using Apache Cordova use WebViews which are based on browsers. Hence, this method could also be applied to Apache Cordova apps for Android and iOS.
Note that these performance logs started as being Webkit specific, and hence are not readily available for Firefox or IE yet.

Conclusion

All the techniques described above are packaged together in the browser-perf
package. Browser-perf only adds the extra logic to smooth out the edge
cases like the scrolling timeout, or ability to pick up specific
metrics, or processing the trace file as a stream. I invite you to check
out the source code of browser-perf to understand these optimizations.
These metrics can be piped into performance dashboards and watching
out for these metrics should give us an idea of general trends of how
the web site has been working across multiple commits. This could be one
small step to ensure that websites continue to deliver a smooth user
experience.