Add-on Performance Part 1: Measuring Add-on Performance

In previous posts, we’ve written about the ways we’re making IE9 much faster, like the new script engine that uses multiple cores, and the new rendering subsystem that uses the dedicated graphics chip found on modern PCs. Another aspect of browser performance involves the external code that IE runs on behalf of users, or add-ons.

Add-ons introduce unique features that can enhance the browsing experience. However, they also decrease the browser’s performance in crucial activities like navigating to webpages and creating new tabs. In this way, add-ons affect key usage scenarios like startup and navigation.

Add-on performance is integral to an overall fast browsing experience. IE users expect the browser to be fast, with or without add-ons. We work towards several common goals with add-on developers: providing valuable features with the smallest performance and reliability impact possible (more on reliability in another post).

This blog post is the first in a series on how add-on developers can improve add-on performance. In this post, we’ll share data on the performance impact of add-ons today and how IE enables users to identify the performance impact of their add-ons and stay in control of their PCs. We’ll describe the user scenarios that are important for measuring performance and will walk through how to measure them.

We want add-on developers to have all the information they need to deliver fast, reliable add-ons that respect user choices. We want to make it clear how to test add-on performance. We ask add-on developers to start measuring add-on performance today and making their add-ons faster.

Another common type of extension is plug-ins, specifically ActiveX controls, like Adobe Flash, Apple QuickTime, and Microsoft Silverlight. Unlike add-ons that run in the browser across all web-pages, plug-ins run inside webpages and their performance impact is localized to the webpages that use them. The specifics of this post are about add-ons. Plug-in developers have similar opportunities to make the browsing experience faster and more reliable.

Accelerators, Webslices and Search Providers are a third class of extension. These are written in pure XML format, and were designed to not impact page or browser performance, reliability, or security.

Toolbar Buttons are another type of extension but they only impact IE’s performance when users press them and they’re mapped to an action that launches an add-on.

Understanding Add-on Performance Impact

Severalstudies regarding website response time report that users notice any delay over 0.2 seconds. Actions that are faster than 0.2 seconds appear instantaneous. Scenarios with response times slower than that threshold can feel “slow” to users.

Tab creation is one such scenario. IE initializes add-ons every time the user creates a new tab to make sure that the add-ons can interact with a webpage properly (like the Skype IE add-on, which identifies phone numbers and makes them “clickable”). When the user starts a new instance of IE, one of the first things IE does is create a new tab that, in turn, initializes all the user’s add-ons. The time it takes each add-on to initialize is its load time, and usually depends on the amount of code the add-on executes during its initialization routine.

According to our telemetry data, over 95% of IE8 users today have at least one add-on installed, and users have an average of 9 add-ons installed. If a user’s add-ons each take more than 20 milliseconds to load, his new tabs will take a noticeably longer time to create than usual.

The following chart shows the median load times for the newest versions of the 50 most used add-ons in IE today. Users who have many of these add-ons installed will definitely notice a performance impact when starting IE or creating a new tab:

These are just the 50 most popular add-ons. Many add-ons that aren’t listed above may also have a performance impact on tab creation. We recommend users to install the newest versions of add-ons as they generally have the best performance.

End-users and Add-on Load Times

What’s important to an individual user is his or her own experience, not the broad ecosystem data. In IE8, we introduced a feature to enable users to see for themselves how their add-ons affect their browser performance.

IE measures the load time for each enabled add-on every time IE initializes it. In the Manage Add-ons dialog, IE displays the average load time for each add-on based on the last 10 initializations. You can access the Manage Add-ons dialog via the Tools Menu.

With this information handy, users can choose the add-ons they want to keep enabled, and maintain control over their browsing performance:

Add-on Developers and Add-on Load Times

We collected telemetry data on the load times across the add-ons that IE users have installed to evaluate their performance characteristics. The following chart shows the distribution of all add-on load times experienced by all IE8 users:

Even though most of the time add-ons load in less than 100 milliseconds, add-ons take longer than 200 milliseconds to load over 25% of the time. Users will perceive a noticeable impact on performance in these situations, more so if several of their add-ons have long load times. Measuring your add-on’s performance allows you to identify how often these long load times occur.

Let’s look at some specific examples of slower add-ons from our measurements, and some of the reasons these add-ons have such a large effect on IE’s performance during tab creation. If you are an add-on developer and would like to receive the following load time data for your add-on, you can email us at ieaddonp@microsoft.com.

Sample Add-on #1

Users with this add-on enabled typically see noticeable delays during tab creation. The majority of load times for this add-on are greater than 200 milliseconds. The amplitude of the curve is higher than that of the previous chart. Further investigation revealed that this add-on is generally running more code than is minimally required during initialization.

Sample Add-on #2

This add-on also incurs an impact on the user’s tab creation performance as evidenced by the majority of load times that take over 200 milliseconds. We found that the add-on is performing operations that take a consistent amount of time to complete across all machines (600-1500 milliseconds). We recommend moving these operations off of the initialization routine to expedite tab creation.

Sample Add-on #3

Even though the distribution of load times for this add-on is different from the others, it also incurs an impact on tab creation performance. We investigated the cause behind the uniform distribution of load times beyond 100 milliseconds and found that the add-on is making network calls during initialization. Since network calls take a variable time to complete depending on the connection speed and locale, add-ons must refrain from making network calls during initialization. We recommend spinning off background threads to make these calls at a later time instead.

It’s clear that some IE8 users experience significant performance problems with add-ons during tab creation. Add-on developers who improve add-on load time take an important step towards improving IE performance for end-users. At the same time, we know that add-on performance extends beyond tab creation. Let’s look at the other scenarios in more detail.

Recommendations to Add-on Developers around Performance

In this post, we focus on understanding and measuring scenarios in terms of the elapsed time with the add-on enabled. There are additional metrics that are also important, such as CPU time (amount of time the CPU spent to complete the scenario) and working set (amount of memory required to run the scenario). We’ll cover these two metrics in a future post.

Tab creation and webpage navigation are two of the most relevant scenarios for IE performance. Add-ons have a large impact on the performance of these scenarios because they can block IE from completing the user task. Depending on how the developer wrote the add-on, IE may have to wait for the add-ons to finish running code before IE can finish creating the tab or navigating to the web page.

We recommend add-on developers start by measuring add-on performance in these two scenarios:

1. Tab Creation, Switching, and Closing Some add-ons call across the network during tab create (which causes users to wait), or choose to save their state (with registry reads and writes) as the user is closing a tab, or to repaint their UI as the user is switching between tabs.

Because add-ons can interrupt IE as it handles the user’s request to create a tab, switch between tabs, or close a tab, add-ons can make these operations far from instantaneous.

2. Webpage NavigationAs the user navigates to a webpage, Internet Explorer fires a set of browser events. Add-ons can choose to run code in response to these events. Add-ons that handle one or more of these events can delay users from finishing the web page navigation. The impact is profound on webpages that users expect to open immediately, such as search results or stock market tickers. Some examples:

BeforeNavigate: A security add-on inspects the destination URL before allowing the navigation to proceed.

DocumentComplete: Once a webpage finishes loading, an add-on searches through the entirety of every webpage to add a button next to all the email addresses in order to launch a custom mail application

NavigateError: If an error occurs during webpage navigation, theadd-on redirects the user to a custom error page.

Given how important performance is for users, transparency and control of add-ons will also be important to them.

We also recommend that add-on developers measure and optimize their add-on performance using the publicly available Windows Performance Tools.

We’ve recently blogged about how to measure browser performance with the Windows Performance Tools. Add-on developers can use these tools to measure various performance characteristics of the browser with their add-on(s) enabled. Here are the steps you can take to measure the tab creation and tab close scenarios for an add-on with the Windows Performance Tools installed.

Start an elevated command prompt and execute the following command to start a performance trace and log the results to a file (such as AddonTrace.etl): xperf -start browselog -on Microsoft-IEFRAME -f AddonTrace.etl

Make sure that your add-on is enabled.

Launch Internet Explorer to your home page. Wait several seconds after the navigation completes, and then open several new tabs.

Close each tab in order until the browser window shuts down.

Stop the trace by executing: xperf -stop browselog

Convert the contents of the .etl file into a text file that you can parse and measure for: xperf -i AddonTrace.etl -o AddonTrace.txt

Next, inspect the text file to find the following pairs of ETW events that correspond to the scenarios. To find the elapsed time for the event, subtract the timestamp of the end event from that of the start event:

Scenario

Start Event

End event

Tab Creation

(Load time)

CoCreateInstance():

Microsoft-IEFRAME/ExtensionCreate/Start

SetSite():

Microsoft-IEFRAME/ExtensionSetSite/Start

CoCreateInstance():

Microsoft-IEFRAME/ExtensionCreate/Stop

SetSite():

Microsoft-IEFRAME/ExtensionSetSite/Stop

Tab Close

SetSite(NULL):

Microsoft-IEFRAME/ExtensionSetSiteNull/Start

SetSite(NULL):

Microsoft-IEFRAME/ExtensionSetSiteNull/Stop

In addition, you can use the Windows Performance Tools to measure add-on performance in the other scenarios we described earlier. Here are the ETW events that correspond to the other scenarios:

Scenario

Start Event

End event

Tab Switch

Microsoft-IEFRAME/ LeftButtonAction/win:Info

The following sequence of events:

Microsoft-IEFRAME/Browseui_Tabs_SwitchTabs/win:Stop;

Microsoft-IE/MSHTML_CDOC_ONPAINT/win:Stop;

Microsoft-IEFrame/Browseui_Tabs_WaitMessage/win:Start

Webpage Navigation

Microsoft-IEFRAME/FRAME_URLENTERED/win:Info

The following sequence of events:

Microsoft-IE/MSHTML_CMARKUP_ONLOADSTATUSDONE/win:Info

Microsoft-IE/MSHTML_CDOC_ONPAINT/win:Stop

Microsoft-IEFrame/Browseui_Tabs_WaitMessage/win:Start

We recommend running these tests on a fixed environment against a set of the most popular websites to realistically measure performance. Caching these sites on a proxy server helps minimize variance in the website content. We also recommend running these tests across multiple machine configurations.

Add-on Performance: We’re all in, together

Performance is important to end-users: they’re paying attention and will take action to get better performance.

We encourage users to look at the performance impact of the various add-ons installed on their machines using the IE8 Manage Add-ons dialog. This information helps users stay in control of their browsing performance while using the add-ons they want.

We also encourage add-on developers to make those decisions easier for end-users by writing faster add-ons. We recommend developers start by measuring add-on performance using the Windows Performance Tools and follow the guidance we’ve included with this post. Set up a performance lab infrastructure with a server of cached websites and multiple machine configurations. While the IE8 add-on load time measurements serve as a gauge for end users, we recommend measuring performance across all the tabbed browsing and webpage navigation scenarios as well.

Using these measurements and our guidance above, you can tune your add-on for the best performance. As you find optimizations or best practices, we encourage you to share them with the developer community blog post comments here or on Connect.

Improving add-on performance is critical to building a fast and highly compelling browsing experience and we’re all in it together. We want IE users to enjoy great add-on functionality and great performance.