Time to Deploy improvement of 25 percent

Since all software has bugs, it’s more important to consider how long it takes to get a fix out when a security issue is discovered than it is to count bugs. Number of vulnerabilities identified is a function of how many bugs are present, but is probably more influenced by things like who is looking, and how good they are at finding security issues. That makes it a misleading metric.

We spend a lot of time thinking about how we can get fixes out faster to users. But the window of risk is actually determined by two factors. The first is the time it takes us to create a patch, let call this Time to Fix. This includes the time to investigate a security issue, develop and test a fix, and finally ship the update. This is a better measure for understanding how safe a user is going to be than simply counting bugs.

But there’s another aspect to getting the fix to the user that often goes overlooked. That is the Time to Deploy. Time to Deploy is how long it takes for users to get a patch installed once the fix is available from the vendor. Auto-update has gone a long way toward minimizing Time to Deploy for Firefox, but there are still areas on which we can improve.

This chart shows how long it took for users to move from 1.5.0.5 to 1.5.0.6 last year:

This shows that it took eight days for about 90 percent of Firefox users to get updated. When I saw this last year I thought it was pretty fantastic. Firefox has millions and millions of users. Getting almost everyone updated in just eight days seemed pretty incredible to me.

I ran the numbers again this year after we shipped 2.0.0.4.

This chart shows how long it took for users to move from 2.0.0.3 to 2.0.04 last month:

This time it only took six days to update 90 percent of users. That’s a 25 percent decrease in Time to Deploy and a significant improvement in reducing the window of opportunity for attackers to take advantage of security vulnerabilities. It appears that some of the improvements in infrastructure have contributed to these numbers so a big thank you to everyone working in IT and to our partners that host mirrors. You’re helping to keep Firefox users safe.

I’m not sure if this actually came with FF 1.5, but I love the way in which the background downloading works in FF2: It downloads the update in the background and then asks you if it should update.

You can safely click “Later” and forget about it and only after the next restart, Firefox will be updated.

This is SO much better than a popup telling you that a new version is out *before* starting the download and then forcing the user to wait through the update, in the mean time forgetting about all the tasks one wanted to do.

Whoever came up with this: This is – IMHO – the greatest invention since sliced bread and I’m sure it helps a lot at keeping the transition day small.

[…] And as a result, we’ve got a lot of newer initiatives all over the project to make firedrills more smoother and faster, from automated regression testing in Tinderboxen to automated smoketests and BFTs to automated release deliverable verification. These are all going to improve on Firefox 2’s already-pretty-awesome patch-delivery time. […]