Firefox, the Browser that has your Back[up]

One of the most important features of Firefox, in my opinion, is Session Restore. This component is responsible for ensuring that, even in case of crash, or if you upgrade your browser or an add-on that requires restart, your browser can reopen immediately and in the state in which you left it. As far as I am concerned, this feature is a life-safer.

Unfortunately, there are a few situations in which the Session Restore file may be corrupted – typically, if the computer is rebooted before the write is complete, or if it loses power, or if the operating system crashes or the disk is disconnected, we may end up losing our precious Session Restore. While any of these circumstances happens quite seldom, it needs to be applied as part of the following formula:

How it works

restarting Firefox without data loss after a crash of either Firefox, the Operating System, a driver or the hardware, or after Firefox has been killed by the Operating System during shutdown;

restarting Firefox without data loss after Firefox has been restarted due to an add-on or an upgrade;

quitting Firefox and, later, restarting without data loss.

In order to handle all of this, Firefox needs to take a snapshot of the state of the browser whenever anything happens, whether the user browses, fills a form, scrolls, or an application sets a Session Cookie, Session Storage, etc. (this is actually capped to one save every 15 seconds, to avoid overloading the computer). In addition, Firefox performs a clean save during shutdown.

While at the level of the application, the write mechanism itself is simple and robust, a number of things beyond the control of the developer can prevent either the Operating System or the hard drive itself from completing this write consistently – a typical example being tripping on the power plug of a desktop computer during the write.

The new mechanism involves two parts:

keeping smart backups to maximize the chances that at least one copy will be readable;

making use of the available backups to transparently avoid or minimize data loss.

The implementation actually takes very few lines of code, the key being to know the risks against which we defend.

Keeping backups

During runtime, Firefox remembers which files are known to be valid backups and which files should be discarded. Whenever a user interaction or a script requires it, Firefox writes the contents of Session Restore to a file called sessionstore-backups/recovery.js. If it is known to be good, the previous version of sessionstore-backups/recovery.js is first moved to sessionstore-backups/recovery.bak. In most cases, both files are valid and recovery.js contains a state less than 15 seconds old, while recovery.bak contains a state less than 30 seconds old. Additionally, the writes on both files are separated by at least 15 seconds. In most circumstances, this is sufficient to ensure that, even of hard drive crash during a write to recover.js, at least recovery.bak has been entirely written to disk.

During shutdown, Firefox writes a clean startup file to sessionstore.js. In most cases, this file is valid and contains the exact state of Firefox at the time of shutdown (minus some privacy filters). During startup, if sessionstore.js is valid, Firefox moves it to sessiontore-backup/previous.js. Whenever this file exists, it is valid and contains the exact state of Firefox at the time of the latest clean shutdown/startup. Note that, in case of crash, the latest clean shutdown/startup might be older than the latest actual startup, but this backup is useful nevertheless.

Finally, on the first startup after an update, Firefox copies sessionstore.js, if it is available and valid, to sessionstore-backups/upgrade.js-[build id]. This mechanism is designed primarily for testers of Firefox Nightly, who keep on the very edge, upgrading Firefox every day to check for bugs. Testers, if we introduce a bug that affects Session Restore, this can save your life.

As a side-note, we never use the operating system’s flush call, as 1/ it does not provide the guarantees that most developers expect; 2/ on most operating systems, it causes catastrophic slowdowns.

Recovering

All in all, Session Restore may contain the following files:

sessionstore.js (contains the state of Firefox during the latest shutdown – this file is absent in case of crash);

sessionstore-backups/recovery.js (contains the state of Firefox ≤ 15 seconds before the latest shutdown or crash – the file is absent in case of clean shutdown, if privacy settings instruct us to wipe it during shutdown, and after the write to sessionstore.js has returned);

sessionstore-backups/recovery.bak (contains the state of Firefox ≤ 30 seconds before the latest shutdown or crash – the file is absent in case of clean shutdown, if privacy settings instruct us to wipe it during shutdown, and after the removal of sessionstore-backups/recovery.js has returned);

sessionstore-backups/previous.js (contains the state of Firefox during the previous successful shutdown);

sessionstore-backups/upgrade.js-[build id] (contains the state of Firefox after your latest upgrade).

All these files use the JSON format. While this format has drawbacks, it has two huge advantages in this setting:

it is quite human-readable, which makes it easy to recover manually in case of an extreme crash;

its syntax is quite rigid, which makes it easy to find out whether it was written incompletely.

As our main threat is a crash that prevents us from writing the file entirely, we take advantage of the latter quality to determine whether a file is valid. Based on this, we test each file in the order indicated above, until we find one that is valid. We then proceed to restore it.

If Firefox was shutdown cleanly:

In most cases, sessionstore.js is valid;

In most cases in which sessionstore.js is invalid, sessionstore-backups/recovery.js is still present and valid (the likelihood of it being present is obviously higher if privacy settings do not instruct Firefox to remove it during shutdown);

In most cases in which sessionstore-backups/recovery.js is invalid, sessionstore-backups/recovery.bak is still present, with an even higher likelihood of being valid (the likelihood of it being present is obviously higher if privacy settings do not instruct Firefox to remove it during shutdown);

In most cases in which the previous files are absent or invalid, sessionstore-backups/previous.js is still present, in which case it is always valid;

In most cases in which the previous files are absent or invalid, sessionstore-backups/upgrade.js-[…] is still present, in which case it is always valid.

Similarly, if Firefox crashed or was killed:

In most cases, sessionstore-backups/recovery.js is present and valid;

In most cases in which sessionstore-backups/recovery.js is invalid, sessionstore-backups/recovery.bak is pressent, with an even higher likelihood of being valid;

In most cases in which the previous files are absent or invalid, sessionstore-backups/previous.js is still present, in which case it is always valid;

In most cases in which the previous files are absent or invalid, sessionstore-backups/upgrade.js-[…] is still present, in which case it is always valid.

Numbers crunching

Statistics collected on Firefox Nightly 32 suggest that, out of 11.95 millions of startups, 75,310 involved a corrupted sessionstore.js. That’s roughly a corrupted sessionstore.js every 158 startups, which is quite a lot. This may be influenced by the fact that users of Firefox Nightly live on pre-alpha, so are more likely to encounter crashes or Firefox bugs than regular users, and that some of them use add-ons that may modify sessionstore.js themselves.

With the new algorithm, assuming that the probability for each file to be corrupted is independent and is p = 1/158, the probability of losing more than 30 seconds of data after a crash goes down to p^3 ≅ 1 / 4,000,000. If we haven’t removed the recovery files, the probability of losing more than 30 seconds of data after a clean shutdown and restart goes down to p^4 ≅ 1 / 630,000,000. This still means that , statistically speaking, at every startup, there is one user of Firefox somewhere around the world who will lose more than 30 seconds of data, but this is much, better than the previous situation by several orders of magnitude.

It is my hope that this new mechanism will transparently make your life better. Have fun with Firefox!

I’m not sure what you are asking about security. If a malicious/crooked piece of software has access to your hard drive, it has access to your session data. However, this is nothing new, as it also has access to all the data files of Firefox (or any other browser you may be using), including your cache, your history, your cookies, etc.

Initially, for historical reasons. Session Restore has been implemented with JSON since the first version. Changing now would 1/ require a very consequent refactoring; 2/ break a number of add-ons. Also, it is not clear to me how safe the data would be if we disallow flush() calls. Finally, backups are useful not only against data loss, but also against user errors, as a last defense.

Slight off topic: Yoric, do you know when you guys are going to implement “lazy unfolding” of sessionstore into the memory?

Because as it is now if I have many tabs — even when “click-to-load” option is on — FireFox still allocated empty memory pages for each and every tab, which slows down start, as well as occupies HALF GIGABYTE or more memory even if you only see one tab and the rest are not loaded.

This situation is obviously ridiculous and has to change. Can you help pro users, please?

Yes, we are aware of the issue.
The problem is that it requires a very deep refactoring of the implementation of `tabbrowser`, something which is both complicated, long, and hard to test. It would be great if we could have someone work on this fulltime for a few months, but I’m not sure when this is going to happen.

The following bug was opened by a contributor who has ideas on how to get this done.

rather than entirely writing the file, we simply rename recovery.js, and renaming has basically no chance of corrupting data, unless your file system is so damaged that you have much bigger concerns;

the probability of recovery.js being corrupted is a decreasing function of the time since the write order was sent to the operating system, with “almost 0″ generally being reached somewhere between 10 seconds and 20 seconds unless the harddrive is badly broken, in which case you also have much bigger concerns – by the time we perform the rename, we know that at least 15 seconds have elapsed;

as a side-note, the “almost 0″ probability is reached faster on a desktop, slower on a laptop running on low battery, and we plan to land shortly a patch that delays by 30 seconds instead of 15 the write if the laptop is running on battery, both to save battery life and to increase the chance of being in the “almost 0″ region;

What does this mean ? I understand this sentence as : “The probability of losing more than 30 seconds of data at startup is 1.” Which is wrong, hopefully !

I wanted to give the image that if every single user of Firefox on Earth decides to start Firefox at the same time, according to the probabilities above, only one of them might lose more than 30 seconds of data.

You must mean the probability *if the system is still alive*. I don’t think that a file being corrupted at t + 1 s would uncorrupt itself with time, for instance at t + 10 s.

Not exactly, so let me rephrase: the probability of recovery.js ending up corrupted is a decreasing function of the duration between the instant the write order was sent to the operating system and the instant the computer freezes. If the computer doesn’t freeze at all within 10-20seconds after the write, there is basically no chance that the file may be corrupted (again, unless you have bigger problems).

An aside : the probabilities of not losing > 30 s of data discussed here are presuming that the mechanism of session store itself does not lose data. ;-)

You mean that now, and even before Fx dump all the tabs, their state (I suppose including the position in the page, and text entered in the textarea) every 15 seconds ?
That explain at least why unloaded tabs are still slowing your browser.
But still session store is just a few Mo (for the thousands of tabs I have opened), so I don’t really understand the slow down (especially for opening an closing) do you have any hint on that ?

I am not a user of Session Manager, but I seem to remember that Session Manager is a bit more flexible. However, with this refactoring, adding the flexibility of Session Manager to Session Restore would just be a few lines of code. If you know someone who is interested (perhaps the authors of Session Manager?), I’d be glad to help them putting Session Manager in Firefox.

Also, I seem to remember that Session Manager needs to rely on some slow mechanisms, which are pretty much the only way to communicate with Session Restore at the moment, so that would also help the performance for users of Session Manager.

Anyway – my comment is, I cannot STAND session restore. Some fire fox is better than IE and Opera and I will not use Chrome due to it being Google’s.

However sometimes FF starts to eat up more and more CPU to the point I need to shut it down and sometimes it hangs when I’m shutting down all my FF’s and I have to “end process” out of them – a LAST resort because I KNOW that when I finally shut them all down – when I restart FF – it will do the session restore and I’m back to square 1 with it reloading all the heavy websites that brought my system to a crawl in the FIRST place.

“You have to wonder at the complexity of what looks like a simple task. You’d think Ff would simply keep a trail of opened tabs (History?!). Instead they seem to fire a steady stream of access to places.sqlite.
Recently i realised that the cpu was constantly working away while idle at 2-5% and traced it to a horrendous amount of disk activity. Changing the Browser.sessionstore.interval to 120000 (2min) seemed to calm things significantly. Everything has a price.”

Well, places.sqlite is what contains the history, so it makes sense that writing sessionstore.js & co. requires looking at it. However, indeed, hitting the disk every ~15 seconds because of that doesn’t sound like a good idea. I have just filed bug 1096072 on the topic.

These days, I am looking at the implementation of history, for unrelated performance reasons. If I see an obvious and bad reason for which we end up hitting the disk, I will try and take the opportunity to fix it.

My previous comment is still awaiting moderation after I posted it on 18th day, so here it is again without a website link which might trigger the pre-moderation.

What’s the performance cost of this feature (15 seconds)?

Somebody commented on ghacks website that:

“You have to wonder at the complexity of what looks like a simple task. You’d think Ff would simply keep a trail of opened tabs (History?!). Instead they seem to fire a steady stream of access to places.sqlite.
Recently i realised that the cpu was constantly working away while idle at 2-5% and traced it to a horrendous amount of disk activity. Changing the Browser.sessionstore.interval to 120000 (2min) seemed to calm things significantly. Everything has a price.”

In the past, my Firefox crashed from time to time and then again sometimes the session was lost. It did not happen often, but when it did was annoying. Now with the new restore feature I could just recover fro such a crash and restore my session out of one of the backups, which was a big help!

Starting with firefox 33, with a new firefox profile, the faviconizetab extension doesn’t restart tabs in their iconized state anymore. Things still work as expected if using an existing profile (created with older firefox versions). Is this related to the session restore changes and is there anything I could try for a workaround ?

Heya, thought you might like to know this may be the best break down of how all the parts of Session Restore operate available online. :)

I recently had a situation where I had to mess with these files, trying to find the most recent one to best recover from a tricky crash.

At first I thought sessionstore-backups/upgrade.js-[…] was going to be my best bet. Only 3 weeks old; not too catastrophic.
But then I read your line here:
“sessionstore.js (contains the state of Firefox during the latest shutdown – this file is absent in case of crash);”
And I realized a hack so straightforward that I (and I think many others) forgot about it. I navigated to the Firefox Profiles folder and pulled up the Windows Previous Versions tool (right click>properties>previous versions). I then selected Yesterday’s backup, copied it to a new folder, found sessionstore.js in there, and then copied that over to the present Firefox Profiles folder.

It doesn’t work. Closed the browser as normal, restarted after removing AVG tuneup and my session is gone. Go to the profile folder and last session is gone. One backup is from weeks ago, one from 8 days ago and one from after I’ve restarted the browser. So like much of what Firefox has done recently, absolutely useless and if anything a step backwards.

Back to using Session Manager add on, Google Chrome and Firefox becoming, more and more, my 2nd or 3rd choice browser. Such a shame as I have moved so many people over to Firefox but it is falling far behind the competition, it really is awful. I never recommend it to anyone any more as all I get now is complaints.

I never used to get frustrated with a web browser (after leaving IE) but Firefox really is verging on useless these days. The only reason I’m still using it is its open source nature.

Hi, sure, I experienced a crash, and all my tabs in the session is no longer available in Firefox anywhere… but I do have all the .js files! However, how in the world do I restore my last good session using these files? Please help?