MemShrink progress, week 73–74

B2G! Fennec! Social API! They’re all happening, and memory consumption is a big deal for all of them. Time for a MemShrink report.

B2G

Lots of B2G work is happening, unsurprisingly. All of these changes have been backported to the Aurora channel.

Kyle Huey landed an important patch that merges system compartments together. This avoids wasted space caused by having 100s of small compartments. As Chris Jones said: “It’s a completely different phone with these patches.” Unfortunately, the preference controlling this behaviour is currently turned off on B2G, because it’s causing some Marionette test failures. Hopefully they’ll be dealt with soon. Note that this preference won’t be turned on in desktop builds, and there’s a medium-term plan to avoid this wasted space in a less hacky fashion.

Memory Reporting

Lots of work happened on the memory reporting front. This was inspired by “heap-unclassified” typically being much higher on B2G than on desktop. Desktop Firefox uses a single process and its memory consumption is usually measured in the 100s of MiBs or more. In contrast, B2G uses one “main” process and then one process per running app, and the smaller ones are typically around 10 MiB. As a result, the sundry small per-process things that don’t matter much for desktop loom larger on B2G. (For the same reason, changes that reduce per-process memory by smallish amounts have outsize value on B2G.) Most of the following changes have also been backported to the Aurora channel.

I fixed the “explicit/atom-tables” memory reporter, which was erroneously always reporting 0 bytes. It now measures anywhere from a few 100 KiBs to several MiBs of memory. This was a frustrating bug to find. DMD exists to check that memory reporters are measuring memory properly, and the reporter was doing its measurements just fine. But DMD cannot check that the measured amounts are added correctly, and that’s what was going wrong here — a line that should have been this:

return n;

instead was this:

return 0;

In other words, the code I had written was doing the difficult part correctly, but botched the trivial part. How annoying.

Social API

Felipe Gomes fixed a document leak that occurred when the social sidebar was hidden, and disappeared only when it was unhidden.

Fennec

Kartikaya Gupta (a.k.a. Kats) turned on tab expiration for Fennec. What this means is that Fennec will now unload the contents of background tabs in certain circumstances — when memory is low, and in some cases when a tab hasn’t been viewed for over an hour — and then reload them when they are brought back into the foreground. Read Kats’ blog post for more details.

(This change was prompted by Project 256meg, which aims to make Fennec usable on phones with only 256 MiB of RAM. (Fennec currently requires 512 MiB, according to the official specs.) Kats has just got to the point where Fennec can actually start on 256 MiB devices. Although Fennec only uses a single process, there is obviously significant overlap between this goal and B2G’s memory reduction goals.)

An obvious follow-up idea is to add tab expiration to desktop Firefox. But the bug tracking that idea has seen more heated discussion. The reason is that tab expiration might cause data loss in some cases. This is less of a problem on Fennec because (as far as I can tell) there is less of an expectation that in-flight data will be saved on mobile devices. For example, having processes killed due to memory constraints on mobile is much more common than on desktop. Still, I expect plenty more arguing before this issue is resolved one way or the other. One option is to allow it on desktop but have it disabled by default.

Add-ons

We hit a notable milestone on the add-on front this fortnight: on November 1st hit bug for tracking known leaks in add-ons — filed 16 months ago — had zero blockers. In other words, we reached the point where no add-ons were known to leak! Unfortunately this didn’t last long, and at the time of writing the tracking bug has three blocking bugs. Still, it’s confirmation that what used to be our biggest memory consumption problem is well under control.

Perhaps there are other cases of unsavable data situations, but the first that comes to mind is plugins (aka, Flash & friends).
One could be in the middle of a game or other data-storing transaction. I don’t know if we have any method to store & restore such states. This would also include Java and even the pause point within a QuickTime movie.
As Nick said, the problem rings true across the board, but there simply is a lesser expectation on mobile of long term data safety.

If you’re suggesting writing it to disk, there’s often information in forms that’s not entered in password fields that I don’t want written to permanent storage. Having to purge my form autocomplete list every second time I need to enter my SSN because the developer used a plaintext input box is bad enough. Having to find an additional place to do so would be even worse.

then what about landing that on desktop for background tabs that don’t have active plugins (with click-to-plugin, this percentage is sure to grow), and use that to pressure plugin makers (adobe, oracle) to come up with a solution to the problem?

I don’t; and after the first time I had to locate and nuke the autocomplete cache because of the issue I added sites with a plaintext SSN input field (or anything else I don’t want cached) to the list of sites I never use FF for. Opera’s form filler has it’s own set of issues; but it only stores data I explicitly authorized it to.

The tab expiration is nice for some cases, but I want to have an option to disable it (as well as the “don’t load contents at startup, wait until they are accessed”). I’m willing to wait a bit when I start firefox (not that I really have much choice :-), but after it’s started, running into a delay whenever I select a tab is annoying. Sometimes I take the several minutes to flip through all my windows and do “reload all tabs”, but it’s a hassle.

I haven’t posted for a while but thought I would update you with one of my very basic memory tests that I do from time to time.

Using 18 b1 I opened all my bookmarks in one go. Roughly 70 tabs. I surfed about a bit and the memory usage got up to about 850,000 kb.

Then I closed all but one tab using the “Close Other Tabs” feature.

Memory dropped to 430,000 kb but never went any lower with only one tab open.

I did the test again and when I closed all but one tab again using the “Close Other Tabs” feature I decided that the one tab I leave open this time was the homepage of this blog.

Again the memory dropped from 850,000 kb, this time stopping at 340,000 kb but never going any lower.

340,000 kb of memory used with JUST this blog open doesn’t seem right.

Is this a known issue? Whilst the memory usage when many tabs are loaded is a lot lower than it was 18 months ago, and no longer climbs exponentially of its own accord, the speed with which memory is released once tabs are closed is still either shockingly or non existent.

My laptop is 2.3 GHz dual core, 4 GB of RAM and Windows 7.

The only add-ons I have installed are British English Dictionary 1.19.1 and Test Pilot 1.2.2.

The problem is fragmentation. That 340,000 kb would contain a lot of empty space. The good news is that if you reopen lots of tabs, the space will be filled in, so it’s not a growing problem. Compacting GC will help some with this for the JS heap, but so long as Firefox is single-process this behaviour will occur for the C/C++ heap. That’s one advantage of a multi-process model — if you close a tab and the process is killed, all the memory is definitely recovered.

I meant to say that if you want I can email you my bookmarks file if that would help at all. As far as I can make out there are no sites in there that would be causing memory leaks once the tab was closed, but I guess I don’t know for sure.

This is one memory test where Chrome still wins because each tabs is a separate process. That said, Chrome’s memory usage with 40 tabs open is well over double what Firefox uses according to the latest Tom’s Hardware tests.

I know you’re not a fan of the tests they do on Tom’s but it’s still the same tests done on the same system with the same software and hardware installed, same network and internet connection etc. It is at least in that sense a level playing field. It appears Firefox is leading the pack in some areas but not others.