“Browser X is using Y MB of memory with Z tabs open” is a meaningless observation

I read an exchange today about the memory usage of Firefox 7 (which is currently in beta).

It uses 600 MB of RAM with 10 tabs open (different websites).

lol wut? I’m not getting the same results as you. I have 12 tabs open here… RAM usage rarely exceeded 300mb for me. That’s already much better than Chrome, which seems to be using 190mb for just 2 tabs.

Memory consumption is around 300 MB for me too.

I’ve seen variants of this exchange countless times. Unfortunately, observations such as “browser X is using Y MB of memory with Z tabs open” are meaningless. To understand why, consider this alternative, hypothetical exchange:

I ran 3 different programs on my computer, they took 55 seconds to finish.

lol wut? I just ran 8 programs and they only took 20 seconds to finish!

You’d never see an exchange like that, because it’s obvious that a comparison of the run-time of multiple programs is meaningless if you don’t specify what those programs are. Workload matters. And yet, people make comparisons of browser memory usage like this all the time.

Here’s a simple example to show why this isn’t meaningful. I just did some memory measurements with a development build of Firefox 9, using a moderately-used profile with no add-ons installed. First I started it and opened 10 instances of www.google.com.au, one per tab, and the resident set size (RSS) was 139MB. Then I re-started it and opened 10 instances of gmail.com, and the RSS climbed to 651MB.

This is hardly surprising. gmail.com is a fully-fledged email client. www.google.com.au is not much more than a search box.

Why do people so often make these meaningless “Y MB of memory with Z tabs open” comments? My theory is that it’s because most people have no idea how complicated web browsers and web pages are. Mild experience with HTML authoring is endemic — heaps of people have thrown together a basic web page, and it’s just a document with with some structure and styling, right? So they think the difference between the website they wrote for their scout troop in 1997 (the one with the “under construction” animated GIF) and the front page of TechCrunch is merely a difference of degree, not kind.

They’re wrong. For one, HTML and related technologies are hugely more powerful and complicated now than they were a few years ago. Also, their scout troop website probably didn’t contain multiple megabytes of JavaScript code tracking its visitors’ every move. A web browser is not a document viewer, it is a full-blown programming environment with some very sophisticated text and graphical capabilities. A web page is not a document but a program. Therefore, the memory (and CPU) usage of different web pages varies dramatically.

Here’s a check-list of information that you should include if you want an observation about browser memory usage to be meaningful.

What sites do you have open? The more specific, the better. E.g. “Gmail and a few nytimes.com pages” is ok, but listing the exact URLs is better.

How did the browser get in this state? Did you just start it, or have you been using it and visiting other sites for hours?

What memory metric are you using, and how did you measure it? “Explicit” from Firefox’s about:memory? “Private bytes” from the Windows Task Manager? “RSS” from “top” on Mac or Linux? (The full output of Firefox’s about:memory is hugely useful, because it includes dozens of highly relevant measurements.)

In Firefox, do you have any add-ons installed? That can (and often does) make a big difference. (And that’s a topic for another day.)

If you include all that, it’s highly likely that somebody else can reproduce your measurements, which means it’s a highly meaningful observation.

On a related note, I wrote previously about Firefox 7’s memory improvements, saying “Firefox 7 uses less memory than Firefox 6 (and 5 and 4): often 20% to 30% less, and sometimes as much as 50% less.” The presence of the words “often” and “sometimes” in that sentence were deliberate. We’ve seen those numbers in our testing, but once again, workload matters. I really hope the numbers we’ve seen match what normal users see. In fact, I thought that people would independently test these claims shortly after I wrote them. But, to my knowledge, that hasn’t happened so far, even though those claimed improvements have been reported far and wide. (I’ve even seen numerous headlines that say “Firefox 7 to use 50% less memory”, alas.) Maybe once Firefox 7 is officially released people will make independent measurements.

(A final note: suspicious readers may think that I’m trying to obliquely absolve Firefox from any responsibility for its memory usage by blaming web pages instead. I’m not. Firefox is doing much better now, but there’s still plenty of ways it can be improved; please contact me if you want to help.)

23 Responses to “Browser X is using Y MB of memory with Z tabs open” is a meaningless observation

Thanks for setting the record straight. Especially from a developer’s mouth.
80% of reader has absolutely no idea webpage matters in a measurement. They just read the news being reported. And worst things is, most tech site dont understand it either and just report what others find.

I do experience ~30% less memory usage with Firefox 7, but most of the time this is still over shadow by Add on that takes many MB and Places DB that are huge in size. Last week i just help someone on forum to reset up their Firefox Profile, his previous one was running at nearly 600MB and i have absolutely no idea what happen to it.

Excellent article. I myself have made some of those very comparisons without stopping to think about the context and the variables. Hopefully this stems so of those benign comparisons.

Since there is such a nice selection of great browsers today, this ‘how much memory is my browser using’ obsession has become the new metric for how good a browser is. This really clarifies how flawed it is to judge them that way.

Firefox needs something like the task manager in Chrome. The ability to see how much ram each tab and addon use would allow users to become aware of the fact that not all tabs are equal in their resource usage and would shift accountability to the people actually responsible for that resource usage.

But what about going further and having a simple to use button “report my memory usage”
with a few privacy related options (like opened URLs included or not) that would send an extended report to mozilla and would let you better understand situations faced by users and track down real life troubles ?

It could be enabled on in alpha/beta versions ?
Or take the form of an add-on recommended to people facing trouble with their browser

We currently have telemetry, which users can opt into, which provides a few memory measurements. But it would be good to have a “I’m having problems, report my memory usage to Mozilla” button — that idea has come up before and definitely has merit. The privacy issue is critical, we’d need to be careful with that… but if the URLs are removed then the usefulness of the reports drops greatly.

Excellent blog post! Especially this sentence: “A web page is not a document but a program.”
You need to make put it in bold and 100px font and send it to the guys who are developing “Adobe Muse”, to all the print designers who think they can just MS-Frontpage-like “design” a web page – since they think it’s just an online version of their print magazine – and it will work perfectly without any problems. (Read this if you don’t understand what I’m talking about: http://www.netmagazine.com/features/developers-respond-adobe-muse)
People really need to learn that web pages are more than just a document. I would even say it’s software. Like Word for example…it’s software running inside another software (the operating system). And a web page (in this analogy) is also software running inside another software (the browser, the web pages’ operating system).
So if you think about a web page as a program you will understand that those guys are actually only designing the GUI of the software they want to ship. They are designers who want to build an entire “software” by just drawing the circles, squares, texts and images. And they solely rely on Adobe Muse that their software (the web page) works in every operating system (the browser).

I’m sorry for ranting about this on your blog but after reading your post I just had to write this.

Having seen numerous exchanges on tech news sites along those lines, posts like these are definitely needed. At least there’s now something to point people at when they insist on drawing conclusions from their anecdotal observations (whichever the browser used). Once again, spot on njn

Now let’s talk about V8 test, or WebM pictures with enormous gain achieved by droping a huge color profile embeded in the JPEG file, and all these crazy benchmarks out there.
Seriously, benchmarks should be done by measuring real world things most of the time : loading Facebook, Gmail, the NYT, etc.
Of course we need “fictive” benchmarks for advanced technologies like WebGL but measuring 1 millions of dates created in javascript for example, WTF ?!

NJN, how about a test suite? areweslimyet.com/test
With Selenium, TestPilot and other tools, surely it’s not hard to provide users with a consistent memory testing recipe, is it?

Users should be able to run standard memory usage tests on a regular basis to help avoid regressions if nothing else. However it could also be useful for users to be able to see the memory handling over time based on a consistent recipe.

I do not think a post like this will ever do too much good. I understand your frustration however I firmly believe that it’s the responsibility of those with tech skills to facilitate structured user feedback. The average user is not likely to be scientific in their testing unless you give them a specific scientifically structured experiment to run.

Why do people so often make these meaningless “Y MB of memory with Z tabs open” comments? My theory is that it’s because most people have no idea how complicated web browsers and web pages are.

I think there is an even simpler explanation. People see the browser as one application. They start it and then don’t leave it like they do when switching applications.

You most likely have a similar misunderstanding when it comes to any application that works with documents/files, such as office applications. If you have a big document, or photo, it takes more resources. However, since its the users own files they are more aware of the size compared to web pages.

Memory usage from Firefox on OSX is appalling. Doesn’t matter what websites I visit but after an hours browsing it will reach 800-1GB and I have to force quit the application about 6 times a day because of this. This is on 3 different OSX machines and I’m not alone. It never used to be this bad in older versions of FF.

Just using the Windows task manager, private working set for a rough estimate. The browser takes about 1GB instead of 1.2GB at launch after restoring all my tabs.

I can’t say it particularly felt more responsive at launch, but I definitely noticed that it stayed more responsive as I used it. In the past it would climb to 1.5GB pretty quickly and slow way down. With FF7 it climbs to ~1.2GB and stays much more responsive.

Are there meaningful ways we can shift the way people talk about this, while still giving them a succinct way to talk about memory usage? EG, instead of just “Z tabs”, would something like “Z tabs, Y documents, X scripts, W images” be a usefully better (if still imperfect) comparison?

Maybe if we generated that as an easy-to-copy string in about:memory (or about:support), people could start to make use of it and see that memory is influenced by many things.

Do you want to wait 1 year to get an improvement like it was with Firefox 4? And try to explain an average user to install firefox x.y.zzzz. Why so complicated? Install latest firefox release and you’re fine. And every 6 weeks you get new improvements. No need to wait 1 year.

I actually disagree with one small detail: I feel the web browser is still a document viewer first, runtime environment second, and in fact the document viewer aspect of the browser is what causes most of the things I dislike about the idea of a, “web-browser operating system” like ChromeOS.