Designing great products that make sense & bring joy

Measuring Australis.

The goal of the Australis project has been to make Firefox simpler and more engaging, and thereby help users be more effective. Naturally, with every user-interface change we make, we want to evaluate how the design is performing and course-correct where necessary and helpful.

Now that the new interface is being released to a wider audience in Aurora, we are taking the opportunity to further check our work, make final refinements, and gather ideas for the next version.

One of the ways we can understand the impact of new designs is through surveys and user interviews, which we use throughout the design process. With a larger user population like the one using Aurora, though, collecting feedback at scale requires us to go beyond those methods. For that we need to get more into the quantitative side: we need to instrument the browser, and link the data we collect back to the things we care about, such as user-performance and delight.

(As another proxy for the kind of data we get from the surveys, we’re also talking with our User Advocacy and Support team, to see what reactions they’re hearing, how they differ from previous releases, and more importantly how they differ from previous releases with a similar volume of UI changes.)

Our quantitative understanding of how Firefox is being used comes mostly from users who have opted in to sending us their Telemetry data. From this data, we can extract reasonable answers to questions such as “Are people customizing Firefox more?” or “How many tabs is typical?” This is interesting in its own right, but we can also use it to gauge both effectiveness and a sense of agency over one’s browser. Additionally, it will tell us how well we did in determining which menu items we should provide by default and which should be available through customization).

And so we landed a new UITelemetry.jsm module to make adding those probes easier for both desktop Firefox and Firefox for Android. Once it was easy to add new probes, we needed to get a baseline of the current activity so that we have something to compare against to tell how things are changing. To that end, we added around 40 Telemetry probes to the pre-Australis version of Firefox (Holly) To gather data that we can compare with this baseline, we also added similar probes into the Australis version of Firefox. (They will only be similar probes because some of the UI elements have been significantly changed in Australis.)

This will give us some data we can use; however, we know that the people who have opted in to sending Mozilla their Telemetry data don’t behave in exactly the same way as Firefox’s general user base, so to be able to normalize the telemetry data to better match our release population, we will also be checking the data from Telemetry against similar data we receive in the Firefox Health Report. The Firefox Health Report will also give us some more detailed and representative data for other questions we have.

Now that the probes have been on the pre-Australis version of Firefox for a cycle, we have a good baseline for the information we’re tracking. And now that Australis has ridden the train to Aurora, we should have a set of data in a week or two that we’ll be able to compare to the pre-Australis Aurora numbers and get a better idea of whether Australis is having the effect we want.

15 responses

Huh, does Telemetry recognize if people have things like the Classic Theme Restorer installed? Because I would imagine that a lot of people that couldn’t deal with the maimed customizability would install it or a similar add-on.

Also, how would you measure things that are simply broken? Like how you can’t get to some of the printing options without knowing that you have to press alt and go into the file menu. Which is, by the way, worse than the Win 8 shutdown button…

Measuring things that are broken is an interesting question. I think the best way for us to tell if something is broken is by looking for menu items (or buttons or text fields) that see a sharp decline in usage (and then doing some digging to try and figure out why they have declined). Of course, we would need to correlate that with the other ways to access that item. For instance, if the “Bookmark This Page” menu item drops in usage, but way more people click the Bookmark Star, then that’s not really broken, just shifting.

> Our quantitative understanding of how Firefox is being used comes mostly from users who have opted in to sending us their Telemetry data.
> From this data, we can extract reasonable answers to questions such as “Are people customizing Firefox more?”

Does this mean that Mozilla will be able to determine how many users are using the “Classic Theme Restorer” add-on to avoid using the Australis UI?

There is an important difference between telemetry data that indicates “70% of users are using Firefox 29 or later” and “70% of users are using Firefox 29 or later, of which 90% are using the Classic Theme Restorer add-on”.

Well, kind of. Anyone can tell how many people have downloaded the Classic Theme Restorer, by checking the add-on’s page. But that doesn’t tell us why those 1,609 users (at the time of this comment) installed it. Perhaps they really like square tabs, or a differently coloured menu button, or perhaps they are trying to entirely avoid the new UI. Still, if 90% of Australis users install the Classic Theme Restorer, we will certainly take a closer look at why!

Problem is, raw telemetry data sometimes isn’t very helpful. IMHO, the main problem in Australis is the the Firefox menu: it moves literally to the opposite place and, *at the same time*, changes completely how it looks, making it harder to locate it for the user – it s/he manages at all (“Firefox said something about an update and then the big orange button dissipated. I want it back!”). I’m afraid plain users, especially those who aren’t enough into smartphones to recognize the new icon, will have trouble finding it. But most will eventually locate it and start using it.

All this poses two problems. First, once an user finds the new menu button, s/he will start using it as before. So I expect the usage numbers to be comparable to the old button’s ones (which wouldn’t tell us much!). But there is a greatest problem: even if the usage changed a lot, we wouldn’t be sure how it affected plain users. I would bet that installing an alpha version of a web browser isn’t something most plain users do. Because of that, as pointed in the article, telemetry data will only come from advanced users, and it won’t be comparable to data from Firefox 26 (in use by all kinds of users), nor will it allow to draw a picture of the plain user.

Worries like the one you expressed are one of the reasons we explicitly call out the new Menu Button in the first-run tour. And the behaviour you describe would still show up in telemetry as a temporary dip in the Menu Button usage, so that’s something we would look into.

For the second problem you mention, I feel it’s mitigated by two things. First, the numbers we get are only one input into the design process. They aren’t going to force us to make a decision one way or another, they’re merely something we take into account, and use to learn more about our users. Second, we feel we have a reasonable understanding of how the early adopters are different than our general population, and so can compensate for that when we think about what the telemetry data is showing us.

As a theoretical question in reply, if everyone eventually finds the new Menu Button, and uses it just as much as they used the old Firefox Button, is it really a problem?

Well, I think your last question isn’t rethorical – or shouldn’t be. You know, there is another browser that is being pushed a lot by a company that is spending money in ads for it – and that has already noticeably eroded Firefox’s market quota. Non-tech users (i.e.: you uncle, your grandparents, the man running the corner’s hardware store…) keep using the same old software because “it’s familiar”, and will only consider switching when disrupted. My fear with Australis is precisely that such a radical change to Firefox will encourage switching. Also, as the tech-savvy relative and friend of some of those Firefox plain users, I expect to spend some time at the phone the days after Firefox 29 is released (“Antonio, I need your help! I can’t print a web page because the orange button is gone!”).

Anyway, this is only theory. I have 15 years of experience as an UI designer in a small software company (and I feel I know my users pretty well), but I haven’t done usability testing on Firefox Australis. So I would love to be proven wrong. When will you have publishable usage data? Maybe I’m asking for too much, but it would be good to be able to see some usability sessions with the new design, too (it’s always humbling to see a real user facing a new user interface). Will any videos be released?

I didn’t mean it rhetorically, but you raise a good point. On the other hand, one person’s “familiar” is the next person’s “old and dated”, and sticking with the same UI for 12 years doesn’t seem like a good path to greater relevance. (Can you imagine using something like https://wiki.mozilla.org/File:Fx2-new-theme-in-xp-v1.jpg today?)

I don’t know when we’ll have publishable usage data, but I strongly suspect we’ll write some blog posts when we do. :) And we have done usability sessions, which I believe were recorded, but I doubt we’ll release the videos (for privacy reasons). Watching them was quite eye-opening, as it almost always is.

Yes, because that’s exactly how I configure Firefox on all of my machines, minus the bookmarks toolbar.

Again, the common complaint with Australis is that customisability has been lost. The counter-argument of “more options (such as tabs on bottom) means a bigger code-base for us to maintain” is certainly a valid point, but then you might as well say that supporting multiple languages, operating systems, and web technologies gives you more to maintain; it’s true, but the resultant gains make it worthwhile.

Additionally, trying to use the same interface across PCs (large non-touch screen + keyboard) and smartphones (small touch-screen, no keyboard) is a fools errand. It’s analogous to programmers who know one language (such as BASIC) who then refuse to adapt to local idioms when using others (such as C++); it’s better to adapt to the environment you’re in than to take your own faux monocultural one with you wherever you go.

The designers of GNOME 3 and Windows 8 both fell into the “Everything Is Basically An iPhone” UI trap before Mozilla did, and I’d be sorry to see Firefox repeat the same mistake.

It’s amazing how similar Mozilla’s response to the users hatred of Australis is to how Microsoft acted regarding Windows 8. Essentially, most users think it’s complete crap, and yet they insist the users are wrong, or just don’t get it.
To be honest, the ONLY reason I didn’t switch to Chrome (like everyone I know has been telling me to for a couple years) is that I didn’t really like the interface.
Now, I can’t really justify staying with Firefox, as the one last thing Firefox had going for it is vanishing also.

I would be interested to see the studies you’ve done which lead you to conclude that “most users think it’s complete crap”. The user tests and surveys and telemetry measurements we’re doing indicate that the vast majority of users are neutral to positive, and it’s only a tiny (but surprisingly vocal) minority who dislike the new look.

And, as I mentioned above, if 90% of Australis users install the Classic Theme Restorer, we’ll certainly take a long, hard look at why, but it’s currently sitting at 2,563 users, which is a vanishingly small proportion of Australis users. (Hopefully you’ll make that 2,564, by installing it from https://addons.mozilla.org/en-US/firefox/addon/classicthemerestorer/ , so that you can keep the classic theme you love! :)

@Blake-Winton: It’s something very easy. It’s called “opening your eyes and listening with your eyes and ears instead-of looking at the Metrics”.

The metrics won’t tell you EVERYTHING.

Like I keep telling my dad (whose major in College was “Business Administration with emphasis in Marketing” (and my mom, whose major in College was “Business Administration with emphasis in Finance”)), who’s OBSESSED with Numbers and Metrics (if it were a woman, he’d have sex with it (literally. I kid you not)), “Numbers” and “Metrics” won’t tell you the whole story. Nor will they tell you why an individual person likes or doesn’t-like a certain product. The best thing you can do to get the REAL picture, is ask point-blank, one person at a time, one question at a time, why they like or DON’T-like a product, etc. etc.

@Nate: I *COMPLETELY* AGREE with everything Nate said in his two paragraphs.

@Thomas: I *COMPLETELY* AGREE with everything Thomas said in his responses. (Including:

<>

> Can you imagine using something like this today?
Yes, because that’s exactly how I configure Firefox on all of my machines, minus the bookmarks toolbar.
Again, the common complaint with Australis is that customisability has been lost. The counter-argument of “more options (such as tabs on bottom) means a bigger code-base for us to maintain” is certainly a valid point, but then you might as well say that supporting multiple languages, operating systems, and web technologies gives you more to maintain; it’s true, but the resultant gains make it worthwhile.
Additionally, trying to use the same interface across PCs (large non-touch screen + keyboard) and smartphones (small touch-screen, no keyboard) is a fools errand. It’s analogous to programmers who know one language (such as BASIC) who then refuse to adapt to local idioms when using others (such as C++); it’s better to adapt to the environment you’re in than to take your own faux monocultural one with you wherever you go.
The designers of GNOME 3 and Windows 8 both fell into the “Everything Is Basically An iPhone” UI trap before Mozilla did, and I’d be sorry to see Firefox repeat the same mistake.

<> ).

I also have that same interface in ALL the computers (and other can-access-the-World-Wide-Web devices) I use in which you can install Firefox on. (And FYI Mozilla Firefox team (including but not limited to the development team)? I’ve been using Firefox since Firefox version 2.0. And if you release Australis, it’s bye-bye Firefox and no using the World Wide Web unless I really have to (and even then, it will only be for a few minutes then I’ll shut off my computer). Chrome? Too “privacy invasion”. Opera? Same, and too Chromey. Chromium? Why would I want a open-source version of a browser I [already] don’t want and will not use and don’t want to use, etc. etc. Mozilla SeaMonkey? I don’t use it because I don’t have yet a use (and/or need) and/or purpose for it yet. PLUS, I’m not a developer.

Internet Exploder? Why would I go back to a browser that can’t handle as many tabs as I have open at any given one time in Firefox, AND doesn’t have Tab-Groups/Panorama, AND I cannot configure it to have my tans back down to below the address-bar (where they rightfully belong), AND it’s not open-source, AND it’s not as customizable as Firefox, AND is not as light on computer-resources as Firefox is?

Mozilla, you guys are now treading on thin ice. You are now not just going “one step foward, two steps back, one steps foward, two steps back, one step foward, two steps back”, but you guys have ALSO turned your browser into a Chrome-wannabe/Chrome–stand-in/Chrome-lookalike (which also adds it to the list of the other [98% of the l] browsers out there that already LOOK like Chrome and Chromium).

Congratulations: You’re a baby who crawled (on all 4 legs) BACKWARDS and then, because he didn’t see where he was going (nor watch what he was doing nor watch where he was going), he fell right off a cliff.