Designing great products that make sense & bring joy

Say Hello to the first Mobile Test Pilot study

After many months of development and testing, I’m pleased to announce that we have finally released the first version of Test Pilot for mobile Firefox. This add-on will let users of mobile Firefox participate for the first time in quantitative usability studies to help improve Firefox design, the same way that users of Test Pilot for desktop Firefox have been able to do since 2009.

Mobile Test Pilot is not bundled with beta versions of mobile Firefox at this time. To participate in studies, you can download the add-on from addons.mozilla.org to your Android device. (The easiest way to do this is to go to the control panel, click the Add-ons tab, scroll down to “Get Add-ons”, and search for “test pilot”.)

The mobile version follows the same privacy policy as the desktop version. That means that you are anonymous; Test Pilot collects no personally-identifiable data, and does not record what sites you visit or what search terms you searched for.

Mobile Test Pilot is very stripped-down compared to desktop Test Pilot. Almost all of the UI is gone, as is much of the code. It will show an Android notification when a study needs your permission to submit. Or you can even turn the notifications off by enabling auto-submit (through the Add-ons manager, under Test Pilot options) and then you’ll never even notice that Test Pilot is there.

Developing for the small screen is an excellent exercise in minimalism. It’s shown me that I can do away with even more UI cruft than I thought possible. Honestly, we could replace desktop Test Pilot with a port of the mobile version and it would probably be an improvement. Less is more!

Of course the Test Pilot extension is nothing without a study to run. Simultaneously with the extension we have also released the first study for mobile Firefox, the Mobile Heatmap study. Inspired by the desktop Firefox heatmap, this study will simply count the times a user interacts with each element of the mobile Firefox UI. We’ll use this data to get a very general overview of how people are using mobile Firefox, identify the most and least used interface elements, and see if there are controls that are frequently used together. I hope this basic study will be a starting point to inspire many follow-up studies that explore particular questions in greater depth.

Because smartphones and tablets are only a few years old, there are many more unanswered questions in the field of smartphone and tablet UX than in desktop software UX. Designers are just beginning to figure out the best ways to design for small screens and multi-touch input. I feel like there are more opportunities here for original research that can lead the way to better designs, and I hope that this study will be the first of many to help us create the mobile software interactions of the future.

If you’re using Firefox on an Android smartphone or tablet, please download Test Pilot today and help us improve the product by anonymously sharing your usage data. (And of course, let us know about bugs that you find or studies that you would like to see.) Thank you!

18 responses

I have to give a big big round of applauds on this. Jono has been working solo on Test Pilot extensions for a long time, and all of us around him know how difficult it was to get these releases out of the door (all these code reviews, QA, bugs, FX release schedules, add-on bar changes, different platforms, security and performance reviews etc.) It required a lot of patience, passion, responsibility and coordination to carry these work forward. Without Jono, we couldn’t build our research platform to receive quantitative data.

We also want to give our appreciation to whoever helped us on these releases, thank you for sending us your feedback in the discussion group, reporting bugs and messaging us on twitter. Thank you, guys, we really appreciate your help!

I’m curious as to how you will be able to discover the things we would have done if the interface allowed but which weren’t possible.
Being dissatisfied with the current webkit and opera on my android, I reckon I may try your test pilot :-)
There, my main issue right now is with drop-downs and scrolling in search – seems like pop-up notifications are favoured over selection from history.
Just one other thing…which permissions does it require?

Hi Ann,
It sounds like you’re asking about installing mobile Firefox. (Test Pilot, which this blog post is about, is just an add-on for mobile Firefox.)
You can find out about mobile Firefox from the Android Marketplace page: https://market.android.com/details?id=org.mozilla.firefox&hl=en
According to that page, the permissions it wants are: USB and SD storage, full internet access, and fine GPS. (The latter is because a web page such as Google Maps might ask for your GPS location and the browser needs to be able to provide it if you say yes.)
Test Pilot doesn’t require any additional permissions beyond what Firefox has. It can be installed from the Add-ons Manager tab in the Firefox config screen, if you’re interested in participating.

As the main developer responsible for Test Pilot, Jono is the one to blame for those notifications about new studies that keep popping up in your beta version of Firefox. He's interested in the intersection of usable software design with open-source development, privacy issues, and the need to preserve an open Web. Jono writes games in HTML 5 and plays the accordion and Japanese taiko drums.