Contents

Motivation

Having stronger sense of which features Firebug users actually use, which could help Firebug developers and contributors to better direct the development and design efforts.

Goal

The main thing of a user study is to be clear about what specific question should be actually answered. The critical thing and whole point of the study is to get meaningful data that can be analyzed and results helps to improve Firebug itself. The recommendation from Test Pilot team is as follow:

The main thing is that we strongly recommend you come up with specific questions you want to answer. You can collect a huge amount of metrics, but that may not be as helpful as a more targeted approach. Think of the scientific method: you have a testable hypothesis in mind, "We think that more people use feature A than feature B" and then you design an experiment that attempts to disprove your hypothesis.

An example question in case of a Firefox study can be: How many tabs have Firefox users usually opened at the same time?

Another reason why to start with simple question is to perform the first study quickly, analyze data and start to seeing some results. This helps to make sure it's effecting and working way how to improve Firebug, before investing a lot of development time.

Possible questions we can try to answer for Firebug could be as follows
(subject for discussion)

What options are users actually changing?

What command line APIs users use the most?

What Firebug panels users use the most?

How many people use Firebug detached?

Analysis

As soon as a collected data give us a clear answer for a specific study, the other question is how this helps to improve Firebug? This is the hardest and final part of a study.

Here are some example analysis and possible impacts on Firebug UX (using the same studies/questions from the previous paragraph).

What options are users actually changing?

What if the result says that less than 1% of users never change the 'Show Preview Tooltips' option. Couldn't this be a hidden option accessible through about:config only then? The less options in the UI the better.

What command line APIs users use the most?

What if group of users using heavily the command line, almost never use the 'debug()' function? Does this mean that it's not documented properly and they don't know it exists or how to use it?

What Firebug panels users use the most?

What if less than 3% users use the CSS panel? Couldn't we hide it by default?

How many people use Firebug detached?

If more than 40% of users use Firebug UI detached, doesn't this mean that we need to put more efforts to testing this use case?

Risks

The main risk here is obvious. In case of vague collected data, or missing contexts, there can be less chance to get an effective and helpful results. To minimize this risk, it should be discussed what is expected from a study and what impact it could have on Firebug.

Test Pilot

There is already an existing platform aimed at collecting structured user feedback for various Mozilla labs experiments. So, the natural step is to consider how this existing service could be utilized in case of Firebug user studies and how Test Pilot tool can help.

From a preceding discussion, following seem to be the best approach.

Firebug implements its own data collection mechanism. This way, the study is not limited to users who have the Test Pilot extension installed.

Firebug utilizes an existing Test Pilot infrastructure. In particular, servers for data collection and perhaps also help with data analysis.

Technical Details

Test Pilot uploads gathered data from the client to the data collection server by making an HTTP POST via an XMLHttpRequest. For this to work, the study needs to have a unique test ID number that is consequently used as a parameter of the request to identify particular study.

To summarize, two parameter are required for each such request.

testidUnique ID of a test study

dataArbitrary text string

Test pilot tools are optimized for processing CVS files and so, it could be effective for Firebug to use the same format. However, e.g. JSON format can be also considered.