Changing How Users Report a Broken Web Site in Firefox

When I began my analysis of the Broken Site Reporter (seen above) in Firefox (first post can be read here and second here), I imagined a series of 4 or 5 posts with a concluding post on changes I thought would be beneficial to the tool. For now, the end is near for the Reporter series. Why did I cut the analysis short? In my mind, the Broken Site Reporter in its current form leads to too many data issues to draw a whole lot more meaningful analysis from it. The subsequent changes I propose in this post should alleviate these concerns, and after getting fresh data we should be able to breath life back into the series.

So what are the problems we’ve seen data-wise with the Reporter? The first issue that comes to mind is category-confusion. By this I mean certain categories are underrepresented/overrepresented because users are getting confused about where their individual problem fits with respect to the categories we let them choose from. Here are the category types:

Three categories that stand out are Behavior wrong, Appearance wrong, and Printed output is wrong. Both Appearance wrong and Printed output is wrong seem to be subsets of Behavior wrong (for that matter, most reported problems technically belong to subsets of Behavior wrong). It is also unclear what meaning is assigned to Printed output is wrong — is it output printed to the screen (in which case the category is dangerously close to Appearance wrong) or is it output physically printed on paper? In this case, some sort of example next to the category could help my confusion.

Another problem is that Other content missing borders close enough to Other problem that the latter is going to cannibalize reports meant for the former.

With these issues at hand I have come up with a very rough sketch of changes I’d like to see made to the Reporter (Keep in mind that I know absolutely nothing about UI — these pictures are more intended to outline my ideas in a concise way than to provide any sort of finalized draft of what the Reporter should look like; criticism is welcome).

The biggest change in this new Reporter is the narrowing of the problem types. I have drilled them down to five distinct problems (Behavior wrong, Disability access, Printed output is wrong, and Other content missing have all been eliminated). In this way, much of the old Behavior wrong and Printed output is wrong responses can be absorbed into the more concise Appearance Wrong category; reports from these categories that do not get redirected to Appearance Wrong will hopefully be sent to Other Problem. The question mark next to the categories is meant to be a link to a site with descriptions of each problem type and examples of reports and their corresponding appropriate category.

Disability access represents a major hurdle for data analysis in the current Reporter implementation. The category is meant for human-disabilities, but most of the reports it receives are from users’ connections timing out or “Server not Found” errors. These are not the types of errors the Reporter is built for, and they end up skewing our data. In light of this, I am proposing that the Reporter detect these errors when present, and when a user tries to report a broken website, give the user the option of reporting the error in a way like this:

With an error checking page such as this we can both clean up our data and get new useful information about the state of different websites (which sites are down at what time, how often certain sites are reported as down, etc.).

With modifications to the Reporter like these we can get out much cleaner data that will help us better respond to problems in Firefox. Hopefully we’ll see some changes to the tool soon so we can get more actionable data!

11 responses

We’re waiting on feedback for these changes. This is really the first discussion about making any changes to the tool, so we’ll gauge response and then go from there. Hopefully there will eventually be a bug or two relating to these changes

I don’t understand why knowing about timeouts is sometimes useful to Mozilla. I have no idea what to do with the choice between “report connection time out” and “report broken site” and I wouldn’t expect users to know what to do with it either. Instead, I think an extra check box should appear in that case, checked by default, with an ⓘ thingie next to it if needed.

Is “Plugin not shown” really worth keeping? I wouldn’t expect many users to be able to figure this one out. Instead, if Firefox can detect that a plugin is missing, make an extra checked-by-default checkbox appear saying “Web page requires plugin for application/x-director, which I don’t have” or “… which I have disabled”.

Another case where we could automatically add a new checkbox is if the user has tried to print the page. Then it’s safe to put “[ ] Looks wrong when printed on paper” or even “[ ] Printed output wrong”.

Please drop the Title Case.

It might be good to add a checkbox labeled “Tell Mozilla what addons I have”. Then we can find out if AdBlock Plus or NoScript breaks some sites in subtle ways.

I think the web site reporter.mozilla.org needs a lot more love than the in-product dialogs. It’s nearly impossible to get useful information out of it because the query UI is so bad, you have to click each item to see the description, and hostname queries time out.

It’s great to see the site reporter getting some much needed attention. Until recently I regularly looked at the submitted reports to check out problem sites.

A few things I noticed:

Users appear to frequently be using the Site Reporter thinking it is a way to get user assistance. Perhaps there is some way to make the purpose of the reporter clearer? Users want “support”, and I think they may be seeing and misunderstanding the word “support” in that first option. It might be a good idea if it were reworded to “site does not permit Firefox” or something like that.

Users report problems with Flash not installing or working on sites that need it. I wonder if there is any way to programatically detect that? (and better yet, automatically fix the problem for the user if possible)

Users report sites that look wrong because of inadvertently blocked images (When images are blocked from a site, there REALLY should be some visual indication something is being blocked and easily permitting people to unblock images, similar to how the popup blocker works.)

I had noticed numerous reports about being unable to open sites after Firefox updates. (I would suspect a firewall problem but they can contact the site reporter.)

And yes, I had also noticed many people submit a report whenever they get a 404 page, a connection timed out, or some other similar site error. Programatically detecting these errors would be very helpful.

It might also be nice if the reporter utility required or encouraged the user to enter some text to describe the problem. As it is when I query the database these days I exclude reports with no description as they are usually useless.

I would like to see just one small but significant change: rename the menu item. “Report broken web site” sounds like it’s only for when you think there’s something wrong with the web site (i.e. a Tech Evangelism bug); I think “Report a problem” would be much more appropriate.

Other thoughts: We might want to consider asking for contact information, and can we merge this system with hendrix?

Maybe the following about timeouts would interest Mozilla: Practically all my timeouts are from Mozilla itself (mostly bugzilla.mozilla.org and addons.mozilla.org) at startup with a multitab homepage. (Reloading succeeds once all tabs have finished loading — or failed to load.)

We’d like to isolate (but keep) connection information because we’re interested in knowing when websites are down or not working properly. This is because in the future we might choose to integrate this type of data into a project like Harvard’s Herdict program, which runs a real-time map of website outages across the globe.

Isn’t it worth keeping a separate category (perhaps with a better name) for printing problems? It’s a clearly-defined subset, and printing has been a historical weakness of ours so knowing how much users run into it is useful.

Zack: that’s by design. Reporter is about Tech Evangelism bugs, it’s not a general Firefox bug reporting mechanism.

ehergenrader: Aren’t there people, e.g. NetCraft, who already keep such data to a much better level of fidelity than we ever could with random point-in-time reports from users?

I changed web browsers from internet explorer to firefox by moxilla hoping for improvements but both have continual connection and time on site problems..it sucks to pay for browsing on the web with poor results. I’ll keep looking…

Help > Report Broken Web Site is gone from Firefox 4.0b8pre (bug 572695), replaced by Help > Submit Feedback… As this blog post is one of the top search results for the feature and its title directly applies to the change in FF4, maybe you could add an UPDATE explaining what happened. Thanks!