Add-on Verification Suite is on AMO

RJ Walsh has just completed the first revision of the add-on verification suite we proposed a few months ago, now live on addons.mozilla.org. This suite gives add-on authors and editors an automated tool to examine add-ons for common problems, and recommends fixes where appropriate.

The goals for the project are:

Detect problems in add-ons, including issues with security, compatibility, and other areas. The author can receive alerts if problems are found.

Help editors by automating simple, repetitive tests so they can spend their time doing more interesting things

Maintain the level of excellence in public add-ons that Firefox users expect

Accessing the verification suite

If you are an add-on author log in to your developer control panel and choose “Versions and Files” for one of your add-ons. Find the version you want to analyze and click “No Test Results.” This takes you to the testing screen. We’ll make this process easier in a future update.

What you’ll see

After a brief analysis, a summary is displayed at the top of the page with what tests were run and their results. If you see all green check marks, congratulations! Your add-on has passed the verification suite’s current set of tests.

If you see any warnings or errors scroll down the page to see a detailed report. Beside each message that is reported is a [help] link which points to a page of documentation explaining each test in more detail.

Things to remember

Keep in mind that many tests are written to catch a broad number of cases and may display false positives. Seeing warnings in the add-on analysis does not necessarily mean your add-on has problems – it’s simply raising a flag to double check that area. As usual, we are very open about exactly what we are testing here. This tool, just like AMO, is completely open source and can be browsed online.

Future plans

Over the next few months we’ll refine the tool to prevent false positives and add additional tests. If you can think of other tests that would be helpful for authors or editors please submit a patch or just let us know. We expect to have the tests running automatically when an add-on is uploaded by the end of the year.

If you have any questions, comments, or ideas for improvements please let us know, either as a comment on this blog or on IRC.

14 responses

Ah, I am finally able to run those tests 🙂
Lots of nice warnings. However, finding /eval/ in localizations (particularly in the middle of a word) isn’t really helpful. Maybe change that to /\beval\s*\(/ at least?

Heh, “eval” as part of words in dictionaries also produces security warnings. 🙂

And I saw that you choke on missing install.js for SeaMonkey compatibility on dictionaries, which is nice for MySpell dictionaries, but for HunSpell it’s wrong, as we only support those in 2.x and even SeaMonkey doesn’t need/want install.js there. So, if minVersion for SeaMonkey is 2.0a1 or higher, please omit this check for install.js

1. Would be awesome (probably impossible) to simulate running/testing an add-on on various OS’s. Many to most developers only use or have access to one OS.

Yeah, that would be difficult. We’re looking into VMs for editors to be able to test add-ons on, and we’re looking at some other automated testing like memory leak stuff. Nothing jumps to mind on how to install in multiple OSs with this tool and then actually use the add-on to see if it works. There are just too many variables for each add-on to be able to test functionality.

2. How about the option to add an AMO validation badge/button to an add-on’s page?

That’s an interesting idea. We could even make it automatically show up on the add-on page if there are no warnings/errors. I’m a little concerned that false positives with the current setup will mean “oh well” but will cause some discontent if we started doing public promotions like that.

may I suggest a test to check for hardcoded user interface strings?
Or more simply a detection of the absence of locale support, triggering something like “It seems your extension has no locale support. Though it is no big issue, it is recommended to have at least one en-US locale subfolder for your user interface strings”
(and a link to appropriate documentation)