Test Swarm Alpha Open

TestSwarm, the project that I’ve been working on over the past 6 months, or so, is now open to the public. Mozilla has been very gracious, allowing me to work on this project exclusively. At the beginning of April I moved from my old position as a JavaScript Evangelist on the Mozilla Evangelism team to that of a JavaScript Tool Developer on the new Developer Tools team (whose other major project is Bespin).

For more information on Test Swarm I’ve written up a detailed explanation of what Test Swarm provides and where it fits into the landscape of JavaScript developer tools.

TestSwarm ended up being a very challenging project to get to an alpha state (and probably will be even more challenging to get to a final release state). Dealing with cross-browser incompatibilities, cross-domain test suite execution, and asynchronous, distributed, client execution has been more than enough to make for a surprisingly difficult project. It’s mostly written in PHP and uses MySQL as a back end (allowing it to run in virtually any environment). Patches will absolutely be appreciated.

This project has been a long time coming now, the first inklings started back in 2007. Some of us on the jQuery team were discussing ways to distribute the test suite load to multiple browsers in an automated fashion. Andy Kent came along and proposed a participatory application for testing visual code (such as jQuery UI). We worked on that code base for a while but it didn’t get off the ground. Eventually I decided to re-tackle the problem early on in 2009. Even in its rough alpha state we’ve already been able to make great use of TestSwarm. For example, here’s a view of jQuery commits run in TestSwarm:

The vertical axis is SVN commits to jQuery (newer commits at the top), the horizontal axis are all the different browsers that we target. Using TestSwarm we’ve been able to easily spot regressions and fix them with a minimum amount of hassle (especially since all the results are logged).

And this is only the beginning. There are so many different directions in which Test Swarm can be taken. For example:

A pastebin-like service where you can drop in code and see the results come back, from many browsers, in real-time.

IDE integration for sending minor changes out for quick testing.

Manual testing of user interface code. Pushing manual tests, with instructions, to users for them to walk through.

Distributing tests to any number of browsers, rather than a specific sub-set. (You could use this to embed a tiny iframe in your site to collect test results from a small sampling of our users.)

The ability to drive and test browser code or extensions.

And the list goes on. I’m definitely curious to see what directions the community is interested in driving the code base. I’ve gotten it to a level where it’s particularly useful for me and the jQuery team – where should we go from here?

@James Brooks: It’s going to depend upon which browser you’re using – if you’re not using a browser that’s needed by the swarm then there isn’t much need to join :-) As I mentioned in the “don’t need help note” please send your useragent info to the Google Group so that we can look over it, in case there’s a mistake.

@Michaël: Opera is open on OS X, just scroll down. Unfortunately Opera doesn’t announce what version of OS X it’s on so it’s in a generic “OS X” bin (neither does Firefox 2.0). I’m going to wait until there’s an alpha or beta of Chromium before opening it up on the swarm.

Our company would like to use spare CPU cycles on our machines to the benefit of Testswarm. My idea is to simply put an 1×1 pixel IFRAME including http://testswarm.com/run/username on our intranet homepage (which is automatically loaded on bootup) – would that be sufficient to have the test start running? Any drawbacks with this approach. Could IFRAME:ing affect the test correctness in any way?

@Svenne: I’d imagine that that technique would work – I doubt iframeing the tests will have a severe effect – especially considering that they’re iframed already. Any help would be appreciated – thanks!

I’ve been looking in to the source code, and while checking it out I noticed there’s quite a lot of $_REQUEST calls. These are dangerous, because they’re coming from three places (Get, Post, Cookie). It’s always better to specify the origin of the variable.

And furthermore, ereg* functions are deprecated, you should use preg* everywhere. They’re faster, and there’s a lot of goodies that come along with it. :-)

John, what about time measurements for each test? I know, in such distributed testing systems it’s hard to made correct measurements because environment for each browsers are quite different, but it’s very attractive to use similar system for performance testing.

@Richard: I was very deliberate in my use $_REQUEST. Note that I only use $_POST when I explicitly want a POST request coming in and use $_REQUEST for other cases where non-destructive operations are occurring (and where it doesn’t matter if a GET or POST occurs). I did another pass through but didn’t notice any cases where $_REQUEST was being used for a destructive operation.

As far as ereg/preg goes, good catch. I’ll definitely check in to that.

@Dmitry: I actually record the time information for the tests but I don’t have it displayed anywhere yet. I’ll see if I can find a useful way to show that data.

$_REQUEST is the most common source of XSS-attacks, due to the fact that $_COOKIE values are the last values to be copied into it. If you would want just post and get values, just use a simple function:

@John: Just an idea, it’s possible to run small common performance test to determine overall environment performance and then use this info to normalize full execution time somehow (and make it less specific to different environment).

I signed up but haven’t been given any tasks to do yet. Is there a way I can manually kick off to run the tests for a commit? Even if it doesn’t contribute anything, I’d like to run them to see that cool grid. Its a little bit of a bummer get excited at the video and sign up, but not have anything to do :]

I imagine it could work like:

– go to my testswarm page
– no new tests to run. I’m useless :(
– … but on the page is a list of the projects in the system. I can click on jquery
– now I see a list of the recent commits for jquery. I can click a button to run the last commit, but instead I click on a commit link.
– now I see the msg and diff for the commit. I can click a button to run the tests for this commit.

This seems like a great way to get people aware of what active development is going on in their library, and hence more likely to contribute to it in some way.

I think a simple way to chat with other logged on users would also be very cool. In my above scenario, the jquery project could have its own irc-like chat room, though much less daunting for new users! People can talk about their library there, ask questions, watch as new commits come in, use the performance data to brag about how fast their computer is.

As you can see I think this can have a lot of potential even beyond testing – which already is exciting enough!

My cpu is just sitting here waiting to be used john!! Where are my tests?! =)

Very cool stuff tho! Great work as always! Just out of curiosity, have you considered using http push? I’m not familiar with the implications of using it but this seems like it might be a good application for it.

Just a detail, John: I see you already replaced Opera’s icon with the one from Opera 10 on the Test Swarm main page. But Firefox still uses the icon from Firefox 3 and not the new, better one from Firefox 3.5. It’s a little bit funny because, since you work for Mozilla, I would have bet that Firefox would have been the first to have its icon updated on the Test Swarm website :)