News

Crowdsourcing JavaScript Integration Testing with Test Swarm

Posted by Dionysios G. Synodinos on Mar 24, 2009

Community: .NET,; Ruby,; Java
Topics: Javascript ,; Rich Internet Apps
Tags: jquery ,; Continuous Integration

John Resig creator of the jQuery JavaScript library, has released Test Swarm, a platform for distributed continuous integration testing for client-side JavaScript. Frustrated with traditional JavaScript testing environments that don’t scale, John’s new project which is currently is private alpha, aims to provide a systems for outsourcing browser related testing to large groups of people or communities.

Testing jQuery in the various browsers is a daunting task as John explains:

In the jQuery project we try to support the current version of all major browsers, the last released version, and the upcoming nightlies/betas (we balance this a little bit with how rapidly users upgrade browsers - Safari and Opera users upgrade very quickly).

At the time of this post that includes 12 browsers.

Internet Explorer 6, 7, 8. (Not including 8 in 7 mode.)

Firefox 2, 3, Nightly.

Safari 3.2, 4.

Opera 9.6, 10.

Chrome 1, 2.

Of course, that's just on Windows and doesn't include OS X or Linux. For the sake of sanity in the jQuery project we generally only test on one platform - but ideally we should be testing Firefox, Safari, and Opera (the only multi-platform browsers) on all platforms.

The end result is that we need to run 10 separate test suites in 12 separate browsers before and after every single commit to jQuery core. Cross-Browser JavaScript testing does not scale.

The problem for jQuery testing, as for any JavaScript-heavy web application becomes even worst when the tests require interaction with a user:

Some test suites (such as Yahoo UI, jQuery UI, and Selenium) have ways of automating pieces of user interaction (you can write tests like 'Click this button the click this other thing'). For most cases this works pretty well. However all of this is just an approximation of the actual interaction that a user may employ. Nothing beats having real people manually run through some easily-reproducible (and verifiable) tests by hand.

This is the biggest scaling problem of all. Take the previous problem of scaling automated test suites and multiply it the number of tests that you want to run. 100 tests in 12 browsers run on every commit by a human is just insane. There has to be a better way since it's obvious that Cross-Browser JavaScript testing does not scale.

Current solutions like Selenium Grid have several limitations:

As far as I can tell, Selenium Grid requires that you use Selenium to run your tests. Currently no major JavaScript library uses Selenium (and it would be a major shift in order to do so).

It isn't able to test against non-desktop machines. Each server must be running a daemon to handle the batches of jobs - this leaves mobile devices out of the picture.

It can't test against unknown browsers. Each browser needs special code to hook in to triggering the loading of the browser by Selenium, thus an unknown browser (such IE 8, Opera 10, Firefox Nightly, or Chrome) may not be able to run.

And most importantly: Selenium Grid requires that you actually own and control a number of machines on which you can run your tests. It's not always feasible, especially in the world of distributed Open Source JavaScript development, to have the finances to have dedicated machines running non-stop. A more cost effective solution is required.

John’s solution, Test Swarm, is based on a JavaScript client that continually pings a central server looking for more tests to run. The server collects test suites and sends them out to the respective clients:

Here's how I envision TestSwarm working out: Open Source JavaScript libraries submit their test suite batches to the central server and users join up to help out. Library users can feel like they're participating and helping the project (which they are!) simply by keeping a couple extra browser windows open during their normal day-to-day activity.

The libraries can also push manual tests out to the users. A user will be notified when new manual tests arrive (maybe via an audible cue?) which they can then quickly run through.

All of this help from the users wouldn't be for nothing, though: There'd be high score boards keeping track of the users who participate the most and libraries could award the top participants with prizes (t-shirts, mugs, books, etc.)

InfoQ contacted John Resig for more information.

InfoQ: With what license will TestSwarm be released?

It will be released under an MIT license (the server software will be PHP and MySQL, client-side will be some simple JavaScript).

InfoQ: Do you think that it will be valuable to other organizations besides framework vendors?

I absolutely think that this will be useful to other groups and vendors. I imagine the growth would work something like this:

Start by opening to the major JavaScript libraries (all of which have established test suites and testing regimes).

Open up to any Open Source JavaScript project.

Open to any organization that is comfortable with having their tests be run by anonymous users.

Of course, through all of this, any organization or group could simply download the server software and run their own instance of the swarm - there is no requirement that it has to go through the main Test Swarm server.

On his original announcement John also mentions a "corporate" version of Test Swarm:

There's already been a lot of interest in a "corporate" version of TestSwarm. While I'm not planning on an immediate solution (other than releasing the software completely Open Source) I would like to have some room in place for future expansion (perhaps users could get paid to run through manual tests - sort of a Mechanical Turk for JavaScript testing - I dunno, but there's a lot of fodder here for growth).

If you are interested in Test Swarm you can sign up for the alpha.

You can lso find more information on JavaScript, jQuery, Rich Internet Applications and Continuous Integration right here on InfoQ.