Testing jQuery in the various browsers is a daunting task as John explains:
In the jQuery project we try to support the current version of all major browsers, the last released version, and the upcoming nightlies/betas (we balance this a little bit with how rapidly users upgrade browsers - Safari and Opera users upgrade very quickly).
At the time of this post that includes 12 browsers.
- Internet Explorer 6, 7, 8. (Not including 8 in 7 mode.)
- Firefox 2, 3, Nightly.
- Safari 3.2, 4.
- Opera 9.6, 10.
- Chrome 1, 2.
Of course, that's just on Windows and doesn't include OS X or Linux. For the sake of sanity in the jQuery project we generally only test on one platform - but ideally we should be testing Firefox, Safari, and Opera (the only multi-platform browsers) on all platforms.
Some test suites (such as Yahoo UI, jQuery UI, and Selenium) have ways of automating pieces of user interaction (you can write tests like 'Click this button the click this other thing'). For most cases this works pretty well. However all of this is just an approximation of the actual interaction that a user may employ. Nothing beats having real people manually run through some easily-reproducible (and verifiable) tests by hand.
- It isn't able to test against non-desktop machines. Each server must be running a daemon to handle the batches of jobs - this leaves mobile devices out of the picture.
- It can't test against unknown browsers. Each browser needs special code to hook in to triggering the loading of the browser by Selenium, thus an unknown browser (such IE 8, Opera 10, Firefox Nightly, or Chrome) may not be able to run.
The libraries can also push manual tests out to the users. A user will be notified when new manual tests arrive (maybe via an audible cue?) which they can then quickly run through.
All of this help from the users wouldn't be for nothing, though: There'd be high score boards keeping track of the users who participate the most and libraries could award the top participants with prizes (t-shirts, mugs, books, etc.)
InfoQ contacted John Resig for more information.
InfoQ: With what license will TestSwarm be released?
InfoQ: Do you think that it will be valuable to other organizations besides framework vendors?
I absolutely think that this will be useful to other groups and vendors. I imagine the growth would work something like this:
- Open to any organization that is comfortable with having their tests be run by anonymous users.
Of course, through all of this, any organization or group could simply download the server software and run their own instance of the swarm - there is no requirement that it has to go through the main Test Swarm server.
On his original announcement John also mentions a "corporate" version of Test Swarm:
If you are interested in Test Swarm you can sign up for the alpha.