Functional testing tools for web applications come in a variety of styles, but one of the most fundamental differences between the choices are between the tools that drive one or more real web browsers in order to fully recreate a realistic environment, like Selenium, and tools that simulate the way a web-browser operates, like Canoo WebTest. Marc Guillemot compared the two, and in his opinion, WebTest wins, with a score of 13-5.
In comparing the two, Marc scored WebTest and Selenium as winners respectively in each of these categories:
| Canoo WebTest | Selenium | Tied |
| Reports Speed Integration into Development Process Scalability Capture JS Errors Documentation Predictable Behaviour XPath Support Extensibility Data-Driven Tests Internationalization Support Support for Non-HTML Content |
Browser Fidelity Beginner Friendly Support for Badly Formatted HTML Code Multi-Language Support |
Testing Ajax |
Marc suggested that tests are never fast enough, but that "WebTest has simply less to do and everything happens in the JVM." He also argues that Selenium doesn't catch Javascript errors that don't otherwise cause tests to fail:
Would you accept compilation errors in your program as long as your unit test pass? Surely not! But in fact this is exactly what you do with Selenium as it doesn't detect the javascript errors contained in your application (unless they directly impact the specific tests causing them to fail).
He also argued that Ajax tests, generally considered a weakness for browser-simulators, can be considered a tie:
Contrary to popular belief, you don't need to run your test as JavaScript inside a browser to test AJAX functionality. HtmlUnit and thus WebTest is just as well up to the task. It can even be considered superior as it allows better control over how to schedule the in-page requests making the unpredictable browser behavior predictable (see for instance my previous post).
On the other hand, he gives Selenium credit for supporting multiple languages, "Selenium RC has bindings in different languages (Java, Ruby, PHP, etc.) whereas WebTest is bound to Ant", supporting badly-formatted HTML and browser fidelity:
HtmlUnit's JavaScript support has made impressive progress but it still doesn't (and will never) behave exactly like a normal browser. Even though Selenium modifies the normal JavaScript execution of an application, it uses the browser itself and therefore is nearer to the standard behaviour of the browser.
As the lead developer for Canoo WebTest and HtmlUnit, Marc is clearly biased towards this style of tool, which he admits, and asks only that you read his analysis before arguing with him:
To be clear, as a WebTest (and HtmlUnit) committer I'm undoubtedly biased. On the other hand, I have experience with huge functional test suites being developed and maintained over periods of years. Trying to be objective, I may overcompensate in the other direction and give Selenium too much credit. Of course I will diligently fix errors I may have in my Selenium understanding. But please read this post until the end before starting with criticisms.
Feedback has been mixed. Vitaly suggested that WebTest and Selenium are apples and oranges: "Selenium, WebTest(HttpUnit), DBUnit, JUnit and others are comp[le]mentary. There are things that you can do with one and cannot do with another." Others discusssed the relative merits of record/playback vs. scripted tests, and approaches for test maintainability. Murali suggested PragmaticQA Element.
Christian rebutted WebTest's Ajax support, saying that in his application, "Even the simplest pages are throwing exceptions, because HtmlUnit can't parse the Dojo import statements."
For Simon, browser fidelity is the most important point:
Tools like WebTest strike me as being somewhat theoretical - they prove that the code works perfectly, but only in an ideal environment that bears little resemblance to production. Real users are using IE or Firefox, and Selenium allows us to do our testing under 'realistic' conditions, with flaky browsers that leak memory, and don't conform to standards.
[N]o client will ever be using the application on the particular engine that WebTest uses, meaning that while it's nice to know it works on something, it doesn't really matter. In contrast, our Selenium tests run on Firefox, and they run on IE, and they catch a lot of the problems that occur when someone hasn't written something in a cross-platform manner.
Finally, Kent Tong imagines bridging the two approaches:
Possible to develop a middle layer that allows one to write one set of tests that can be run on both WebTest and Selenium? This way people can get the best of both worlds by frequently runing WebTest and running Selenium nightly.
Do you use one of these, or another functional testing tool altogether? Does this inspire you to join the discussion about the next-generation of functional testing tools? For more information, read on about Canoo WebTest, Selenium, Testing and the next generation of functional testing tools.
Community comments
If you want to be able to run both...
by Jason Yip,
WebTest / HtmlUnit Ajax Support
by Daniel G.,
Selenium... 2 more features to consider
by Alvaro Gareppe,
Not just Apples and Oranges
by Porter Woodward,
Re: Not just Apples and Oranges
by Paul King,
If you want to be able to run both...
by Jason Yip,
Your message is awaiting moderation. Thank you for participating in the discussion.
See WebDriver, www.youtube.com/watch?v=tGu1ud7hk5I
code.google.com/p/webdriver/
WebTest / HtmlUnit Ajax Support
by Daniel G.,
Your message is awaiting moderation. Thank you for participating in the discussion.
It's been very interesting to see the various views on this matchup!
Regarding the quote above that "even the simplest pages are throwing exceptions, because HtmlUnit can't parse the Dojo import statements," I would like to point out that similar statements have *not* been made about other libraries like Prototype or jQuery.
Why? Because WebTest and HtmlUnit have excellent Ajax support, and many people are successfully using it with these libraries. Dojo (and to some extent YUI) are the last holdouts in this regard. Don't expect that to last long ;-)
Selenium... 2 more features to consider
by Alvaro Gareppe,
Your message is awaiting moderation. Thank you for participating in the discussion.
Canoo WebTest looks like a very powerful tool.
I work with selenium, and I would like to add that XPath is supported by the tool, and Extensibility is not only supported but encouraged by openSQA (selenium open project).
Not just Apples and Oranges
by Porter Woodward,
Your message is awaiting moderation. Thank you for participating in the discussion.
While an opinion piece like this is useful to keep the conversation going as regards effective testing in a web environment, it (the initial opinion piece linked to) doesn't just suffer from bias. It suffers from a methodological error. I don't know that it's possible to simply score testing tools as though it were 13 to 5 or any other simplistic means.
As one of the rebuttal pieces points out "Browser Fidelity" is probably one of the most important "points" that could be scored on. "Predictable Behavior" is only a point if that's what you can count on in the real world.
I've found Selenium to be fairly reliable - places where the tests break down are often indicative of something wrong with the application; or something that is likely to be problematic when the application goes into production. Not to mention it's hard for people to get into the habit of learning to do Selenium tests as "atomic" features.
BAD Selenium Test:
Start on Home page -> Test Feature A -> Test Feature B -> Test Feature C, etc.
GOOD Selenium Test:
Start on Home page -> Test Feature A -> Go Home -> Test Feature B -> Go Home, etc.
Essentially a bad Selenium test is like a run-on sentence, or the way a small child tells a story: "And then... And then... And then..."
HTTPUnit and Canoo are great _developer_ testing tools. They're particular good about exercising code in ways that something like Selenium (and typical users of Selenium) doesn't care about. Inspecting HTTP headers, checking out the various representation types of a REST-based application, etc. I'm not entirely certain that most of what people do with Canoo really constitutes "functional" tests - a difference that comes down to semantics in large part. A suite of unit tests which closely - but do not exactly - simulate the user environment - might better be called "feature tests". You still have to see how it works in a real browser - not idealized tests which explain how it should work.
Re: Not just Apples and Oranges
by Paul King,
Your message is awaiting moderation. Thank you for participating in the discussion.
My experiences with using WebTest is that you rarely use the low-level checks like checking HTTP Headers. They are there if you want them but mostly we write functional tests in a DSL that matches the customers domain language (typically a layer above interactions with the browser). We tend to use Groovy for these and write a thin API layer above WebTest. WebTest's excellent reporting provides us with a great way to get feedback directly from the customers about whether the test is doing what is intended.