Functional testing tools for web applications come in a variety of styles, but one of the most fundamental differences between the choices are between the tools that drive one or more real web browsers in order to fully recreate a realistic environment, like Selenium, and tools that simulate the way a web-browser operates, like Canoo WebTest. Marc Guillemot compared the two, and in his opinion, WebTest wins, with a score of 13-5.
In comparing the two, Marc scored WebTest and Selenium as winners respectively in each of these categories:
Canoo WebTest | Selenium | Tied |
Reports Speed Integration into Development Process Scalability Capture JS Errors Documentation Predictable Behaviour XPath Support Extensibility Data-Driven Tests Internationalization Support Support for Non-HTML Content |
Browser Fidelity Beginner Friendly Support for Badly Formatted HTML Code Multi-Language Support |
Testing Ajax |
Marc suggested that tests are never fast enough, but that "WebTest has simply less to do and everything happens in the JVM." He also argues that Selenium doesn't catch Javascript errors that don't otherwise cause tests to fail:
Would you accept compilation errors in your program as long as your unit test pass? Surely not! But in fact this is exactly what you do with Selenium as it doesn't detect the javascript errors contained in your application (unless they directly impact the specific tests causing them to fail).
He also argued that Ajax tests, generally considered a weakness for browser-simulators, can be considered a tie:
Contrary to popular belief, you don't need to run your test as JavaScript inside a browser to test AJAX functionality. HtmlUnit and thus WebTest is just as well up to the task. It can even be considered superior as it allows better control over how to schedule the in-page requests making the unpredictable browser behavior predictable (see for instance my previous post).
On the other hand, he gives Selenium credit for supporting multiple languages, "Selenium RC has bindings in different languages (Java, Ruby, PHP, etc.) whereas WebTest is bound to Ant", supporting badly-formatted HTML and browser fidelity:
HtmlUnit's JavaScript support has made impressive progress but it still doesn't (and will never) behave exactly like a normal browser. Even though Selenium modifies the normal JavaScript execution of an application, it uses the browser itself and therefore is nearer to the standard behaviour of the browser.
As the lead developer for Canoo WebTest and HtmlUnit, Marc is clearly biased towards this style of tool, which he admits, and asks only that you read his analysis before arguing with him:
To be clear, as a WebTest (and HtmlUnit) committer I'm undoubtedly biased. On the other hand, I have experience with huge functional test suites being developed and maintained over periods of years. Trying to be objective, I may overcompensate in the other direction and give Selenium too much credit. Of course I will diligently fix errors I may have in my Selenium understanding. But please read this post until the end before starting with criticisms.
Feedback has been mixed. Vitaly suggested that WebTest and Selenium are apples and oranges: "Selenium, WebTest(HttpUnit), DBUnit, JUnit and others are comp[le]mentary. There are things that you can do with one and cannot do with another." Others discusssed the relative merits of record/playback vs. scripted tests, and approaches for test maintainability. Murali suggested PragmaticQA Element.
Christian rebutted WebTest's Ajax support, saying that in his application, "Even the simplest pages are throwing exceptions, because HtmlUnit can't parse the Dojo import statements."
For Simon, browser fidelity is the most important point:
Tools like WebTest strike me as being somewhat theoretical - they prove that the code works perfectly, but only in an ideal environment that bears little resemblance to production. Real users are using IE or Firefox, and Selenium allows us to do our testing under 'realistic' conditions, with flaky browsers that leak memory, and don't conform to standards.
[N]o client will ever be using the application on the particular engine that WebTest uses, meaning that while it's nice to know it works on something, it doesn't really matter. In contrast, our Selenium tests run on Firefox, and they run on IE, and they catch a lot of the problems that occur when someone hasn't written something in a cross-platform manner.
Finally, Kent Tong imagines bridging the two approaches:
Possible to develop a middle layer that allows one to write one set of tests that can be run on both WebTest and Selenium? This way people can get the best of both worlds by frequently runing WebTest and running Selenium nightly.
Do you use one of these, or another functional testing tool altogether? Does this inspire you to join the discussion about the next-generation of functional testing tools? For more information, read on about Canoo WebTest, Selenium, Testing and the next generation of functional testing tools.