WebTest vs. Selenium: Real and Simulated Browser Testing
Functional testing tools for web applications come in a variety of styles, but one of the most fundamental differences between the choices are between the tools that drive one or more real web browsers in order to fully recreate a realistic environment, like Selenium, and tools that simulate the way a web-browser operates, like Canoo WebTest. Marc Guillemot compared the two, and in his opinion, WebTest wins, with a score of 13-5.
In comparing the two, Marc scored WebTest and Selenium as winners respectively in each of these categories:
Integration into Development Process
Capture JS Errors
Support for Non-HTML Content
| Browser Fidelity
Support for Badly Formatted HTML Code
He also argued that Ajax tests, generally considered a weakness for browser-simulators, can be considered a tie:
On the other hand, he gives Selenium credit for supporting multiple languages, "Selenium RC has bindings in different languages (Java, Ruby, PHP, etc.) whereas WebTest is bound to Ant", supporting badly-formatted HTML and browser fidelity:
As the lead developer for Canoo WebTest and HtmlUnit, Marc is clearly biased towards this style of tool, which he admits, and asks only that you read his analysis before arguing with him:
To be clear, as a WebTest (and HtmlUnit) committer I'm undoubtedly biased. On the other hand, I have experience with huge functional test suites being developed and maintained over periods of years. Trying to be objective, I may overcompensate in the other direction and give Selenium too much credit. Of course I will diligently fix errors I may have in my Selenium understanding. But please read this post until the end before starting with criticisms.
Feedback has been mixed. Vitaly suggested that WebTest and Selenium are apples and oranges: "Selenium, WebTest(HttpUnit), DBUnit, JUnit and others are comp[le]mentary. There are things that you can do with one and cannot do with another." Others discusssed the relative merits of record/playback vs. scripted tests, and approaches for test maintainability. Murali suggested PragmaticQA Element.
Christian rebutted WebTest's Ajax support, saying that in his application, "Even the simplest pages are throwing exceptions, because HtmlUnit can't parse the Dojo import statements."
Tools like WebTest strike me as being somewhat theoretical - they prove that the code works perfectly, but only in an ideal environment that bears little resemblance to production. Real users are using IE or Firefox, and Selenium allows us to do our testing under 'realistic' conditions, with flaky browsers that leak memory, and don't conform to standards.
[N]o client will ever be using the application on the particular engine that WebTest uses, meaning that while it's nice to know it works on something, it doesn't really matter. In contrast, our Selenium tests run on Firefox, and they run on IE, and they catch a lot of the problems that occur when someone hasn't written something in a cross-platform manner.
Finally, Kent Tong imagines bridging the two approaches:
Possible to develop a middle layer that allows one to write one set of tests that can be run on both WebTest and Selenium? This way people can get the best of both worlds by frequently runing WebTest and running Selenium nightly.
Do you use one of these, or another functional testing tool altogether? Does this inspire you to join the discussion about the next-generation of functional testing tools? For more information, read on about Canoo WebTest, Selenium, Testing and the next generation of functional testing tools.
If you want to be able to run both...
WebTest / HtmlUnit Ajax Support
Regarding the quote above that "even the simplest pages are throwing exceptions, because HtmlUnit can't parse the Dojo import statements," I would like to point out that similar statements have *not* been made about other libraries like Prototype or jQuery.
Why? Because WebTest and HtmlUnit have excellent Ajax support, and many people are successfully using it with these libraries. Dojo (and to some extent YUI) are the last holdouts in this regard. Don't expect that to last long ;-)
Selenium... 2 more features to consider
I work with selenium, and I would like to add that XPath is supported by the tool, and Extensibility is not only supported but encouraged by openSQA (selenium open project).
Not just Apples and Oranges
As one of the rebuttal pieces points out "Browser Fidelity" is probably one of the most important "points" that could be scored on. "Predictable Behavior" is only a point if that's what you can count on in the real world.
I've found Selenium to be fairly reliable - places where the tests break down are often indicative of something wrong with the application; or something that is likely to be problematic when the application goes into production. Not to mention it's hard for people to get into the habit of learning to do Selenium tests as "atomic" features.
BAD Selenium Test:
Start on Home page -> Test Feature A -> Test Feature B -> Test Feature C, etc.
GOOD Selenium Test:
Start on Home page -> Test Feature A -> Go Home -> Test Feature B -> Go Home, etc.
Essentially a bad Selenium test is like a run-on sentence, or the way a small child tells a story: "And then... And then... And then..."
HTTPUnit and Canoo are great _developer_ testing tools. They're particular good about exercising code in ways that something like Selenium (and typical users of Selenium) doesn't care about. Inspecting HTTP headers, checking out the various representation types of a REST-based application, etc. I'm not entirely certain that most of what people do with Canoo really constitutes "functional" tests - a difference that comes down to semantics in large part. A suite of unit tests which closely - but do not exactly - simulate the user environment - might better be called "feature tests". You still have to see how it works in a real browser - not idealized tests which explain how it should work.
Re: Not just Apples and Oranges