Functional Test Tools Workshop

Posted by Mark Levison on Sep 17, 2009

Sections

Process & Practices,

Architecture & Design,

Development

Topics

Tools ,

Software Testing ,

Agile ,

Agile Techniques

Tags

Acceptance Testing ,

Testing ,

Agile2009 ,

WatiN ,

Watir ,

Selenium

The 3rd workshop of the Agile Alliance Functional Testing Tools group was held the Sunday before Agile2009. The open space session was open to anyone interested in advancing the state of the art with Functional (vs Unit) test tools. Among others, the participants were the creators of and contributors to: Selenium, SWAT, Cucumber, WebTest, RobotFramework and Twist.

As an open space, the agenda and sessions are created on the fly – this year there were 3 time slots and 4 locations allowing for up 12 sessions.

According to Lisa Crispin, co-author of Agile Testing, the first session was a series of lightening talks demoing various tools that aren’t as well known as Selenium and Watir: Canoo WebTest, Twist, Cucumber, Robot Framework and SWAT. Lisa says: “I was impressed by Robot Framework, an open source tool whose developer is Pekka Klärck. It has a lot of flexibility. It uses a tabular format similar to FitNesse except there is only one type of table. You can do keyword driven, data driven and even BDD style tests. It also takes command line arguments. It lets you have internal and external libraries such as Swing … There was a lot of interest in SWAT, nobody had seen it before and I think they were particularly impressed by the IDE.”

Paul King, WebTest and Groovy contributor, proposed: “doing more mixing and matching of testing frameworks/drivers/utilities/tools. He had a nice graphic showing what is available. We all agreed there are enough test runners, and the developer community should focus on solving new problems”

In another morning session, Matt Wynne, Richard Lawrence, Aslak Hellesøy and this reporter discussed what would be required to port Cucumber to .NET. As it turns out, as long as users are prepared to run the Cucumber test run under Ruby then creating .NET bindings for Cucumber is relatively simple if the Cucumber developers can find an simple way of communicating with the test application. A model similar to the FitNesse Slim approach was proposed. By the end of the session Matt and Richard started work on the solution.

In last years session it was agreed that we should document the existing tools in this space but the effort floundered. This year, Gerard Meszaros created a spreadsheet (contact this reporter mark AT mlevison DOT com for editing access) based on what he saw in the morning demos. In one of the afternoon sessions a group got together and helped fill in more tools in the spreadsheet – as of today they spreadsheet documents: SWAT, Cucumber, WebTest, RobotFramework, Twist, TestSwarm, JBehave, Fit, FitNesse, FitNesseSlim, UltiFit, Watir, Watin, Abbot, Fest, White, Sahi and Sahi-Java. Notably missing are notes about Selenium.

In the late afternoon there was a discussion around the role of record and playback tools such as the Selenium IDE. Lisa Crispin thinks that capture/playback tools “can be a great way to help learn a new tool, and also can be helpful in debugging test scripts or figuring out the right statements to use in a particular test. However, people shouldn’t get bogged down in only using capture/playback”. Jason Huggins, Selenium developer, explained that he is troubled by the general use of the Selenium IDE (really just a record/playback tool). It was originally intended as a “little 'trainer' airplane that jet pilots train on first. The pilots can learn a lot from the trainer, but eventually they have to move up to a real jet.” Suggestions for making this distinction more clear included (notes from Mike Longin):

• Rename Selenium IDE to Selenium Trainer to help promote the idea that recording really isn’t the end all for automation
• Create some kind of complexity meter on the recorder that shows when your recording is getting so complex that maybe its time to learn new automation techniques

Mike echoes Paul’s comment “We have numerous drivers, frameworks, and runners out there and now maybe is the time to start looking at integration techniques.  How do we take all of these great tools and start combining them.”

Pekka Klärck thanked Jennitta Andrea and Elisabeth Hendrickson for organizing the workshop again this year.

1. Back to top

   Document of available tools

   by Kay Johansen

   Kudos to Gerard for creating the spreadsheet of available tools. Should this information go on Wikipedia?

   Reply

2. Back to top

   Re: Document of available tools

   by Mark Levison

   Thanks for the comment Kay - Gerard does deserve thanks but there were a lot more people who have worked on it since and deserve a thanks too. As to Wikipedia - but its not flexible enough for what we really need. We have a community license for Atlassian's Confluence and it will do the job. The next step is to get the rest of the major tools documented.

   Reply

3. Back to top

   I updated the information in the spreadsheet on Selenium for you.

   by Jason Huggins

   Of course, it is my strong opinion that most of the rows/features in the spreadsheet have *nothing* to do an automation tool, and *everything* to do with the capabilities of programming languages, development IDEs, and unit test harnesses. For Selenium Remote Control, why ask if a tool has keyword completion if it's an API? Why ask if it has database support if it's an API? If it's an application, why ask if it supports source control? It's like asking if a tractor has moon landing support. Answer yes or no to that type of question ignores the silliness of the question. This testing tools spreadsheet should at least clearly make the distinction betwen APIs and Applications... Selenium has both an API (Remote Control), and an application (Selenium IDE for Firefox). Between the two of them, almost all the answers are "Yes"... But most of the answers are "Yes"... because if you write tests in a real programming language, you can do anything and everything you want.

   Reply

4. Back to top

   Re: I updated the information in the spreadsheet on Selenium for you.

   by Mark Levison

   Jason - great points about the spreadsheet, why not use two or three columns for Selenium? Why don't we tackle this over skype with a browser in hand?

   Reply

5. Back to top

   Re: I updated the information in the spreadsheet on Selenium for you.

   by Jason Huggins

   Yes, Selenium needs to be broken into 2 columns. And yes, let's tackle this over Skype in the next few days.

   Reply

6. Back to top

   Re: Document of available tools

   by Richard Perfect

   I just saw your post and thought perhaps I could mention our brand new testing tool Trace Analyst that provides traceability and progress tracking information. It produces Burn-Down charts by comparing planned test cases versus passing test cases and integrates with Ant/Maven to produce reports and charts about project progress. You can find out more on our website www.traceanalyst.com

   Reply

7. Back to top

   Left out some of the larger solutions

   by Eric Schumacher

   I was disappointed to see that tools like HP QuickTest Pro were left out of the comparison.

   Reply

8. Back to top

   Re: Document of available tools

   by Mark Levison

   Richard your welcome to add your tool to the spreadsheet. If we need a new sheet because yours is a new category of tool let me know.

   Reply

9. Back to top

   Re: Left out some of the larger solutions

   by Mark Levison

   Eric - the list of tools is by now means complete. If you know HP QuickTest Pro or any other tool well enough to document it please add it to the spreadsheet. No one at the gathering knew those tools and so we couldn't document them.

   Reply

10. Back to top

    Re: Left out some of the larger solutions

    by Eric Schumacher

    Fair enough. Did you happen to invite anyone from HP the other larger vendors?

    Reply

11. Back to top

    Re: Left out some of the larger solutions

    by Mark Levison

    Eric - to be clear I wasn't one of the organizers just an attendee. The meeting is open to anyone. Its advertised on the "aa-ftt" and "agile testing" mailing list (see my list of lists for details: www.notesfromatooluser.com/2009/06/agile-mailin...).
    
    I don't personally know anyone from HP or any other major vendor (i.e. Borland Silk test) etc to invite. Please if you know people at a vendor please invite them to join the mailing lists.

    Reply

12. Back to top

    Wrong link

    by Daniel Anjos

    I know that it looks the same, but Fitnesse links to fitness.org (should be fitnesse.org).
    
    Anyway, now that Selenium is on the spreadsheet, it's easy to see that Selenium surpasses all other tools!
    
    Quoting Jason: "if you write tests in a real programming language, you can do anything and everything you want"

    Reply

13. Back to top

    Re: Wrong link

    by Ryan Slobojan

    I know that it looks the same, but Fitnesse links to fitness.org (should be fitnesse.org).

    
    This has been fixed - thanks for the heads-up!
    
    Ryan Slobojan
    Chief Editor, InfoQ

    Reply

• Older >