Next-Generation Functional Testing

Posted by Geoffrey Wiseman on Oct 10, 2007

Sections

Process & Practices,

Architecture & Design,

Development

Topics

Unit Testing ,

Delivering Quality ,

Agile ,

Customers & Requirements

Tags

Agile Alliance ,

Testing ,

TDD

What should the next generation of functional testing tools offer? Should they serve as readable documentation of requirements? Should they come with an advanced test editor that supports test completion for user-interface elements as well as test code refactoring and analysis? Or an editor that supports multiple representations, like wireframes and state machines? Visualize the test results? Suggest and generate tests based on herustics?

The Agile Alliance is holding a workshop to envison the next-generation of functional testing tools, from October 11th to 12th, 2007 in Portland, Oregon.

The workshop background describes the context for this initiative by describing how far functional testing has come and how far it has yet to go:

The good news is that tool support for automated functional tests has grown significantly in recent years. There is a large variety of commercial and open source testing tools/frameworks available that support Agile development practices. The FIT framework was a significant boost to the state of the art of automated functional testing, both in terms of the syntax of the specification (tables), the detailed test execution feedback (cell by cell), and the development/execution environment (desktop tools rather than development or specialized tools).

However, we believe that it's time for another significant boost to the state of the art.

• We are lacking integrated development environments that facilitate things like: refactoring test elements, command completion, incremental syntax validation (based on the domain specific test language), keyboard navigation into the supporting framework code, debugging, etc.

• We need more expressive test specification languages, possibly integrating executable: text, tables, shapes, and colors together into a single test.

• We need specification languages that can describe user interaction in a readable and maintainable fashion.

• We need to be able to view/navigate the tests in multiple different ways in order to see how the pieces of the puzzle contribute to the bigger picture of the domain/feature: organize tests based on their domain context; search for tests based on user-defined keywords (cross cutting concerns).

• and things that we haven't even thought of.

This event is the first step of the Agile Alliance's Functional Testing Program, headed up by Jeannitta Andrea with Ron Jeffries and Elisabeth Hendrickson. Jeanitta's been writing about next-generation functional testing tools for some time now, most recently in Envisioning the Next-Generation of Functional Testing Tools

So what do you need and want from your functional testing tool? What might the participants say at the workshop? What tools are you using now, and where do they work, where do they break down?

We'll stay in touch with this program as it progresses, and keep you informed here at InfoQ. You can use the Customers and Requirements tag to see more on this topic.

1. Back to top

   Finally, someone is asking the right questions!

   by Deborah Hartmann

   I can't tell you how disappointed I was with FitNesse, after hearing all the hype. My complaints are not about the framework itself, which meets a real need. It's about the useability for business people, tech writers, even business analysts (where these are part of the team, usually proxying a "customer").
   
   A pile of stories does not adequately document an application. Particularly where the application already has funtioning user documentation: a team goes Agile, starts helping their customer write stories, and suddenly they're turning out working features/enhancements at a decent rate! Months pass, and the tech writer is having trouble keeping up... he/she is directed to FitNesse, but the fragmentation is getting worse and worse.
   
   Feature X used to work like [this] but now it works like [this + story A[modified by story B and imfluenced by stories C and D]]. It can be difficult to identify stories that are orthogonal to many features.
   
   In addition, since "the working code is the final documentation" we can't necessarily trust the stories to be the final story... there's a reason some teams rip up story cards when they are finished. Ron Jeffries talks about "the three C's" - card, conversation, confirmation. Parts of the requirements are carried in personal interactions, and only captured in the code. This is only natural, and no matter how we try to retrofit what we code into the stories (surely a form of "muda" or waste) we'll never be sure it's 100%.
   
   I believe part of the solution lies in leveraging the emerging role of the Information Architect and integrating their technical writing repositories into our toolsets. Tech writers are now modularising, structuring and re-using pieces of their documentation in standards-based relational repositories that track things like user role and language. Surely, if they have a way to track re-usable pieces of documentation and flag areas thet require review when something changes, we have an opportunity to collaborate to create what I've called "emergent documentation". Already, I believe Ixiasoft has hooks into JavaDoc... what other integration points make sense?
   
   Is it time to get tech writers in our planning sessions and standup meetings? They seem, to me, to represent a wealth of knowledge about useability and standards, as customer-side players on the team, knowledge that we ignore until afterwards to our own detriment.
   
   I'd be at the workshop, had I not a prior engagement. I wish I could be there, and I thank the participants in advance for taking the time to put their heads together on this, particularly Jennita and the other organizers. I look forward to hearing what comes of it!
   
   deb

   Reply

2. Back to top

   CubicTest

   by Eric Lewin

   May something like the CubicTest tool be a viable approach to define, visualize, organize and run tests for user interaction?

   Reply

3. Back to top

   Documentation

   by Geoffrey Wiseman

   I've wondered a few times if a good testing framework with good descriptions (e.g. RSpec or the upcoming Story Runner) could take screenshots and be used as documentation.

   Reply

4. Back to top

   Re: Finally, someone is asking the right questions!

   by Kurt Christensen

   At my current client, we include the tech writers as part of our sprint teams, and we have boilerplate "docs" tasks associated with each story. Finishing the user-facing documentation then becomes part of what it means for a story to be "done".
   
   The tech writers don't often come to the daily stand-up meetings, but they're involved as stories get signed off, and they ordinarily take part in the sprint planning and sprint review and retrospective meetings.

   Reply

5. Back to top

   Follow-Up

   by Geoffrey Wiseman

   www.testobsessed.com/2007/10/15/aa-ftt-the-agil...
   tech.groups.yahoo.com/group/aa-ftt

   Reply

6. Back to top

   More Follow-Up

   by Ben Simo

   In Praise of Abstraction
   www.developertesting.com/archives/month200710/2...
   
   Better Tools for Individuals through Collaboration
   www.questioningsoftware.com/2007/10/better-tool...
   
   Crowdware
   www.exampler.com/blog/2007/10/14/crowdware/
   
   AA-FTT - Agile Alliance Functional Test Tools Workshop
   www.testingreflections.com/node/view/6066

   Reply

• Older >