Next-Generation Functional Testing
What should the next generation of functional testing tools offer? Should they serve as readable documentation of requirements? Should they come with an advanced test editor that supports test completion for user-interface elements as well as test code refactoring and analysis? Or an editor that supports multiple representations, like wireframes and state machines? Visualize the test results? Suggest and generate tests based on herustics?
The workshop background describes the context for this initiative by describing how far functional testing has come and how far it has yet to go:
The good news is that tool support for automated functional tests has grown significantly in recent years. There is a large variety of commercial and open source testing tools/frameworks available that support Agile development practices. The FIT framework was a significant boost to the state of the art of automated functional testing, both in terms of the syntax of the specification (tables), the detailed test execution feedback (cell by cell), and the development/execution environment (desktop tools rather than development or specialized tools).
However, we believe that it's time for another significant boost to the state of the art.
We are lacking integrated development environments that facilitate things like: refactoring test elements, command completion, incremental syntax validation (based on the domain specific test language), keyboard navigation into the supporting framework code, debugging, etc.
We need more expressive test specification languages, possibly integrating executable: text, tables, shapes, and colors together into a single test.
We need specification languages that can describe user interaction in a readable and maintainable fashion.
We need to be able to view/navigate the tests in multiple different ways in order to see how the pieces of the puzzle contribute to the bigger picture of the domain/feature: organize tests based on their domain context; search for tests based on user-defined keywords (cross cutting concerns).
and things that we haven't even thought of.
This event is the first step of the Agile Alliance's Functional Testing Program, headed up by Jeannitta Andrea with Ron Jeffries and Elisabeth Hendrickson. Jeanitta's been writing about next-generation functional testing tools for some time now, most recently in Envisioning the Next-Generation of Functional Testing Tools
So what do you need and want from your functional testing tool? What might the participants say at the workshop? What tools are you using now, and where do they work, where do they break down?
Finally, someone is asking the right questions!
A pile of stories does not adequately document an application. Particularly where the application already has funtioning user documentation: a team goes Agile, starts helping their customer write stories, and suddenly they're turning out working features/enhancements at a decent rate! Months pass, and the tech writer is having trouble keeping up... he/she is directed to FitNesse, but the fragmentation is getting worse and worse.
Feature X used to work like [this] but now it works like [this + story A[modified by story B and imfluenced by stories C and D]]. It can be difficult to identify stories that are orthogonal to many features.
In addition, since "the working code is the final documentation" we can't necessarily trust the stories to be the final story... there's a reason some teams rip up story cards when they are finished. Ron Jeffries talks about "the three C's" - card, conversation, confirmation. Parts of the requirements are carried in personal interactions, and only captured in the code. This is only natural, and no matter how we try to retrofit what we code into the stories (surely a form of "muda" or waste) we'll never be sure it's 100%.
I believe part of the solution lies in leveraging the emerging role of the Information Architect and integrating their technical writing repositories into our toolsets. Tech writers are now modularising, structuring and re-using pieces of their documentation in standards-based relational repositories that track things like user role and language. Surely, if they have a way to track re-usable pieces of documentation and flag areas thet require review when something changes, we have an opportunity to collaborate to create what I've called "emergent documentation". Already, I believe Ixiasoft has hooks into JavaDoc... what other integration points make sense?
Is it time to get tech writers in our planning sessions and standup meetings? They seem, to me, to represent a wealth of knowledge about useability and standards, as customer-side players on the team, knowledge that we ignore until afterwards to our own detriment.
I'd be at the workshop, had I not a prior engagement. I wish I could be there, and I thank the participants in advance for taking the time to put their heads together on this, particularly Jennita and the other organizers. I look forward to hearing what comes of it!
Re: Finally, someone is asking the right questions!
The tech writers don't often come to the daily stand-up meetings, but they're involved as stories get signed off, and they ordinarily take part in the sprint planning and sprint review and retrospective meetings.
Better Tools for Individuals through Collaboration
AA-FTT - Agile Alliance Functional Test Tools Workshop
Shane Hastie on Distributed Agile Teams, Product Ownership and the Agile Manifesto Translation Program
Shane Hastie Apr 17, 2015