Book Review: Experiences of Test Automation
“Experiences of Test Automation” by Dorothy Graham and Mark Fewster is a follow-up to the reference book in test automation from the same authors:
The technology of test automation has moved on significantly since our last book. We wanted to find out what approaches have been successful, what types of applications are now being tested and how test automation has changed.
The new book is a compilation of (mostly successful) experiences in the field from a multitude of organizations and technologies. An intentional diversity, as explained by the authors to InfoQ:
We set out to find case studies spanning a wide range of business types, technologies and environments. The only consistency we wanted was a focus on system level test execution automation (although not exclusively).
The authors selected, compiled and reviewed all the 28 case studies from a mix of invited and self-proposed authors, having boiled down the selection criteria to:
- Was the story a real story (what actually happened rather than a personal opinion)
- Did it contribute something unique to the book (i.e. was different from existing stories)
- Was it interesting to read (i.e. a good story)
- Did it contain useful insights (that would help other people)
A table categorizing the characteristics of the case studies included in the preface allows tracking a story by application domain, project lifecycle or ROI measurement, for instance. Combined with a short summary of each chapter’s content it makes it easy for readers to navigate the book and search for relevant stories. For instance someone interested in test automation in agile development for governmental organizations can quickly identify chapter 7 and check out if the chapter summary sounds appealing.
The sheer diversity of experiences (ranging from database to SAP or Android testing) is clearly the book's strongest point by serving as a reference to anybody interested in the latest on-the-job approaches to test automation at the system level.
But this diversity comes at a price. Each case study has a limited length and the reader might be left craving for more insight on practical aspects of the approach described. Although the test architecture is usually well covered, concrete code and test data examples are rarely seen.
Another inevitable drawback of compiling stories from multiple authors is the diversity of writing styles. Not all experiences read equally well. Some are compelling stories written in a narrative fashion while others are closer to commented fact sheets.
However, this lack of internal consistency across chapters is partially compensated by the authors’ reflections which do a good and insightful job of extracting common issues across multiple chapters. Divided into Management and Technical sections, they cover several critical aspects to consider when undertaking a test automation initiative.
On the technical side, the authors highlight the importance of addressing test automation code with the same level of discipline and good practices as production code. Investing in a layered architecture that supports reusability and maintainability (by separation of concerns/responsibilities – test cases vs. testware vs. test execution tooling) is also pointed out as a key factor for successful test automation efforts:
We certainly agree that the testware architecture is critical and that many organizations neglect this topic. Assigning specific responsibility for testware architecture is one good solution (possibly creating a testware architect role).
The authors also shared with InfoQ their take on how to avoid technical debt creeping into automation code in the long run:
Good testware architecture provides a good foundation that helps avoid much technical debt in testware but this is something that needs to be frequently reviewed. People should not be afraid to change the automation because what they start with will never satisfy all future needs. However, the change needs to be managed. For example, Lisa Crispin’s chapter describes their use of a sprint every 6 months devoted to automation refactoring.
On the management side, establishing clear and realistic objectives stands out as a common requirement for success in test automation. Measuring ROI helps provide visibility to those objectives. However, most of the case studies calculating ROI don’t go beyond contrasting the automation effort against effort saved in repeated manual executions. In particular organizational aspects are generally overlooked. Automating extensive system tests might not pay off in the same manner in an organization with extensive unit testing and static analysis (or other types of software verification) as it would in an organization relying only on traditional end-of-production-line QA.
Although a few stories talk about the impact of agile development and closer collaboration with developers and other stakeholders, the topic of methodology and process adoption is mostly left out of scope. It might just be too early to gather experiences on the impact of agile testing or specification by example on increased system testing automation. And what are the obstacles to get there.
Hardly an end-to-end read, “Experiences of Test Automation” is a useful reference for experienced readers by providing examples of approaches, obstacles and solutions in a variety of domains and technologies as well as insightful overviews from the authors. However, newcomers to system level test automation are better advised to pick up the original “Software Test Automation” book for an initial grasp of the topic.
About the Book Authors
Dorothy Graham is a world-renowned consultant, speaker, and author with nearly forty years of experience in software testing. After nineteen years with Grove Consultants, she now concentrates on conferences and writing. She was Programme Chair for the 1993 and 2009 EuroSTAR conferences and holds the European Excellence Award in Software Testing. Her Addison-Wesley books include Software Inspection (coauthored with Tom Gilb in 1993), Software Test Automation (coauthored with Mark Fewster in 1999), and Experiences of Test Automation (coauthored with Mark Fewster in 2012).
Mark Fewster has thirty years of software testing and automation experience. As developer and manager for a multi-platform graphical application, he designed an architecture for long-lasting test automation. With Grove Consultants since 1993, he provides training and consultancy in all aspects of software testing. His Addison-Wesley books include Software Test Automation (coauthored with Dorothy Graham in 1999) and Experiences of Test Automation (coauthored with Dorothy Graham in 2012).
This interview is based on the book, ‘Experiences of Test Automation: Case Studies of Software Test Automation’ by Dorothy Graham and Mark Fewster, published by Pearson/Addison-Wesley Professional, Jan. 2012, ISBN 0321754069. For further info please visit the publisher site.
Uwe Zdun, Rafael Capilla, Huy Tran, Olaf Zimmermann Mar 09, 2014
Olav Maassen, Liz Keogh & Chris Matts Mar 08, 2014