Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ


Choose your language

InfoQ Homepage News Reaction to Proposed Software Testing Standard

Reaction to Proposed Software Testing Standard

Leia em Português

The International Standards Organisation, in collaboration with other standards bodies, are preparing a new standard for software standard: ISO/IEC/IEEE 29119.  This standard is intended to standardize and codify many of the practices and techniques of testing on software projects. The response from the testing community has been largely negative due to the prescriptive nature of the standard, and the exclusion of the context-driven approaches.

The stated purpose of the standard is:

ISO/IEC/IEEE 29119 Software Testing is an internationally agreed set of standards for software testing that can be used within any software development life cycle or organisation. By implementing these standards, you will be adopting the only internationally-recognised and agreed standards for software testing, which will provide your organisation with a high-quality approach to testing that can be communicated throughout the world.

Stuart Reid, conviener of the ISO working group developing the standard, shared a webinar which explains the intent and content of the proposed standard.  

The new standard will replace existing standards which have been published by the different contributing bodies - 

The reaction from the testing community has largely been negative, even to the extent of petitions being drawn up rejecting the adoption of the standard. 

James Christie gave a talk at the CAST 2014 conference in which he rejected the premise on which the certification is based.  He wrote a blog post  titled "ISO 29119: Why it is Dangerous to the Software Testing Community" in which he expanded on the ideas from his talk:

I argued that ISO has not achieved consensus, or has even attempted to gain consensus, from the whole testing profession. Those who disagree with the need for ISO 29119 and its underlying approach have been ignored. The opponents have been defined as irrelevant. If ISO 29119 were expanding the market, and if it merely provided another alternative — a fresh option for testers, their employers and the buyers of testing services — then there could be little objection to it. However, it is being pushed as the responsible, professional way to test — it is an ISO standard, and therefore, by implication, the only responsible and professional way.

Under the heading "What is wrong with ISO 29119" he states

Well, it embodies a dated, flawed and discredited approach to testing. It requires a commitment to heavy, advanced documentation. In practice, this documentation effort is largely wasted and serves as a distraction from useful preparation for testing.

Such an approach blithely ignores developments in both testing and management thinking over the last couple of decades. ISO 29119 attempts to update a mid-20th century worldview by smothering it in a veneer of 21st century terminology. It pays lip service to iteration, context and Agile, but the beast beneath is unchanged.

The danger is that buyers and lawyers will insist on compliance as a contractual requirement. Companies that would otherwise have ignored the standard will feel compelled to comply in order to win business. If the contract requires compliance, then the whole development process could be shaped by a damaging testing standard. ISO 29119 could affect anyone involved in software development, and not just testers.

Huib Schoots posted a discussion of the ISO 29119 debate in which he summarises the points against the standard. 

James Bach, advocate and founder of the Context Driven school of testing responded to the proposed standard with a post in which he states:

A standard for testing would have to reflect the values and practices of the world community of testers. Yet, the concerns of the Context-Driven School of thought, which has been in development for at least 15 years have been ignored and our values shredded by this so-called standard and the process used to create it. They have done this by excluding us. There are two organizations explicitly devoted to Context-Driven values (AST and ISST) and our community holds several major conferences a year. Members of our community speak at all the major practitioners conferences, and our ideas are widely cited. Some of the most famous testers in the the world, including me, are Context-Driven testers. We exist, and together with the Agilists, we are the source of nearly every new idea in testing in the last decade.

He goes on to say that software testing is not ready for standards yet:

The reason they have excluded us is that they know we won’t agree to any simplistic standard based on templates or simple formulae. We know those things look pretty but they don’t help. If ISO doesn’t exclude us, they worry they will never finish. They know we will challenge their evidence, and even their ethics and basic competence. This is why I say the craft is not ready for standards. It will be years before all the recognized experts in testing can come together and agree on anything substantial.

Gil Zilberfield argues against the standard but feels that "Testers Are Losing The ISO 29119 Battle" because

An organization wants the certification because they need it, or believe it will help their quality. They are looking for the best, simplest and less risky way to get it. The ISO organization gives the simplest common ground that works with “documentation as proof” concept. Everybody’s happy.

Except the practitioners.

The debate is sure to rage on. 

Rate this Article