Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ


Choose your language

InfoQ Homepage News Reaction to Proposed Software Testing Standard

Reaction to Proposed Software Testing Standard

Leia em Português


The International Standards Organisation, in collaboration with other standards bodies, are preparing a new standard for software standard: ISO/IEC/IEEE 29119.  This standard is intended to standardize and codify many of the practices and techniques of testing on software projects. The response from the testing community has been largely negative due to the prescriptive nature of the standard, and the exclusion of the context-driven approaches.

The stated purpose of the standard is:

ISO/IEC/IEEE 29119 Software Testing is an internationally agreed set of standards for software testing that can be used within any software development life cycle or organisation. By implementing these standards, you will be adopting the only internationally-recognised and agreed standards for software testing, which will provide your organisation with a high-quality approach to testing that can be communicated throughout the world.

Stuart Reid, conviener of the ISO working group developing the standard, shared a webinar which explains the intent and content of the proposed standard.  

The new standard will replace existing standards which have been published by the different contributing bodies - 

The reaction from the testing community has largely been negative, even to the extent of petitions being drawn up rejecting the adoption of the standard. 

James Christie gave a talk at the CAST 2014 conference in which he rejected the premise on which the certification is based.  He wrote a blog post  titled "ISO 29119: Why it is Dangerous to the Software Testing Community" in which he expanded on the ideas from his talk:

I argued that ISO has not achieved consensus, or has even attempted to gain consensus, from the whole testing profession. Those who disagree with the need for ISO 29119 and its underlying approach have been ignored. The opponents have been defined as irrelevant. If ISO 29119 were expanding the market, and if it merely provided another alternative — a fresh option for testers, their employers and the buyers of testing services — then there could be little objection to it. However, it is being pushed as the responsible, professional way to test — it is an ISO standard, and therefore, by implication, the only responsible and professional way.

Under the heading "What is wrong with ISO 29119" he states

Well, it embodies a dated, flawed and discredited approach to testing. It requires a commitment to heavy, advanced documentation. In practice, this documentation effort is largely wasted and serves as a distraction from useful preparation for testing.

Such an approach blithely ignores developments in both testing and management thinking over the last couple of decades. ISO 29119 attempts to update a mid-20th century worldview by smothering it in a veneer of 21st century terminology. It pays lip service to iteration, context and Agile, but the beast beneath is unchanged.

The danger is that buyers and lawyers will insist on compliance as a contractual requirement. Companies that would otherwise have ignored the standard will feel compelled to comply in order to win business. If the contract requires compliance, then the whole development process could be shaped by a damaging testing standard. ISO 29119 could affect anyone involved in software development, and not just testers.

Huib Schoots posted a discussion of the ISO 29119 debate in which he summarises the points against the standard. 

James Bach, advocate and founder of the Context Driven school of testing responded to the proposed standard with a post in which he states:

A standard for testing would have to reflect the values and practices of the world community of testers. Yet, the concerns of the Context-Driven School of thought, which has been in development for at least 15 years have been ignored and our values shredded by this so-called standard and the process used to create it. They have done this by excluding us. There are two organizations explicitly devoted to Context-Driven values (AST and ISST) and our community holds several major conferences a year. Members of our community speak at all the major practitioners conferences, and our ideas are widely cited. Some of the most famous testers in the the world, including me, are Context-Driven testers. We exist, and together with the Agilists, we are the source of nearly every new idea in testing in the last decade.

He goes on to say that software testing is not ready for standards yet:

The reason they have excluded us is that they know we won’t agree to any simplistic standard based on templates or simple formulae. We know those things look pretty but they don’t help. If ISO doesn’t exclude us, they worry they will never finish. They know we will challenge their evidence, and even their ethics and basic competence. This is why I say the craft is not ready for standards. It will be years before all the recognized experts in testing can come together and agree on anything substantial.

Gil Zilberfield argues against the standard but feels that "Testers Are Losing The ISO 29119 Battle" because

An organization wants the certification because they need it, or believe it will help their quality. They are looking for the best, simplest and less risky way to get it. The ISO organization gives the simplest common ground that works with “documentation as proof” concept. Everybody’s happy.

Except the practitioners.

The debate is sure to rage on. 

Rate this Article


Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Community comments

  • Wow

    by Roopesh Shenoy,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    All we need is one more standard.

  • Praise standards

    by Bart Szulc,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    Not sure what bothers me more, premise complex problems can be solved by yet another standards, or an idea of craeting a complete standard for software testing.
    Wish ISO guys took different approach, talk to the industry, to context driven testers, to the sceptics, came up with a book of testing ideas, practices, methods, you could implement in your company if they fit your needs, as this book would also explain potential advantages and disadvantages of adopting these techniques - when it works and when it can be wasteful or even dangerous. Maybe even come up with a base that should be adopted no matter of context. But this would require them to be more doubtful. The standard would be weaker. The premise of assuring high level quality by implementing a standard would not have been met.
    PS: I hope these guys at ISO just playing us, like internet trolls, and do this so the testers community grows stronger, fighting against common foe, constructively criticizing all the attempts of including something in the standard... so maybe at the end we will have this book I wish we had

  • What's next?

    by Hermann Schmidt,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    What a pile of nonsense.

    May I propose to standardize users instead? That would make our desing work much easier. Only ISO-certified users are allowed to operate software.

  • Re: What's next?

    by Roopesh Shenoy,

    Your message is awaiting moderation. Thank you for participating in the discussion.


  • ISO standards increasingly irrelevant

    by Nils Weinander,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    I have a feeling that ISO standards are turning increasingly irrelevant, outdated and much ignored. The simple fact that you have to pay expensive fees in order to read the standards isn't helping.

  • Just say no

    by Philippe Kruchten,

    Your message is awaiting moderation. Thank you for participating in the discussion.

    Just contact your national body and recommend that they vote No when this is brought to balloting. ISO is one vote per nation. Regardless of the composition of the working group.
    Dr Reid does not vote, the UK delegate does.
    Shane, time to contact the Australian delegate, the Kiwi delegate. This is were smaller nations have a definite advantage.
    PS: I do not know much about testing, but this smells like a diversion (or perversion) of the standardization process to promote some specific interests.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p