Software testers petition to stop ISO 29119

The proposed testing standards have met fierce opposition in the testing community, including a petition stop the ISO from moving forward.

As the ISO, IEC and IEEE prepare for their final vote on the fourth part of ISO/IEC/IEEE 29119, many prominent software testers are leading a petition to suspend the standard. Fueled by the #stop29119 hashtag on Twitter, these testing professionals are not only challenging the standards being produced, but the right of the standards bodies to produce them.

ISO 29119-4 is proposed to codify a set of software test design techniques. According to SoftwareTestingStandard.org, "The test design techniques that are presented in this standard can be used to derive test cases that can be used to collect evidence that the requirements of each system under test have been met and/or that defects are present." The standard comprises about 25 techniques which are split into three branches, "Specification-based techniques," "Structure-based techniques" and "Experience-based techniques."

ISO 29119-4 builds on the three parts of ISO 29119 that have already been published. ISO29119-1 is a glossary of concepts and definitions designed to provide a common language with which to discuss software testing. ISO 29119-2 defines a generic process model of the software development lifecycle. ISO ISO 29119-3 defines templates for software documentation that organizations can use to document their software testing.

ISO 29119

ISO 29119 is the most commonly used name for the new testing standards which are officially named ISO/IEC/IEEE 29119 and are a collaboration of three separate but cooperative standards bodies.

The group against ISO 29119 feels that any standardization of testing will harm the quality of software produced by organizations that adopt that standard. The group believes that reliance on a standard leads organizations to expect heavy documentation, which in turn drives testers to focus on the production of heavy documentation. This focus on documentation pulls testers away from the real job of software testing, according to Keith Klain, the CEO of Doran Jones.

"Can I show that I've met the standard of what a test case is meant to look like?" asked Klain, "Absolutely. Can I show you I ran thousands of them? Absolutely. Was that testing worthwhile? Was it good testing? Did we get any interesting information? Well, that's a much harder question to answer …" In Klain's view, standards don't actually help answer the good questions. He said standards actually cloud judgment, "because [complying with] a standard, particularly in knowledge work, gives the appearance of being good."

Meanwhile, the standards bodies claim that standards benefit all industries by promoting safety and interoperability. They believe there is more good to be gained from pursuing broad standards than from abandoning them.

The push for standardization has grown out of electrical engineering problems, some of which still haven't been solved today. For example, if someone lives in the U.S. and vacations in Europe, he will probably need to bring an adapter to plug in electronic devices such as a cell phone charger. This is because standards bodies in Europe and the U.S. came up with independent standards that are now too strongly ingrained to be changed. The strength of these national standards makes a unifying international standard impossible. This is a case ISO 29119 is meant to steer software testing around.

I don't think there is a need for a standard.
Iain McCowattPresident, International Society of Software Testing

Jon Hagar is the owner and lead consultant of Grand Software Testing and the author of Software Test Attacks to Break Mobile and Embedded Devices as well as the lead project editor from IEEE on ISO 29119. He pointed out that standards have to be living, evolving things. "In order to remain active," he said, "any ISO standard has to be reviewed and updated no less than once every five years. For ISO 29119, we're looking at reviewing this more like every two years, at least for the foreseeable future."

A bad standard, the petition signers argue, is worse than no standard at all. "I don't think there is a need for a standard," stated Iain McCowatt, one of the presidents of the International Society of Software Testing and the author of the petition to stop ISO 29119. "There is a need for testers who are dedicated to improving their skills and inventing new ways of testing." McCowatt believes the discipline of software testing is still very young and it's too early to employ standardization.

In Hagar's view, the standards are not meant to be a stopping point for the evolution of testing, but a starting point. He compared the evolution of standards to the scientific method. "This is our current history-based hypothesis," he said. "As we gather evidence about it -- what works fully, what works when tailored, what doesn't work at all -- we'll make better hypotheses in the future." If we don't set down and codify our predictions about the process, Hagar argues, we're not equipping ourselves to get better as a group.

Hagar added that the standards are not meant to define the cutting edge of the testing industry, but the "middle of the road." The standards are guidelines that should allow organizations that adopt them to understand generally what to expect from the testers and/or testing services they employ. The standards are also intended to aid in international contract negotiations where testing processes are concerned.

Hagar stressed that ISO 29119 is not intended to be a guide for individual testers to follow. "This is not the ISTQB," he said, "and it's not the SWEBOK." For testers looking to hone their trade, Hagar recommended becoming somewhat familiar with these standards, but more importantly reading textbooks and such that are tailored to the needs of individual testers.

Protests go beyond the details of the standard itself, and also delve into questions about process.

"What lots of people object to is this attempt by one faction to define itself as being the very embodiment of responsible, professional testing." Said James Christie, whose recent session about the new testing standard at CAST 2014 sparked the current #Stop29119 campaign. "The ISO working group [has] effectively defined those who disagree with them as being irrelevant at best and by implication, […] that those who don't comply are irresponsible and unprofessional."

Hagar sees the current objections as a good thing. "I think there are a lot of good points being made," he said, "and I look forward to putting that feedback to good use in the next revisions." He said he's presented at several conferences over the past few years and that up until now the various testing communities have been rather quiet. "I only wish I had gotten this great feedback two years ago," he said. If so, perhaps the current standards would be less objectionable to their detractors.

James Christie said opposition to the standards is nothing new. "I think it's important to stress that the opposition has been there for a long time, but it's only just recently gotten organized." He said that forming organizations to take action is important because individuals often go unheard or "can be dismissed as irrelevant, disaffected individuals."

Next Steps

Stuart Reid, a major contributor to ISO 29119 has made a few
explanatory videos available on YouTube.

Join the conversation

11 comments

Register

I agree to TechTarget’s Terms of Use, Privacy Policy, and the transfer of my information to the United States for processing to provide me with relevant information as described in our Privacy Policy.

Please check the box if you want to proceed.

I agree to my information being processed by TechTarget and its Partners to contact me via phone, email, or other means regarding information relevant to my professional interests. I may unsubscribe at any time.

Thanks Geert, for that thoughtful response. I would agree that testing, at its best, is an art form. But it also has very practical implications. So I'll play devil's advocate here. How does a business determine which tester's unique approach will prevent a disastrous bug from leaking out? Does the business need a Polluck, or a Picasso, or a Peeters? I think the standards and certifications are an attempt to bridge that knowledge gap. I'm certain there are flaws with the process, but I also know I don't know how to fix them. Businesses that rely on certified testers and standards compliance for software testing are probably making a trade. They trade a chance at getting a much better tester to cancel out a chance at getting a very bad tester. They are seeking the mediocre, really, out of a desire for safety.

I think that's why the standards bodies focus on "good practice" and not "best practice". The standards aren't there for the people who passionately follow their art, they're there for the people who need instructions to get it right. Whether this type of bet hedging seems like a good idea to either of us or not, there will be many risk-averse businesses who will be attracted to it. I hope for the sake of those businesses and their customers that the testing standards available improve year after year. But that's a story that remains to be written.

I don't know to which extend experienced testers are involved in defining the standards. I totally agree with the fact that they can, and hopefully will be, a guideline for companies that don't have a decent test history. It would be beneficial to give them at least some references on how to implement a certain level of comfort in finding serious blockers in their software.

To my view, those standards only can be of real value if they are at least thoroughly reviewed by experienced testers that have experience in different domains of software testing.

I once followed the ISTQB certification and my impression was that it promoted a too strict and outdated way of working. Of course, there were interesting test techniques, but it felt as outdated if you take into account the current Agile evolution these days.

I think that there is a need for a interactive platform where experienced testers can share their ideas and work together to build up a good knowledge base and guidelines for those entering the world of software testing.

It is a fairly young profession, although I'm in the business for 15 years by now. There is still a lot to build up, yet.

I can't speak for everyone involved in the standards, but I think most of them are pretty experienced and come from some diverse backgrounds. The two I was in contact with for this story both have careers in software engineering and testing that go back to the 1980's. I get the impression the standards bodies generally welcome more input with open arms. But because it's all volunteer there is obviously a limit to how many experienced testers are willing and able to donate their time and energy. I don't have a lot of insight about ISTQB, but from what I've heard your take on them seems pretty accurate. I do know, from the ISO folks I talked with, that ISTQB has no direct link to the ISO 29119 standards, but that the standards are likely one source of information for them (and potentially a really big one).I really like what you're saying about an interactive platform for experienced testers to share their ideas and work together. I think that's what the standards bodies are trying to do. I think the problem they are facing, and I don't know how to solve it, is that they depend on funding from selling the standards, but charging for them reduces their visibility severely. You can't help make them better if you don't know what they say and if you aren't willing to pay, it's hard to know what they say. That's one of those problems that I don't know how to solve.

I think standards have a place, but if the testing standards are like the majority of development/engineering standard they'll be largely ignored. In today's software business environment, most companies are not going to read a standard for software testing let alone follow it if it involves more testing dedication than they plan to provide. I do think as testing services vendors become more prevalent, a standard is good so businesses have something to refer to what they require. Although, I still doubt business leaders will read a testing standard let alone implement it.

Having lived through one standards obsession (ISO 9001 in the 1990's) and seen the very limited overall quality benefits they provided. Process compliance? That it was able to do. Improved quality? I'm very skeptical, and have seen little in the way of proof that it actually helped. Do I expect better results for ISO 29119? Again, color me skeptical.

I think it is a colossal waste of time, energy, paper, ink, and money. It does little to advance a standard that will be helpful to teams, and as far as I can tell it doesn't even improve on the IEEE standard that came before it. I know some people had high hopes for it, but I'm sorry to say, from what I've read, I've been rather disappointed. The more I study the art and science of testing, the more I realize that its like a performance, more than a modularizable componetized, process with guaranteed results.

It does seem like this standard would do more harm than good. Software testing experts Matt Heusser and Justin Rohrman had more to say about why they oppose the standard in this post (and the ANSI agrees): https://itknowledgeexchange.techtarget.com/uncharted-waters/iso-29119-what-you-should-know-about-software-testing-standards/

Most of the opposition to ISO 29119 seems to be coming from people who have not read the standard itself and thus have no way to know what it does and does not say. Some appear to be blindly following guidance from their own chosen arbiters of what’s good and bad in testing, i.e., folks who were not involved in developing the standard but similarly make money consulting and writing/speaking/training about their presumably-different ideas on testing. Awareness of and participation in standards working groups indeed are issues that have been around at least 40 years, as have folks who use standards in ways that interfere with achieving value. Both need to be address, but a more immediate concern is making standards available without the current prohibitive expense. Let’s fix that and then limit the 29119 petition-signing to those who’ve actually read the standard.

Yeah Robin, I think you sort of hit the nail on the head. It seems like making the standards available to be read only by folks who are willing to pay for the privilege is keeping the standards from getting the visibility they need to gain traction.