ISO 29119 the new international software testing standard - what about it?

“The aim of ISO/IEC 29119 Software Testing is to provide one definitive standard for software testing that defines vocabulary, processes, documentation, techniques and a process assessment model for software testing that can be used within any software development life cycle”.

I'm afraid my hackles rise when I see phrases like “one definitive standard” and “used within any software development life cycle”. It immediately triggers an adverse emotional reaction as I remember this rhyme from Lord of the Rings, about the One Ring that would give the holder power over all.

“One Ring to rule them all, One Ring to find them, One Ring to bring them all and in the darkness bind them In the Land of Mordor where the Shadows lie”.

More pertinently, I also think of this piece of ancient software development wisdom, from McCracken and Jackson in 1982. I'm afraid I don't know of a free source.

“Any form of life cycle is a project management structure imposed on system development. To contend that a life cycle scheme, even with variations, can be applied to all system development is either to fly in the face of reality or to assume a life cycle so rudimentary as to be vacuous”.

The people who are putting ISO 29119 together are experienced, well intentioned professionals, and I'm sure that they are well aware of many of the potential pitfalls that they face.

However, I'm far from convinced that the exercise will ultimately prove worthwhile. Is there a happy middle ground between an excessively prescriptive standard that will be an irrelevant nuisance in many situations, and a high level standard “so rudimentary as to be vacuous”?

Perhaps the framers of the standard will get the balance right. However, even if they do, I worry about what will happen when the standard is released. Even if the standard's creators envisage a set of useful guidelines, from which people can select according to their need, the reality is that other people will interpret them as mandatory statements of “best practice”.

Lawyers and large consultancies will see a business opportunity, and write contracts that require the production of all the documentation examples offered by the standard. Regulators and legislators will pounce on the standard as evidence of professionalism, with failure to comply being interpreted as deviation.

I believe that guidelines are extremely valuable, but allowing them to mutate into rigid standards is dangerous. I've written about this here, and I won't repeat my arguments in any detail.

I have three basic concerns about standards in software development.

1 – They inhibit the growth of novices and constrain experts.

2 – They are consistent with, and in practice encourage, a fundamental misunderstanding about the nature of software development, which is an iterative process of design and discovery, not a linear production process.

3 – Standards, by their very name, encourage non-experts to insist that they should be mandatory, without any understanding of whether they are relevant in a particular context. In my experience goal displacement poses a major threat. The focus of projects can become production of mandatory documentation and adherence to standards, rather than the real work.

The question, as posed in a tweet by Michael Bolton the other day, is “What can we do, then, to stop ISO 29119? A worldwide movement is called for”.

Forums like the Software Testing Club are great, but often we use them to preach to the converted. Testing conferences may reach a more diverse audience, but the real problem is neither the testers who want to introduce standards to our profession, nor the standards themselves. The problem lies with the way that they are applied. We therefore have to look to those outsiders who will turn the standard from guidelines to rules.

So what do we do?

I think we should be reaching out to these outsiders, networking, cultivating contacts and influencing those who can help us, our who might make our lives difficult if they don't understand what we are doing.

I used to work as a computer auditor, so I believe strongly that good auditors can be a great ally. They are often, wrongly, blamed for imposing mandatory standards. Managers often demand slavish adherence to standards, especially detailed documentation, “because the auditors insist”. Only bad auditors insist that software development standards should be applied without exception, in every case, regardless of context. Likewise, the US Sarbanes Oxley is often wrongly blamed for demanding excessive documentation. That's something I expand on in the last past of my article (referred to above) and also in this blog about the similarities between testing and auditing.

What do you think? Are you bothered? Are you happy enough to work with whatever standards are produced? If so, why?

If you are worried, what are you going to do?

This should really be a blog posting, but I want to start a discussion and a forum post works far better for that.

I hope that this is something that people will want to discuss. A new testing standard will affect the way that huge numbers of testers will have to work, whether they like it or not.

The timeline on the ISO site is vague, but it looks like the three final drafts will come out late this year or sometime next year, with the final versions being released in the second half of 2013.

There's no suggestion (as far as I know) of companies being certified as complying with the standard. The idea is that the standard is released as being "best practice". Big corporations will then adopt it and insist on suppliers complying. Suppliers in turn will be happy to claim they' will comply, not because it will help them build better systems, but because the standard will allow them to carry out work that is unnecessary, but lucrative.

The publication of a new standard bothers me. My current employer is unlikely to insist on its application, so it will not immediately affect me in that sense, but I believe it could be harmful to the profession as a whole.

My main gripe is that because this is being presented as an ISO standard, it means the recommendations it makes are likely (in my experience) to be 'forced' on testers, regardless of whether they are suitable for the task at hand.

I haven't read through it all yet (although on first glance I note it seems to be quite documentation-heavy) but I hope that management teams choose to scrutinise standards prior to insisting on them, in order to ensure what is being used is applicable and adds value to the project in question. Lengthy test plans and rigorously-documented test cases are no use unless someone is reading them.

I should add - I don't want to bash those defining the standard or the standard itself. I am sure the content and the suggested practices will have value in context. I think the issue is more related to how some portions of the industry view standards in general.

I wonder if this will feed back into the ISTQB/ISEB certification as well.

That's interesting. The drafts don't seem to be publicly available. What I've seen isn't encouraging. The only downloadable document is a presentation from 2010 that doesn't mention "variations" or "lighter". It refers to Agile and iteration only in passing when it mentions the need for trialling the standard in all project types.

I'll be very interested to see how they strike the balance between being overly prescriptive and losing relevance on the one hand, and being too high level to offer practical guidance on the other. And above all I'm keen to see whether they can define the standard, and make it tailorable, in such a way that it supports testers who are trying to adapt it to their particular project, and frustrates lawyers, senior management et al who want to use it to provide a suffocating blanket of documentation.

I see the standards like the filling in a sandwich - the bottom slice of bread are those that don't really know what they are doing, just starting, or looking for info, and hopefully use the standards to rise to an acceptable level of foundation, whereas the top slice are those who have experience and know how, and use what they require from the standards while applying their own methods where necessary. For those that will mandate the standards I think you will find will find something to mandate, so it may as well be the testing standard (better the devil you know than the devil you don't right?).