Intelligence about the people side of software testing & projects
by Fiona Charles

Search This Blog

Saturday, August 23, 2014

Why I oppose adoption of ISO 29119

I don’t oppose the idea of a testing standard, though I’d
like to see a programming standard to accompany it. But ISO 29119 and its
predecessors are not testing standards. They are fundamentally standards for documentation
of testing and things called “testing processes”. There is little that goes
into a testing process practiced by a skilled tester that a document about
documents can capture or codify.

In a long career I have yet to see any indication that
so-called “test” standards have done anything to improve the skill levels of
testers or the quality of their testing. Instead, I’ve seen many organizations
doing mediocre rote testing with testers who are forced to produce reams of
impenetrable, repetitive documents that nobody outside the company testing
circle reads. I repeatedly see test strategy documents showing not an ounce of
strategy yet compliant with standards such as this. Those same organizations
often insist that their testers obtain certification.

Whether or not this is the intent of the ISO 29119
proponents, it is how adoption will play out in real life in many organizations. As James Christie has pointed out, contract lawyers, internal auditors and managers who know nothing about testing
will insist on the grand panoply of fat documents because it’s a standard and
therefore must represent “best practice”. Nervous and unskilled test managers
will embrace templates based on ISO 29119 because all those documents make them
feel secure and important. People on Agile projects will struggle with the
conflicting demands of their projects and the standards.

I have yet to see evidence that compliance to any “testing”
standard equates to good testing.

Testing is a skilled activity. (James Bach calls it a
“performance art”.) The only true measurement of testing is skill exhibited in
live practice. Some proponents of ISO 29119 sneer at the “craftsman” (or
craftsperson) mentality espoused by many of us. I wrote in an earlier post that
I grew up thinking of craft as “skill fuelled by love and integrity”.
You who sneer at the idea of craft and make snide jokes about medieval guilds
should take a look at some highly-skilled professions in the modern world. Do
you think a surgeon never speaks of, nor works to grow, her craft? Is a person
licensed to perform surgery because of the fine strategies, plans and reports
he compiles in templates?

I don’t doubt that surgeons must plan
and devise strategies. They have procedures they must follow and forms they
must fill. But ultimately, surgeons are evaluated—and licensed by their
state-sanctioned governing bodies—based on their results and the skill they
exhibit on a real live human, tools of the trade in hand. They must also pass
exams on their knowledge of the human body and its pathologies, as well as a
range of tools and techniques. But the exams surgeons undergo are much more
rigorous than anything developed so far for a testing certification. And no-one
becomes a surgeon merely by passing exams. Like other craftspeople, surgeons
serve an apprenticeship: studying, practicing and exhibiting on the job the
skills they must have to qualify for their profession. As do lawyers.

I’m not pretending that software testers normally require
the same level of skill as surgeons, nor as extensive a education program. But I
do think that scaled down the analogy holds.

I would welcome a real testing standard, though I’d like to
see a programming standard to accompany it. A true testing standard would focus
on demonstrated skills assessed by qualified practitioners. It might set
boundaries for the levels of testing skill required to work alone or under
supervision, and the types of software testers at differing levels could work
on. Education to meet such a standard would combine classroom studies with
on-the-job practical training, and judging of live testing. At successful
conclusion of her education, a tester could be certified as a professional. Very
skilled testers could become master testers, in demand for very high-risk
software.

We aren’t nearly organized enough to devise a real testing
standard in the near future. But I don’t see ISO 29119 as an acceptable
substitute. It puts too much focus on the wrong things.