Nine Steps to Delivering Defect-Free Software

Hello. I am Terry Colligan, president of Tenberry Software, Inc.
I have been a software developer for over 30 years,
and have been managing software development for over
20 years.
Tenberry Software (formerly Rational Systems) has a
reputation for producing high-quality software and
for having extremely good engineers.

Although I thought I understood the importance of
quality, and took pride in the quality of the software
we produced, I never believed that delivering defect-free
software was possible.
After all, everyone knows that all software
has lots of bugs, right?

Well, no, not necessarily!
Certainly, most experiences with today's
software quality are not encouraging.
Although few people can name even one piece of software which they use
that has no bugs,
defect-free software is
possible to create.
We know it is possible, because we're doing it.

It started with a single engineer.
This engineer was consistently producing work with a defect rate
more than one hundred times smaller than our other
engineers. She has done so for us for over three years
now.
During the same time, she has produced three to five times
as much code as any other engineer.

I found this so exciting that I determined to find
out how she did it, and to see if we could teach our
other engineers to achieve the same quality results.

I later discovered that one of my best friends, an
independent consultant in the mainframe/Cobol world,
has been similarly producing defect-free results on
his projects.

We have developed a process to produce guaranteed
defect-free software. (We are continuing to refine
our process, but it works now.)
To help improve general software quality, we are sharing
the nine steps of our process:

Surprisingly, the first reaction that I get when I
describe Defect-Free Software is to be told that it's just not
possible.
Defect-Free Software seems to be self-contradictory.
Some folks even act as if "Defect-Free Software" is
an attempt at computer humor.

In fact, this attitude is the biggest obstacle preventing
the delivery of defect-free software!
The most striking difference between the two defect-free
engineers and our other engineers (including me!) is their
attitude towards software defects.

The average engineer acts as though defects are inevitable. Sure,
they try to write good code, but when a defect is found, it's
not a surprise. No big deal, just add it to the list of bugs
to fix. Bugs in other people's code are no surprise either.
Because typical engineers view bugs as normal,
they aren't focused on preventing them.

The defect-free engineers, on the other hand, expect their code
to have no defects. When a (rare) bug is found, they are
very embarrassed and horrified. When they encounter bugs
in other people's code, they are disgusted.
Because the defect-free engineers view a bug as a public
disgrace, they are very motived to do whatever it takes
to prevent all bugs.

In short, the defect-free engineers, who believe defect-free
software is possible, have vastly lower defect rates than
the typical engineer, who believes bugs are a natural
part of programming. The defect-free engineers have a
markedly higher productivity.

In the past, I was the single biggest obstacle to
producing defect-free code at Tenberry. Because I didn't
really believe that defect-free code was possible, I made
decisions primarily focused on short schedule times.

In retrospect, virtually every decision against trying
for defect-free and in favor of short schedule time
was wrong and resulted in longer schedules, more bugs, more support,
higher costs and smaller profits!

Making a firm commitment to defect-free code
and holding to that commitment, in spite of schedule
and other pressures, is absolutely necessary to producing
defect-free code.

As a nice side benefit, you will see improved schedules
and reduced costs!

After attitude and commitment, program design and
structure have the biggest impact on defect-free code.
A clean, well structured design simplifies producing
reliable code. A poor design cripples the engineer,
and will make it impossible to achieve defect-free code.

Each function should be precise -- it should have only one purpose.
Each action or activity should
be implemented in exactly one place. When programs are structured
this way, the engineer can easily find the right place to make
a change. In the unlikely event that a bug is discovered in testing,
the engineer can go directly to the code with the defect and
promptly correct it. This saves time and is the major
cause of the faster schedules experienced with Defect-Free Software.

In addition to designing for clarity, it's important to keep the
defect-free goal in mind.
You want to choose designs that will be least likely to
have bugs. In other words, avoid tricky code.
Don't start to optimize code unless you are sure there
is a performance problem.

To me, one of the most surprising techniques used by our defect-free
engineer was the deliberate tracing in a debugger of each line of new
code when it is written.

As each line of code is about to be executed, you should
try to predict what the effect will be -- what data will be changed,
which path a conditional will follow, etc.
If you can't predict what the effect will be, then you don't
understand the program you are working on -- a very dangerous
situation.
If you don't predict correctly, you have probably discovered
a problem that should be addressed.

Tracing all new code shows:

Code that hasn't been tested. By stepping through each line of
code, you ensure that the new code is fully tested.

Inefficient code. For example, calling the same function
multiple times.
The first
time engineers watch a trace of their code,
they are often surprised to see exactly how much
redundant and inefficient code is executed.

Code that is working for the wrong reason.

Confirmation that the design is functioning as intended.

Tracing all new code will ensure that your code will be tested
and is functioning as designed -- both important
characteristics of defect-free code.

Peer code reviews have consistently been shown to be the
single most cost-effective way of removing bugs from code.
The process of explaining a new section of code to another
engineer and persuading that second engineer the code
is defect-free has several positive impacts:

Exposes the design and implementation,
with benefits similar to tracing the code.

Forces the engineer to articulate assumptions. About ten percent
of our code reviews are stopped in progress
as the authoring engineer suddenly
says, "Oops! Never mind!" because he suddenly realized that he had
made an invalid
assumption. (The review later resumes with the revised code.)

Allows more than one engineer to look at the code while it
can still be easily changed. Code will be made simpler and
easier to understand.

Encourages cross-training and sharing of techniques.
By discussing design strategies and implementation techniques,
each engineer learns from the experience of their peers.

Peer code reviews seem to work best.
Code reviews done by managers or senior technical staff
can have some of the same benefits, but sometimes are less
effective due to the interpersonal dynamics.

Obviously, to build defect-free code, you have to be able
to test your code. In addition to including a testing
plan/strategy into the implementation, you should
design specific code to provide for full, automated testability.

The most effective testing we use is
fully automated or regression testing.
This is a series of fully automated tests that are run after
each build of a program. The tests are designed to exercise every
part of the program, and produce a success/failure report.
The idea is to use the power of the computer to make sure that
the program hasn't been adversely affected by a change.

If the design is well structured, most changes should not have
side effects. The purpose of these automated tests is to
provide insurance that the coding assumptions are valid, and that
everything else still works.
By making the tests completely automated, they can be run frequently
and provide prompt feedback to the engineer.

If tests are run by manually testing the program, we have the chance
of human error missing a problem. Manual testing is also very
expensive, usually too expensive to run after every change to a program.

There are a number of commercial testing tools available
which are designed to help you automate your testing, particularly
in GUI environments such as Windows.
Although they are no doubt better than manual testing, we have
not found them to be effective, for a number of reasons.
(For more details, check out our
automated testing strategy.)

By building support for automated testing into your program, you can approach
100% automated testing. Without this customized, built-in testability,
you will be lucky to achieve 35% automated testing, even with the
best commercial QA testing tool.
We recommend that you budget five percent of total engineering
time to creating support for automated QA testing.

Of course, each new piece of code should have a corresponding
set of tests, added at the same time
as the code is added, for the automated QA suite.

In order for fully automated execution of testing to be of
value, the tests that are automatically executed and checked
must cover the software fully. To the extent that they
don't, running the tests doesn't tell you anything about
the part of your software that wasn't exercised by the
testing. (This is true for all testing, whether automated
or manual.)

Once you have a fully automated test suite, you should run
it after every build. This gives developers feedback about
the changes they are making, and it gives management clear,
objective feedback about the project status.

Clear, objective feedback about project status help managers
make better estimates and plans.
This feedback can help you identify and address problems while
you still have time to do something about them. In addition,
this clear, objective feedback puts managers in a better position
to provide correct feedback to their managers (or shareholders).
Finally, this objective feedback helps managers decide when
a project can be shipped or deployed.

The more prompt the feedback to the programmers, the more
useful it is. The shorter the time between the creation of a
defect and its discovery, the easier it is for the programmer
to understand just what they have done wrong. Prompt feedback
of failing tests can work as a kind of positive reinforcement
for development techniques that work and negative reinforcement
for techniques that don't.

By automating the build process as well, you can schedule
builds of your system daily. By building daily, you will
maximize the feedback to both your programmers and your
management.

There are a lot of existing tools that can be used to find
errors in your code in an automatic or semiautomatic manner.
Your programmers should be using these tools wherever possible.

These tools should be used in addition to the clean design,
rather than instead of. No matter how much you use automated checking
tools, using these tools alone will never
turn poorly designed, buggy code into defect-free code.
You can however, find a lot of bugs that would otherwise take much
more time
and effort to find and fix.

Useful automated checking tools include:

Setting your compiler's warning level to maximum.

Using the memory trace subsystem, such as Microsoft's MFC/DBGWIN
combination, that may be provided with your compiler.

Lint. This is a program which does extra fussy checking of C code.
Some people find it very helpful. There is a PC version and
a C++ version available from Gimpel Software.

These kinds of tools are important, particularly for catching the
kinds of errors that don't have obvious symptoms, such as memory
leaks.

Summary

That's the overview of how we create defect-free software.
Obviously, there is a lot of work involved.
There are also lots of details that will need to be adapted
to your specific situation.

Applying these defect-free methods to an existing program will be
worthwhile as well. Although it's harder to achieve a totally
defect-free result with existing code (usually due to the design),
applying these steps will result in a significant reduction in
an existing program's defect rates.

You can deliver defect-free software -- all you have to do is demand it.
By following these steps and working constantly towards the defect-free
goal, you will see more and more of your software become defect-free.

Tenberry helps companies deliver defect-free software in two ways:
We produce defect-free software under contract, and we consult with
companies to help them produce their own defect-free software
processes.