Problems with Process

Douglas C. Schmidt

Editor-in-chief

C++ Report

May 1996

My previous Editor's Corner focused on
dysfunctional traits I've observed while working on software projects
during the past decade. Many traits (such as "Death by Quality")
originate from outdated or ineffective development processes. A
development process consists of the tasks performed and the artifacts
produced when building software.

Effective development processes are essential for producing high
quality software, particularly for large-scale systems. However,
identifying and instituting good software processes is remarkably
elusive. In this Editor's Corner I describe some common problems I've
encountered with existing development processes and outline some
concrete steps that can help produce better software.

Common Problems with Software Development Processes

The following are some common process-related problems I've observed
over the years:

"Applying the wrong process for the product" -- I've
repeatedly seen organizations try to apply processes that are
inappropriate for the type of products they're building. For example,
I've worked on projects that used very structured, Waterfall model
development processes (such as MILSTD 2167A) on next-generation,
large-scale communication systems. The waterfall metaphor in this
process model implies an orderly flow of software development from
upstream activities like analysis and specification to downstreaming
activities like design, implementation, testing, and maintenance.

This model assumes that difficult requirements and constraints can be
identified and resolved in early phases of the project lifecycle.
This is a dubious assumption, unless the problem domain, platforms,
and tools are very well understood and stable. When applied to high
risk projects that use unfamiliar or untested technology, the
Waterfall process almost certainly fails. The main problem is that
developers can't foresee all the traps and pitfalls that lurk ahead.
As a result, bad design decisions made far upstream become very costly
to fix downstream.

"The zero-lines of code seduction" -- I've encountered
organizations that considered the best software development process to
be one where programmers don't write code. Often, these organizations
have been seduced by the quest for the "zero-lines of code" holy
grail. This seduction is peddled by pundits who claim that complex
software systems can be generated automatically by applying
transformations to high-level analysis rules.

The translational approach is well-suited for highly structured, well
understood domains (such as compiler parser construction or GUI
application builders). However, I've found that these techniques do
not scale up yet to more complex domains that possess sophisticated
error handling, concurrency, and distribution requirements. One
consequence of pursuing the "zero-lines of code" grail is that the
contributions of high-quality developers are often devalued until the
project goes awry. At this point, it is very expensive and time
consuming to actually write software that fixes the problems.

"Process bureaucracy" -- I've worked for a number of companies
where the effort associated with producing software was comparable to
ratifying international trade agreements. There were architecture
councils, mandatory TQM meetings, process review boards, and extensive
documentation on standard operating procedures. Unfortunately, there
was very little development taking place to justify all the process
bureaucracy.

The main problem with process bureaucracies is that they take on a
life of their own and become an end in themselves (i.e., acquiring
power), rather than being a means to an end (i.e., shipping product).
As a consequence, process bureaucracies often become the refuge of
technically inept, but politically savvy, individuals who don't
appreciate the importance of rewarding high quality technical staff.
Ultimately, this type of culture drives away the creative, highly
skilled technical talent that are so crucial to the long-term success
of projects and companies.

A Case Study in Frustration

I've worked on several projects over the years that exhibited all the
process-related problems described above. One distributed systems
project was particularly notable. On this project, programmers
weren't allowed to write any code until they'd fully documented all
the interfaces in a CASE tool. There were two motivations for this
approach:

Top-down development -- this development process forced
programmers to complete their designs before implementing
them.

Automatic code generation -- the goal was to use the
forward-engineering capability of the CASE tool to
automatically generate most of the application code
from graphical object and class models.

Although these motivations are reasonable in theory, in practice there
were several problems:

Lack of experience -- most of the developers had never
written distributed applications before. Therefore, they
lacked the experience and intuition necessary to produce
good designs from the top down. In particular, they didn't
realize that certain implementation details can invalidate
interfaces defined in the design phase. For instance,
network latency, error handling, and the semantics of the
I/O and concurrency mechanisms of the operating system
often make the use of purely synchronous interfaces
infeasible.

Inadequate tool support -- automatic generation of software
from graphical descriptions can be a powerful tool, but not
when it's only partially implemented. Unfortunately, the
CASE tool selected for this project didn't support
reverse-engineering adequately. Consequently, it was very
awkward and time consuming to modify the generated
interface classes in response to new requirements and new
architectural insights.

This project was a poster child for the problems that arise when a
development process runs amok. Not only did they choose the wrong
process and tools, but they had an entrenched process bureaucracy that
ignored developer feedback until the project derailed completely. By
that point, the original top-down designs and implementations were
essentially useless and had to be completely reworked. Because the
development process didn't allow rapid prototyping, the development
team wasted person-years of effort that could have been spent building
prototypes to reduce risk. These problems weren't recognized until
well into the project, however, when they were quite expensive to fix.

Towards a Product-Oriented Process

So what can we do to alleviate the problems with existing processes?
In my experience, successful organizations employ product-oriented
processes that help, rather than hinder, software productivity. Some
specific techniques I can suggest for instituting product-oriented
processes include the following:

Develop Complex Systems Iteratively Rather Than Sequentially
Iterative software processes are motivated by the pragmatic
observation that developers rarely have enough information or time to
analyze and model all aspects of complex systems. For instance,
requirements are rarely nailed down completely in early phases of
complex systems. Furthermore, crafting stable architectures,
interfaces, and implementations remains an art rather than a
(predictable) science. Therefore, it's unrealistic to expect to
define an architecture flexible enough to handle all subsequent
changes by using a sequential Waterfall process model.

As a result, it is essential to develop complex systems using a
systematic approach to iterative development (such as the Spiral
model). Iterative process models are designed to reduce unpleasant
surprises at the tail end of a project by identifying key sources of
risk during multiple iteration cycles. Techniques like prototyping or
simulation can be applied at each cycle to gain insight into open
issues rapidly and reduce development risk.

Support for iterative development and prototyping have been key
foundations of good software processes for decades. Back in the 70's,
Fred Brooks prophesied the inevitability of software iteration with
his second law of software: "Plan to throw the first one away, you
will anyway" [1]. This insight is as true today as it was in 1975.
Remarkably, while many software organizations give lip service to
iterative development, I've found that the less successful ones often
try to "swing for the fences" and don't devote adequate resources to
prototyping and risk reduction.

Focus on qualitative reviews rather than quantitative reviews --
Software is inherently abstract, which makes it hard to evaluate its
quality and accurately predict its delivery. Managing a software
project effectively requires the ability to cope with the fluid and
imprecise nature of software design, rather than trying to force a
sequential process driven by PERT charts. Therefore, one of the most
important methods for tracking project progress is periodic design and
code reviews. Although reviews can be time consuming, I've found that
they pay for themselves by exposing junior developers to the good
practices of local experts, increasing communication among developers,
sharing useful patterns and idioms with other team members, and
generally reducing risk.

I've also found that good development processes emphasize the
qualitative aspects of software review more than the quantitative
ones. Qualitative reviews focus on design and code inspections by a
group of peers, whereas quantitative reviews use automated complexity
metrics and tools that check for conformance to coding standards.
Qualitative reviews are more beneficial than quantitative reviews
because they are more effective at identifying and correcting
strategic problems in the architecture and implementation before they
become firmly entrenched in the software.

I believe that quantitative reviews are appealing largely because they
seem to absolve us from having to think carefully about the tough
issues. One of the main causes of project failures is that developers
don't take time to think through their designs and implementations.
All the CASE tools, translational engines, and visual programming
environments in the world can't make up for problems that could have
been foreseen with more disciplined thinking, analysis, prototyping,
and peer review.

Invest in continuous education and training --
Developing quality software requires a great deal of effort over a
long period of time. Ultimately, projects succeed because developers
gain sufficient expertise and commitment to achieve common goals.
However, team members must have the necessary technical skills to
succeed. Just as automated tools are no substitute for peer review, a
development process is no substitute for programmer competence.

A key challenge faced by software organizations is keeping the skill
sets of their developers from becoming obsolete. Technology is
changing more rapidly than ever, and it's hard to keep abreast of all
the new tools, platforms, languages, and techniques. Fortunately,
much of this "new technology" largely repackages the concepts,
features, and mechanisms that have been part of the software culture
for decades. Explicitly recognizing these recurring patterns in our
new tools and technologies makes it easier to retrain programmers by
building upon knowledge they've already mastered in other development
paradigms and platforms.

The study of patterns should be an essential part of any developer
education and training program. The study of design patterns helps
guide the choices of developers who are building, maintaining, or
enhancing software. By understanding the potential traps and pitfalls
in their domain, developers can select suitable architectures,
protocols, and platform features without wasting time and effort
implementing inefficient or error-prone solutions. Likewise, the
study of organizational patterns is essential to help guide the
choices of projects that are considering different development
processes. A wealth of organizational patterns have been documented
by our very own Jim Coplien and others in the Pattern Languages of
Programming series, which is described in more detail at the WWW URL:

Use tools to automatically generate documentation from the software --
Accurate documentation is clearly important to maintain and enhance
large software systems. Tools that generate class and object diagrams
from source code are invaluable for documenting the structure and
evolution of complex software systems. Of course, no tool can make up
for inadequate documentation...

I've generally found that reverse-engineering tools are more useful
than forward-engineering tools. In particular, most
forward-engineering CASE tools are inflexible and don't fully
implement reverse-engineering. Without adequate support for
reverse-engineering, documents and object models produced by these
tools become obsolete and unmaintainable. Moreover, expert developers
can often build quality software more effectively by writing class
interfaces and then using reverse-engineering tools to generate the
documentation from the software.

Another effective way to produce and maintain well-documented software
is to generate manual pages automatically from source code. Like the
reverse-engineering tools, this approach helps ensure that the
documentation doesn't drift out of sync with the software. Many
programming environments support this, e.g., all the online
documentation for Java is generated by the javadoc utility in the Sun
JDK. A freely available set of software tools that generate nroff,
html, and mif format manual pages from C++ header files are contained
within the ACE C++ network programming toolkit at WWW URL:

Wrapping Up

My goal in discussing topics like process and organization structure
in the C++ Report is to raise awareness of the need for better
software development practices. The longer I work with software, the
more I realize that many of our toughest challenges are rooted in
process rather than technology. To be effective, however, solutions
to these challenges must harmonize with contemporary technology
practice and respect the contribution of technologists.

I'm concerned that our industry is increasingly being driven by the
false prophets of Process and Methodology. This leads to
simple-minded solutions to complex software development processes,
which can cause far more problems than they solve. To counteract this
trend, developers must become more proactive in shaping and improving
the software processes they perform. However, this topic is becoming
too far removed from C++. Therefore, next month's Editor's Corner
will focus on tips for building C++ frameworks that are portable
across different OS platforms and compilers.

I'd like to thank Jack Reeves, Tim Harrison, Prashant Jain, and Irfan
Pyarali for their comments on this editorial.