Software Engineering for Everyone

Learning to program is easier than ever before with excellent,
freely available tools like Python and a community of contributors
willing to help newcomers. However, most employers want more than
just programmers, they want software engineers. It takes consistent
discipline to deliver complex systems into an imperfect world.
Whether for commercial air travel, distributing electrical power,
operating the postal service, visiting a comet, or giving depositors
reliable access to their own accounts via automated teller machines
anywhere in the world, we rely on the fruits of engineering discipline
daily. Even when developing simple systems, you'll encounter some of
the same difficulties an engineer confronts in developing large-scale
software.

The elements that help software engineers win at their game can
help you. You can develop engineering discipline as you go about
learning the craft of programming by keeping in mind three elements of
engineering discipline: predictability, collegiality, and
accountability. Working closely with engineers in my early days as a
programmer taught me the importance of these elements. Together, they
form the basis of what I call Software Engineering for Everyone.

Predictability: make it so, keep it so

The key principle of predictability is divide-and-conquer. To
divide, create interface agreements. With interface agreements in
hand you can break down implementations into modules, making parallel
and independent effort possible. This is the most difficult aspect of
engineering discipline that you will ever encounter. To succeed you
must first define and organize programs and the work to produce them
without having already done it.

In 1975, I provided system architecture for a nationwide on-line
processing system. The programming team was short-handed. They gave
me the opportunity to develop real-time display interface software for
minicomputers to be installed in 100 business offices across the
country. It was a critical element. We were outgrowing an initial
release and urgently needed a new system. We rewrote all of the
software, increasing performance and function all at once.

Applying the key principle of predictability, I redefined the
application interfaces for interacting with display terminals first.
I turned one ill-defined interface with a complex set of parameters
into a set of simple interfaces. I introduced one interface per
operation. I gave each operation a single well-defined function
clearly defining all behavior, operation-by-operation. It was like
handcrafting an object-oriented implementation. Designing in an
informal dialect of Algol, with the implementation in assembly
language, our adherence to an object model came entirely out of
disciplined use of the available lower-level tools.

I was nervous about introducing this change in place of something
already familiar. However, the simplicity of debugging and verifying
correct operation with cleaner, separate interfaces was too valuable
to pass up. It also made the inspection and verification of my
real-time implementation much easier. Because I could build the new
interfaces atop the old implementation using a simple shim layer, I
had a working prototype immediately. We began testing the prototype
on a system that was already known to be reliable.

Since I had stable software to work with on the other side of my
interfaces I could easily avoid finger pointing, and I could get at
defects of my new code when it didn't behave the way the shim already
did. This became critical as my software became later and later,
running far behind the original plan. I had an incremental, unit-level
test plan that I was determined to follow rather than do what my boss
wanted which was to turn on the whole thing at once and see what
happened. Although that was temporarily career-limiting, I have never
regretted it.

With a stable interface and a working shim, we quickly caught,
repaired, or worked around defects in the prototype. In the end,
integration and confirmation testing went off without a hitch. Only
one bug was uncovered and that was in a new administrative interface
having no counterpart in the old system.

I would love to offer this as a complete divide-and-conquer success
story, but there is more, an unexpected lesson: when hardware
developers start programming, it is easy to stop thinking like an
engineer. The new software was so fast we uncovered timing problems
in the firmware of the terminal hardware. Because the new interfaces
completely hid the hardware, I was able to add special communication
delays in places where the firmware needed more time to complete its
operations. Application modules weren't touched. Still, some problems
never went away. The terminal hardware would lock up from time to
time. When the firmware became autistic, up to 16 displays simply
stopped talking, and my software quietly timed out, shutting down user
sessions that appeared to have simply gone away.

Hardware technicians pleaded for a way to diagnose the terminals
from the minicomputer. I gave them a way to analyze the data my
software had about all of the displays, but it was useless. The
hardware interface didn't provide the kind of information that the
technicians needed. We were dealing with a computer program hidden in
firmware where none of us could get to it, let alone fix it.
Well-known industry-standard peripheral interfaces didn't have these
limitations, but the engineers who designed this equipment thought it
was more important to use an inexpensive, limited interface. No
amount of add-on software could compensate for that unfortunate point
of inscrutable failure. When I had been at a mainframe manufacturer a
decade earlier, you couldn't build, let alone ship, a computer system
if you couldn't show people how to troubleshoot and repair it. That
principle didn't transfer over so well when software started becoming
part of everything.

Predefining and sticking to your interfaces allows modules to be
brought together -- and substituted -- without surprises. Preserving
predictability across many cycles of alteration and refinement is
important. When you study the interfaces used by different software
packages, you'll notice that some interfaces work better than others,
and you'll begin to see why. It is the result of a wonderful paradox:
freedom for design and innovation is carved out of the space created
by agreeing to constraints that will never be violated. You must
first set up the rules of the game so there is room to play, and to
play again, and then to win.