The Seven Stages of Expertise in
Software Engineering

Meilir Page-Jones

For almost two decades
we at Wayland Systems Inc. have toiled to transfer the latest developments in
software engineering to practitioners at large and small companies around the
world. However, until recently we didn’t do a very good job of quantifying the
long-term field results of our efforts. So we decided to ask a question
reminiscent of the rhetoric of Ed Koch (the former mayor of New
York City): “How’re we doin’?”

We surveyed some of our
clients to whom we’d provided education and consulting over the years and
discovered an interesting but very disturbing fact. Over 50% of the shops into
which we’d introduced Software Engineering techniques had either abandoned the
use of the techniques or had let their band of active practitioners shrink to a
dwindling core of diehards. What was going wrong here?

We turned to
self-examination and reviewed our courses and consulting practices. But,
whatever the problem was, it didn’t seem limited to Wayland Systems: Informal
discussions with our competitors revealed that they too were having similar
problems. The attrition rate across our whole industry was painfully high.

External examination
revealed several factors at work, many of which were to do with corporate
management habits. Examples of these were: unrealistic management expectations;
a lack of commitment to quality; an obsession with coding (which we termed WIMP
— “Why isn’t Marvin Programming?” — management); pernicious deadlines; and a
misallocation of skills to tasks. Other factors included: engineers’ resistance
to change; engineers’ reluctance to participate in team approaches; and the
then-unavailability of useful CASE tools.

The above factors were
significant and worthy of articles in their own right. However, there was
another factor, which is more important and fundamental than the others. This
factor is the way that people are able to absorb sophisticated new techniques
and then to apply them to their work — in other words the stages of expertise
through which we all pass as we learn and use new and powerful disciplines.

At first we were aware
only that there were two types of person: novice and expert. That was hardly
earth-shattering news. However, further research revealed that there are
actually seven stages of expertise through which a person may pass on the
journey from total ignorance to world-class knowledge. With devastating acumen
we numbered these seven stages Stages 1 through 7.
(We also gave them names: Innocent, Exposed, Apprentice, Practitioner,
Journeyman, Master and Expert.) As I’ll discuss below, these seven stages have
a deep impact on the successful — or otherwise — introduction of software
engineering into an organization.

A participant at a
conference asked me how universal these seven stages are.

“Very universal,” I
answered.

“You mean that I could
even apply them to the skill of bear-hunting?” he retorted unexpectedly.

“Yes,” I said.

So, if you’ll bear with
me, I should like to illustrate the Seven Stages of Expertise with reference to
hunting (although I’m most definitely not a fan of such an activity). Following
that illustration I’ll outline the application of the Seven Stages to our own
business of Software Engineering.

Stage 1: Innocent

A Stage-1 person has
never seen or heard of bears. It would not occur to a Stage-1, if he
encountered a bear, that the bear could be hunted. Neither would he realize
that a bear is a potential source of danger.

Stage 2: Exposed

A Stage-2 person has
seen an occasional bear and has read articles in airline magazines suggesting
that bears may be hunted. Moreover, a Stage-2 probably has friends that have
hunted bears and has learned some desultory but intriguing facts about bears
and their habits. He is motivated to learn more.

Stage 3: Apprentice

A Stage-3 has attended a
5-day seminar on bear hunting. During this seminar, the participants form into
teams of three or four and practice hunting very small bears under the
ever-watchful eye of the instructor. After a few interim setbacks, by Friday
afternoon all the teams have successfully hunted their bears. They fill out
evaluation forms attesting that “bear hunting is very useful and relevant to my
job.” However, they are barely prepared for the world of real bears.

Stage 4: Practitioner

The Stage-4, having completed formal bear-hunting
education, is full of confidence. He’s ready to transcend the minuscule bears
of the 5-day workshop and go out for real bears, larger bears, fierce bears. He’s ready for Ursa
Major. His manager is also keen to send him out with the latest bear-hunting
techniques because the users want fur and they want it yesterday.
Unfortunately, in the resulting scramble the budding bear hunter may be sent
out without a map and with the wrong calibre arrow in his longbow. In the heat
of ursine confrontation, the Stage-4 may also forget or misinterpret his
classroom instruction and precipitate disaster. It’s typical that some Stage-4s
get some bears; but it’s also typical that some bears get some Stage-4s.

Stage 5: Journeyman

The Stage-5 has survived
the traumas of Stage-4 and has bear hunting down cold. The Stage-5 uses modern
bear-hunting techniques naturally and automatically; in fact, he can’t imagine
how he ever got along without them. He is accurate and productive: The Steering
Committee merely points out the bear and he hunts it within both budget and
deadline. The Stage-5 is the exemplary modern hunter that salespeople of
bear-hunting seminars refer to in their brochures.

Stage 6: Master

Stage-6 bear hunters
have internalised not only the mechanics of bear hunting but also the
principles underlying the techniques. Stage-6s know more than rules: They know
why the rules exist and even when it’s permissible to break them. For example,
a Stage-3 or 4 may stand upwind of a bear accidentally and scare off the bear.
However, a Stage-6 may know that by wearing Yogi-Spray Deodorant he can stand
upwind without being detected and can thus surprise the bear from an unexpected
quarter. Because of their deep knowledge, Stage-6s are very capable of training
others in hunting techniques.

Stage 7: Researcher

Stage-7s are asked to
write books and give talks at bear-hunting user groups. They are also engaged
in extending and generalizing bear-hunting techniques to solve new problems.
For example, a Stage-7 may extend bear hunting to work also on Big Foot or he
may even develop the ultimate Yeti-Oriented Hunting Methodology.

Now let’s come back to
the world of software engineering and see how the Seven Stages of Expertise
apply to us.

Stage 1: Innocent

A Stage-1 may not have
heard of Software-Engineering techniques. Or — more likely nowadays — he may be
vaguely aware of their existence but may not see their possible relevance to
his situation. Indeed, he may be only dimly aware that there are any
software-development problems in his shop. You may find it incredible that
Stage-1s could exist in the 1990s, but they do. And the reason stems from the
way in which software complexity evolved.

Software became
insidiously more and more complex in the 1970s and 1980s as users demanded more
and more sophisticated systems be installed on the more and more powerful
hardware that became available. Yet there was no sharp transition. The earth
was not hit by a Complexity Asteroid in 1975 that suddenly made software three
orders of magnitude more complex and cast our reptilian development techniques
into extinction.

I call the way in which
software complexity actually increased “Frog in the Pan.” This is because
although a frog will jump out of a pan of hot water, a frog that is placed in a
pan of cold water and slowly heated will fail to leap forth and will actually
boil to death. The temperature gradient is so gradual that there will never be
a point at which the frog declares: “Boy, it’s suddenly gotten hot in here! I
think I should hop out.” (Before I get into deeper trouble from animal-rights
folks, I hasten to add that this analogy is apocryphal. I’ve never tried the
experiment and I don’t recommend that you do so either!)

Many Stage-1s are
experiencing “Frog in the Pan” and are trying to tackle problems of the 1990s
with approaches of the 1960s and 1970s, without realizing that the problems
they’re facing are the very ones that modern Software-Engineering techniques
were created to alleviate.

Stage 2: Exposed

Stage-2s have noticed
that the water is getting decidedly warm, if not downright hot. So they are
actively seeking Software-Engineering techniques that will get them out of the
pan or at least reduce the heat. Stage-2s may survey magazines, confer with
colleagues or attend one-day overviews of the techniques. Their interest level
is high but their knowledge level is low, being limited to a few terms and
definitions and not based on any practical Software-Engineering experience.

Stage 3: Apprentice

Stage-3s have attended
one or two 5-day workshops on Software-Engineering techniques. In these
workshops they tackled small but realistic case studies that resembled their
own projects in miniature. The case studies provided valuable kinaesthetic
reinforcement of the formal lecture material and were thus indispensable.
However, the case studies’ apparent realism conveyed to the Stage-3 a
confidence that is often unwarranted.

If a Stage-3 absorbs everything from a seminar, then he is
minimally equipped to tackle a true, full-sized project in the corporate
jungle. Usually, however, a Stage-3 does not grasp everything or has difficulty scaling the techniques up from a case
study to a real project. You could say that most Stage-3s know just enough to
be dangerous!

Stage 4: Practitioner

The rite of passage to
Stage-4 is the use of Software-Engineering techniques on at least one
significant project. Achieving “Stage-4-hood” is for many people the most
difficult transition of the six transitions between stages. The fledgling
Stage-4 is asked to take untried (by him) techniques and apply them to a
corporate project with the usual demonic cocktail of politics, deadlines and
changing requirements. At the same time, he is attempting to recall what he
learned in class and scale up the examples 10- or 100-fold. He continually
needs consulting advice, without which he will encounter a series of minor
setbacks or major failures. Since many people throw up their hands at this
point and revert to their old mediocre but familiar ways, a large proportion of
Stage-3s never make it to Stage-4. If an entire project is peopled with
Stage-3s, then it’s highly likely that the project itself will fail and the Software-Engineering
techniques will be publicly pilloried and then abandoned.

Stage 5: Journeyman

Stage-5s have made it.
Their experience of Software Engineering is “latched” in place and there is
little risk of their reverting to the past. In the Stage-5 the Software
Engineering techniques yield for the first time the productivity the marketing
folks promised; and on each successive project a Stage-5 further hones his
skill and enhances his productivity. A Stage-5 is self-sufficient - more often
the source of Software-Engineering advice than its recipient.

Stage 6: Master

The Stage-6 not only is
an adept technician, but also possesses a profound methodological foundation.
Beyond the “whats” and “hows”,
the Stage-6 knows the “whys” of Software Engineering. This depth allows him
sometimes to break a surface rule, while adhering to a more fundamental
methodological principle. The Stage-6 is a good instructor because his
theoretical and practical knowledge give him the wherewithal to tackle
difficult student questions.

Stage 7: Researcher

The Stage-7 is concerned
with delivering the latest developments in Software Engineering to a wider
audience, via books, articles and conference appearances. The Stage-7 looks out
for flaws in contemporary Software-Engineering techniques and for consequent
ways to improve the techniques. He also scans the horizon for new problems
towards whose solution Software Engineering can be extended and generalized.
The Stage-7 is at a methodological pinnacle.

These Seven Stages of
Expertise are valuable in their own right. You might think about a
Software-Engineering technique that you know and consider which of the stages
you are in with respect to that technique. You may also decide what (if
anything) you should do to progress to the next stage.

However, there are
organizational implications behind the Seven Stages. Below I discuss four of
these: The Productivity Curve, Pilot Projects, The Critical Consulting Core and
Ephemeral Technology.

A. The Productivity Curve

Vendors who sell
Software-Engineering techniques do so by convincing management of their boon to
productivity. Although vendors rarely say so explicitly — we at Wayland Systems
never used to — they compare the productivity of Stage-5s to that of Stage-2s.
Unfortunately, this presentation (see Fig. 1) is not a complete picture.
Between Stage-3 and Stage-4 there’s a hair-raising “J-curve” of productivity
(see Fig. 2), which represents the difficult rite of passage of the incipient
Stage-4.

Fig. 1

Fig. 2

This J-curve has led
more than one manager to abandon all hope in Software-Engineering techniques.
Such a manager is not merely impatient; no one warned him of the J-curve and so
when it manifests itself he can only assume that the much-touted
Software-Engineering techniques were mere hokum.

A couple of our clients
measured the length of time from Stages 3 to 5. Depending on the individual
person, Stage-3 to Stage-4 took 6 to 18 months (about the length of a project
or phase) and Stage-4 to Stage-5 took 18 to 36 months. (These times are very
approximate because of the subjective nature of the stages.) Except for a few
outstanding people, the shortest time from Software-Engineering seminar to
exquisite software productivity is 24 months. When managers tell me: “We just
can’t wait that long,” I retort: “The 2 years will pass anyway, whether you do
anything or not. Act now!”

(By the way, notice that
Stage-7s have lower project productivity than Stage-6s. This is because a
portion of Stage-7s’ energy is directed away from the specific project and
towards general methodological research.)

B. Pilot Projects

But the J-curve has a
facet nastier than simple delay in productivity. This is the “Who will bell the
cat?” problem. In the fable of that title, a group of mice that’d been
persecuted by a cat decided to hang a bell around the cat’s neck that would
warn them of its approach. All agreed that this was a wonderful idea ... but
there was one minor difficulty: None of the mice wanted to be the one to risk
walking up to the sleeping cat to tie a bell around its neck. This impasse is
known formally as the Technology Trap.

Most project managers
would agree that Software-Engineering techniques are a Good Thing. But which of
them with bone-crushing deadlines and axe-wielding users wants to be the first
to experience the depths of the J-curve. (Me neither!) Thus many shopfuls of intelligent people appear to be dinosaurs
because the Technology Trap holds them in the 1970s.

If “who will bell the
cat?” is a problem, then I recommend that you try Software-Engineering
techniques on a pilot project first. The pilot will uncover all manner of
pitfalls that occur during the Stage-3/4 transition as well as problems in the
techniques themselves. The pilot should be short (6 months) and should yield a
non-critical system. You should learn as much as you can about applying the new
technology from the pilot project; this is one project where the learning
experience is more important than the end product.

A pilot project will not
eliminate J-curves on subsequent projects. People will still have to go from
Stage-3 to Stage-4. But the pilot will make future J-curves shallower, as some
causes of the curve’s nadir are removed. Indeed, although the delivered system
is the most important product of most projects, you should never forget the
learning by-product of every project. Review what you discovered about doing
projects when you complete each project and share that information with others.
In that sense, treat every project as a pilot.

C. Core of Internal / External
Consultants

One of the surest recipes
for disaster is to staff a project with only Stage-3s. Yet many managers do
exactly that, pushing a group of analysts through a seminar one week and
letting them loose on a crucial project the next week. (Jerry Weinberg refers
to this as the “Sheep Dip” approach.)

Every project should
have access to a Stage-5, or - better still — have a Stage-5 on the project
team. Cultivate your own Stage-5s and Stage-6s in-house as part of your
long-term staff-development policy and make them available to project teams. If
you don’t yet have such people on board, then call in external consultants from
Wayland Systems or one of our competitors. However, don’t become chronically
reliant on external consultants, but instead continue to work on building up a
pyramid of internal expertise with Stage-6s at the top that will ensure a
lasting mastery of Software Engineering.

D. Learning Latency vs. Lifetime of
Technique

One manager of a major
organization recently complained to me that by the time a critical mass of his
people had learned a Software-Engineering technique that technique had been
left behind in the mud of obsolescence by the relentless march of progress.
Although there is, of course, some truth in this complaint, it is also true
that “plus ça change, plus c’est
la même chose.” If you buy a power drill to replace a
manual drill, the deep principles of drilling will be largely unchanged.
Similarly, most new Software-Engineering methods are basically founded on
familiar underlying methodological principles (such as the principle of “Safe
Deferral” or of “Partitioning to Minimize Interfaces”).

Develop a large
“capacitance” of Stage-6s in your organization in order to smooth the apparent
high-frequency ripples in technology. These people will be able quickly to absorb
a new technique and to assess its applicability to the shop’s needs. They will
also be able to compare and contrast the new technique to current techniques
and to steady the transition from old to new, because many of the new ways of
addressing software are simply refinements of ways already known to Stages 3, 4
and 5.

Finally — Some Recommendations

·Be aware of the Seven Stages of Expertise and
their effects on productivity under a specific Software-Engineering technique.
Reconcile your expectations for Software Engineering with the realities of the
Seven Stages.

·Take stock of where you are. Have each person
assess his stage with respect to each Software-Engineering technique that your
shop uses.

·Have each person state their medium- and
long-term goals for expertise in each relevant technique and suggest a plan for
achieving those goals.

·Establish similar medium- and long-term goals
for the shop as whole. Determine the most appropriate blend of stages for the
work that you do and set up a plan to achieve that blend.

·Never attempt a crucial project solely with
Stage-3s and below. Seed each project with Stage-4s and allow access to
Stage-5s and 6s if possible. If you don’t have a Stage-5, beg, borrow, steal,
buy, kidnap... one. External consultants are valuable as a short-term measure.

·Stabilize the effects of technological change by
nurturing an in-house expertise (Stage-6) in fundamental methodological
principles. Try to formulate shop strategies (as much as you can) in
non-technological terms. Superimpose technology on to the strategy in order to
yield tactics.

·Have a mission that will never go away. For
example, the mission of Wayland Systems is “Mastery in Software Engineering.”
In other words, we seek to assist our clients in becoming self-sustaining at at least the Stage-6 level of Software Engineering. This
mission will not evaporate for many decades yet, for software is becoming ever
more complex and ever more critical. There will be much to learn — even for Stage-6s
and Stage-7s — as we pass into the new century.