The New MethodologyIn the past few years there's been a rapidly growing interest in agile (aka "lightweight") methodologies. Alternatively characterized as an antidote to bureaucracy or a license to hack they've stirred up interest all over the software landscape.https://www.pilotsystems.net/actus/the-new-methodologyhttps://www.pilotsystems.net/@@site-logo/logoPS_RS.png

The New Methodology

In the past few years there's been a rapidly growing interest in agile (aka "lightweight") methodologies. Alternatively characterized as an antidote to bureaucracy or a license to hack they've stirred up interest all over the software landscape.

In the past few years there's been a rapidly growing interest in
agile (aka "lightweight") methodologies. Alternatively characterized as
an antidote to bureaucracy or a license to hack they've stirred up
interest all over the software landscape. In this essay I explore the
reasons for agile methods, focusing not so much on their weight but on
their adaptive nature and their people-first orientation. I also give a
summary and references to the processes in this school and consider the
factors that should influence your choice of whether to go down this
newly trodden path.

From Nothing, to Monumental, to Agile

Most software development is a chaotic activity, often
characterized by the phrase "code and fix". The software is written
without much of an underlying plan, and the design of the system is
cobbled together from many short term decisions. This actually works
pretty well as the system is small, but as the system grows it becomes
increasingly difficult to add new features to the system. Furthermore
bugs become increasingly prevalent and increasingly difficult to fix.
A typical sign of such a system is a long test phase after the system
is "feature complete". Such a long test phase plays havoc with
schedules as testing and debugging is impossible to schedule.

We've lived with this style of development for a long time, but
we've also had an alternative for a long time: Methodology.
Methodologies impose a disciplined process upon software development
with the aim of making software development more predictable and more
efficient. They do this by developing a detailed process with a strong
emphasis on planning inspired by other engineering disciplines - which
is why I tend to refer to them as engineering methodologies.

Engineering methodologies have been around for a long time. They've
not been noticeable for being terribly successful. They are even less
noted for being popular. The most frequent criticism of these
methodologies is that they are bureaucratic. There's so much stuff to
do to follow the methodology that the whole pace of development slows
down.

As a reaction to these methodologies, a new group of
methodologies have appeared in the last few years. For a while these
were known a lightweight methodologies, but now the accepted term is
agile methodologies. For many people the appeal of these agile
methodologies is their reaction to the bureaucracy of the monumental
methodologies. These new methods attempt a useful compromise between
no process and too much process, providing just enough process to gain
a reasonable payoff.

The result of all of this is that agile methods have some
significant changes in emphasis from engineering methods. The most
immediate difference is that they are less document-oriented, usually
emphasizing a smaller amount of documentation for a given task. In
many ways they are rather code-oriented: following a route that says
that the key part of documentation is source code.

However I don't think this is the key point about agile
methods. Lack of documentation is a symptom of two much deeper
differences:

Agile methods are adaptive rather than predictive.
Engineering methods tend to try to plan out a large part of the software
process in great detail for a long span of time, this works well until
things change. So their nature is to resist change. The agile methods,
however, welcome change. They try to be processes that adapt and
thrive on change, even to the point of changing themselves.

Agile methods are people-oriented rather than
process-oriented. The goal of engineering methods is to define a
process that will work well whoever happens to be using it. Agile
methods assert that no process will ever make up the the skill of the
development team, so the role of a process is to support the
development team in their work.

In the following sections I'll explore these differences in
more detail, so that you can understand what an adaptive and
people-centered process is like, its benefits and drawbacks, and
whether it's something you should use: either as a developer or
customer of software.

Predictive versus Adaptive

Separation of Design and Construction

The usual inspiration for methodologies is engineering
disciplines such as civil or mechanical engineering. Such disciplines
put a lot of emphasis on planning before you build. Such engineers
will work on a series of drawings that precisely indicate what needs
to be built and how these things need to be put together. Many design
decisions, such as how to deal with the load on a bridge, are made as
the drawings are produced. The drawings are then handed over to a
different group, often a different company, to be built. It's assumed
that the construction process will follow the drawings. In practice
the constructors will run into some problems, but these are usually
small.

Since the drawings specify the pieces and how they need to be
put together, they act as the foundation for a detailed construction
plan. Such a plan can figure out the tasks that need to be done and
what dependencies exist between these tasks. This allows for a
reasonably predictable schedule and budget for construction. It also
says in detail how the people doing the construction work should do
their work. This allows the construction to be less skilled
intellectually, although they are often very skilled manually.

So what we see here are two fundamentally different
activities. Design which is difficult to predict and requires
expensive and creative people, and construction which is easier
to predict. Once we have the design, we can plan the construction.
Once we have the plan for the construction, we can then deal with
construction in a much more predictable way. In civil engineering
construction is much bigger in both cost and time than design and
planning.

So the approach for software engineering methodologies looks
like this: we want a predictable schedule that can use people with
lower skills. To do this we must separate design from construction.
Therefore we need to figure out how to do the design for software so
that the construction can be straightforward once the planning is
done.

So what form does this plan take? For many, this is the role
of design notations such as the UML. If
we can make all the significant decisions using the UML, we can build
a construction plan and then hand these designs off to coders as a
construction activity.

But here lies the crucial question. Can you get a design that is
capable of turning the coding into a predictable construction
activity? And if so, is cost of doing this sufficiently small to make
this approach worthwhile?

All of this brings a few questions to mind. The first is the
matter of how difficult it is to get a UML-like design into a state
that it can be handed over to programmers. The problem with a UML-like
design is that it can look very good on paper, yet be seriously flawed
when you actually have to program the thing. The models that civil
engineers use are based on many years of practice that are enshrined
in engineering codes. Furthermore the key issues, such as the way
forces play in the design, are amenable to mathematical analysis. The
only checking we can do of UML-like diagrams is peer review. While
this is helpful it leads to errors in the design that are often only
uncovered during coding and testing. Even skilled designers, such as I
consider myself to be, are often surprised when we turn such a design
into software.

Another issue is that of comparative cost. When you build a
bridge, the cost of the design effort is about 10% of the job, with
the rest being construction. In software the amount of time spent in
coding is much, much less McConnell suggests that for a large
project, only 15% of the project is code and unit test, an almost
perfect reversal of the bridge building ratios. Even if you lump in
all testing as part of construction, then design is still 50% of the
work. This raises an important question about the nature of design in
software compared to its role in other branches of engineering.

These kinds of questions led Jack Reeves to suggest that in fact the source code is a
design document and that the construction phase is actually the use of
the compiler and linker. Indeed anything that you can treat as
construction can and should be automated.

This thinking leads to some important conclusions:

In software: construction is so cheap as to be free

In software all the effort is design, and thus requires
creative and talented people

Creative processes are not easily planned, and so
predictability may well be an impossible target.

We should be very wary of the traditional engineering
metaphor for building software. It's a different kind of activity and
requires a different process

The Unpredictability of Requirements

There's a refrain I've heard on every problem project I've
run into. The developers come to me and say "the problem with this
project is that the requirements are always changing". The thing I
find surprising about this situation is that anyone is surprised by
it. In building business software requirements changes are the norm,
the question is what we do about it.

One route is to treat changing requirements as the result of
poor requirements engineering. The idea behind requirements
engineering is to get a fully understood picture of the requirements
before you begin building the software, get a customer sign-off to
these requirements, and then set up procedures that limit requirements
changes after the sign-off.

One problem with this is that just trying to understand the
options for requirements is tough. It's even tougher because the
development organization usually doesn't provide cost information on
the requirements. You end up being in the situation where you may have
some desire for a sun roof on your car, but the salesman can't tell
you if it adds $10 to the cost of the car, or $10,000. Without much
idea of the cost, how can you figure out whether you want to pay for
that sunroof?

Estimation is hard for many reasons. Part of it is that
software development is a design activity, and thus hard to plan and
cost. Part of it is that the basic materials keep changing rapidly.
Part of it is that so much depends on which individual people are
involved, and individuals are hard to predict and quantify.

Software's intangible nature also cuts in. It's very
difficult to see what value a software feature has until you use it
for real. Only when you use an early version of some software do you
really begin to understand what features are valuable and what parts
are not.

This leads to the ironic point that people expect that
requirements should be changeable. After all software is supposed to
be soft. So not just are requirements changeable, they ought to
be changeable. It takes a lot of energy to get customers of software
to fix requirements. It's even worse if they've ever dabbled in
software development themselves, because then they "know" that
software is easy to change.

But even if you could settle all that and really could get an
accurate and stable set of requirements you're probably still doomed.
In today's economy the fundamental business forces are changing the
value of software features too rapidly. What might be a good set of
requirements now, is not a good set in six months time. Even if the
customers can fix their requirements, the business world isn't going
to stop for them. And many changes in the business world are
completely unpredictable: anyone who says otherwise is either lying,
or has already made a billion on stock market trading.

Everything else in software development depends on the
requirements. If you cannot get stable requirements you cannot get a
predictable plan.

Is Predictability Impossible?

In general, no. There are some software developments where
predictability is possible. Organizations such as NASA's space shuttle
software group are a prime example of where software development can
be predictable. It requires a lot of ceremony, plenty of time, a large
team, and stable requirements. There are projects out there that are
space shuttles. However I don't think much business software fits into
that category. For this you need a different kind of process.

One of the big dangers is to pretend that you can follow a
predictable process when you can't. People who work on methodology are
not very good at identifying boundary conditions: the places where the
methodology passes from appropriate in inappropriate. Most
methodologists want their methodologies to be usable by everyone, so
they don't understand nor publicize their boundary conditions. This
leads to people using a methodology in the wrong circumstances, such
as using a predictable methodology in a unpredictable situation.

There's a strong temptation to do that. Predictability is a
very desirable property. However if you believe you can be predictable
when you can't, it leads to situations where people build a plan early
on, then don't properly handle the situation where the plan falls
apart. You see the plan and reality slowly drifting apart. For a long
time you can pretend that the plan is still valid. But at some point
the drift becomes too much and the plan falls apart. Usually the fall
is painful.

So if you are in a situation that isn't predictable you can't
use a predictive methodology. That's a hard blow. It means that many
of the models for controlling projects, many of the models for the
whole customer relationship, just aren't true any more. The benefits
of predictability are so great, it's difficult to let them go. Like so
many problems the hardest part is simply realizing that the problem
exists.

However letting go of predictability doesn't mean you have to
revert to uncontrollable chaos. Instead you need a process that can
give you control over an unpredictability. That's what adaptivity is
all about.

Controlling an Unpredictable Process - Iterations

So how do we control ourselves in an unpredictable world? The
most important, and still difficult part is to know accurately where
we are. We need an honest feedback mechanism which can accurately tell
us what the situation is at frequent intervals.

The key to this feedback is iterative development. This is
not a new idea. Iterative development has been around for a while
under many names: incremental, evolutionary, staged, spiral... lots of
names. The key to iterative development is to frequently produce
working versions of the final system that have a subset of the
required features. These working systems are short on functionality,
but should otherwise be faithful to the demands of the final system.
They should be fully integrated and as carefully tested as a final
delivery.

The point of this is that there is nothing like a tested,
integrated system for bringing a forceful dose of reality into any
project. Documents can hide all sorts of flaws. Untested code can hide
plenty of flaws. But when people actually sit in front of a system and
work with it, then flaws become truly apparent: both in terms of bugs
and in terms of misunderstood requirements.

Iterative development makes sense in predictable processes as
well. But it is essential in adaptive processes because an adaptive
process needs to be able to deal with changes in required features.
This leads to a style of planning where long term plans are very
fluid, and the only stable plans are short term plans that are made
for a single iteration. Iterative development gives you a firm
foundation in each iteration that you can base your later plans
around.

A key question for this is how long an iteration should be.
Different people give different answers. XP suggests iterations of
between one and three weeks. SCRUM suggests a length of a month.
Crystal will stretch further. The tendency, however, is to make each
iteration as short as you can get away with. This provides more
frequent feedback, so you know where you are more often.

The Adaptive Customer

This kind of adaptive process requires a different kind of
relationship with a customer than the ones that are often considered,
particularly when development is done by a separate firm. When you
hire a separate firm to do software development, most customers would
prefer a fixed-price contract. Tell the developers what they want, ask
for bids, accept a bid, and then the onus is on the development
organization to build the software.

A fixed price contract requires stable requirements and hence
a predictive process. Adaptive processes and unstable requirements
imply you cannot work with the usual notion of fixed-price. Trying to
fit a fixed price model to an adaptive process ends up in a very
painful explosion. The nasty part of this explosion is that the
customer gets hurt every bit as much as the software development
company. After all the customer wouldn't be wanting some software
unless their business needed it. If they don't get it their business
suffers. So even if they pay the development company nothing, they
still lose. Indeed they lose more than they would pay for the software
(why would they pay for the software if the business value of that
software were less?)

So there's dangers for both sides in signing a fixed price
contract in conditions where a predictive process cannot be used. This
means that the customer has to work differently.

This doesn't mean that you can't fix a budget for software
up-front. What it does mean is that you cannot fix time, price and
scope. The usual agile approach is to fix time and price, and to allow
the scope to vary in a controlled manner.

In an adaptive process the customer has much finer-grained
control over the software development process. At every iteration they
get both to check progress and to alter the direction of the software
development. This leads to much closer relationship with the software
developers, a true business partnership. This level of engagement is
not for every customer organization, nor for every software developer;
but it's essential to make an adaptive process work properly.

All this yields a number of advantages for the customer. For
a start they get much more responsive
software development. A usable, although minimal, system can go into
production early on. The customer can then change its capabilities
according to changes in the business, and also from learning from how
the system is used in reality.

Every bit as important as this is greater visibility into the
true state of the project. The problem with predictive processes is
that project quality is measured by conformance to plan. This makes it
difficult for people to signal when reality and the plan diverge. The
common result is a big slip in the schedule late in the project. In an
agile project there is a constant reworking of the plan with every
iteration. If bad news is lurking it tends to come earlier, when there
is still time to do something about it. Indeed this risk control is a
key advantage of iterative development. Agile methods take this
further by keeping the iteration lengths small, but also by seeing
these variations as opportunities.

This has an important bearing what constitutes a successful
project. A predictive project is often measured by how well it met its
plan. A project that's on-time and on-cost is considered to be a
success. This measurement is nonsense to an agile environment. For
agilists the question is business value - did the customer get
software that's more valuable to them them than the cost put into it.
A good predictive project will go according to plan, a good agile
project will build something different and better than the original
plan foresaw.

Putting People First

Executing an adaptive process is not easy. In particular it
requires a very effective team of developers. The team needs to be
effective both in the quality of the individuals, and in the way the
team blends together. There's also an interesting synergy: not just
does adaptivity require a strong team, most good developers prefer an
adaptive process.

Plug Compatible Programming Units

One of the aims of traditional methodologies is to develop a
process where the people involved are replaceable parts. With such a
process you can treat people as resources who are available in various
types. You have an analyst, some coders, some testers, a manager. The
individuals aren't so important, only the roles are important. That
way if you plan a project it doesn't matter which analyst and which
testers you get, just that you know how many you have so you know how
the number of resources affects your plan.

But this raises a key question: are the people involved in
software development replaceable parts? One of the key features of
agile methods is that they reject this assumption.

Perhaps the most explicit rejection of people as resources is
Alistair Cockburn. In his paper Characterizing People as Non-Linear,
First-Order Components in Software Development, he makes the
point that predictable processes require components that behave in a
predictable way. However people are not predictable components.
Furthermore his studies of software projects have led him to conclude
the people are the most important factor in software development.

In the title, [of his article]
I refer to people as "components". That is how people are treated in
the process / methodology design literature. The mistake in this
approach is that "people" are highly variable and non-linear, with
unique success and failure modes. Those factors are first-order, not
negligible factors. Failure of process and methodology designers to
account for them contributes to the sorts of unplanned project
trajectories we so often see.
--
[Cockburn non-linear]

One wonders if not the nature of software development works
against us here. When we're programming a computer, we control an
inherently predictable device. Since we're in this business because
we are good at doing that, we are ideally suited to messing up when
faced with human beings.

Although Cockburn is the most explicit in his people-centric
view of software development, the notion of people first is a common
theme with many thinkers in software. The problem, too often, is that
methodology has been opposed to the notion of people as the
first-order factor in project success.

This creates a strong positive feedback effect. If you expect
all your developers to be plug compatible programming units, you don't
try to treat them as individuals. This lowers morale (and
productivity). The good people look for a better place to be, and you
end up with what you desire: plug compatible programming units.

Deciding that people come first is a big decision, one that
requires a lot of determination to push through. The notion of people
as resources is deeply ingrained in business thinking, its roots going
back to the impact of Frederick
Taylor's Scientific Management approach. In running a factory,
this Taylorist approach may make sense. But for the highly creative and
professional work, which I believe software development to be, this
does not hold. (And in fact modern manufacturing is also moving away
from the Taylorist model.)

Programmers are Responsible Professionals

A key part of the Taylorist notion is that the people doing
the work are not the people who can best figure out how best to do
that work. In a factory this may be true for several reasons. Part of
this is that many factory workers are not the most intelligent or
creative people, in part this is because there is a tension between
management and workers in that management makes more money when the
workers make less.

Recent history increasingly shows us how untrue this is for
software development. Increasingly bright and capable people are
attracted to software development, attracted by both its glitz and by
potentially large rewards. (Both of which tempted me away from
electronic engineering.) Such schemes as stock options increasingly
align the programmers interests with the company's.

(There may well be a generational effect here. Some anecdotal
evidence makes me wonder if more brighter people have ventured into
software engineering in the last ten years or so. If so this would be
a reason for why there is such a cult of youth in the computer
business, like most cults there needs to be a grain of truth in it.)

When you want to hire and retain good people, you have to
recognize that they are competent professionals. As such they are the
best people to decide how to conduct their technical work. The
Taylorist notion of a separate planning department that decides how to
do things only works if the planners understand how to do the job
better than those doing it. If you have bright, motivated people doing
the job then this does not hold.

Managing a People Oriented Process

People orientation manifests itself in a number of different
ways in agile processes. It leads to different effects, not all of
them are consistent.

One of the key elements is that of accepting the process
rather than the imposition of a process. Often software processes are
imposed by management figures. As such they are often resisted,
particularly when the management figures have had a significant amount
of time away from active development. Accepting a process requires
commitment, and as such needs the active involvement of all the team.

This ends up with the interesting result that only the
developers themselves can choose to follow an adaptive process. This
is particularly true for XP, which requires a lot of discipline to
execute. This is where Crystal is such an effective complement as it
aims at being minimally disciplined.

Another point is that the developers must be able to make
all technical decisions. XP gets to the heart of this where in
its planning process it states that only developers may make estimates
on how much time it will take to do some work.

Such technical leadership is a big shift for many people in
management positions. Such an approach requires a sharing of
responsibility where developers and management have an equal place in
the leadership of the project. Notice that I say equal.
Management still plays a role, but recognizes the expertise of
developers.

An important reason for this is the rate of change of
technology in our industry. After a few years technical knowledge
becomes obsolete. This half life of technical skills is without
parallel in any other industry. Even technical people have to
recognize that entering management means their technical skills will
wither rapidly. Ex-developers need to recognize that their technical
skills will rapidly disappear and they need to trust and rely on
current developers.

The Difficulty of Measurement

If you have a process where the people who say how work
should be done are different from the people who actually do it, the
leaders need some way of measuring how effective the doers are. In
Scientific Management there was a strong push to develop objective
approaches to measuring the output of people.

This is particularly relevant to software because of the
difficulty of applying measurement to software. Despite our best
efforts we are unable to measure the most simple things about
software, such as productivity. Without good measures for these
things, any kind of external control is doomed.

Introducing measured management without good measures leads to
its own problems. Robert Austin made an
excellent discussion of this. He points out that when measuring
performance you have to get all the important factors under
measurement. Anything that's missing has the inevitable result that
the doers will alter what they do to produce the best measures, even
if that clearly reduces the true effectiveness of what they do. This
measurement dysfunction is the Achilles heel of measurement-based management.

Austin's conclusion is that you have to choose between
measurement-base management and delagatory management (where the doers
decide how to the work). Measurement-based management is best suited
to repetitive simple work, with low knowledge requirements and easily
measured outputs - exactly the opposite of software development.

The point of all this is that traditional methods have
operated under the assumption that measurement-based management is the
most efficient way of managing. The agile community recognizes that
the characteristics of software development are such that measurement
based management leads to very high levels of measurement dysfunction.
It's actually more efficient to use a delegatory style of management,
which is the kind of approach that is at the center of the agilist
viewpoint.

The Role of Business Leadership

But the technical people cannot do the whole process
themselves. They need guidance on the business needs. This leads to
another important aspect of adaptive processes: they need very close
contact with business expertise.

This goes beyond most projects involvement of the business
role. Agile teams cannot exist with occasional communication . They
need continuous access to business expertise. Furthermore this access
is not something that is handled at a management level, it is
something that is present for every developer. Since developers are
capable professionals in their own discipline, they need to be able to
work as equals with other professionals in other disciplines.

A large part of this, of course, is due to the nature of
adaptive development. Since the whole premise of adaptive development
is that things change quickly, you need constant contact to advise
everybody of the changes.

There is nothing more frustrating to a developer than seeing
their hard work go to waste. So it's important to ensure that there is
good quality business expertise that is both available to the
developer and is of sufficient quality that the developer can trust
them.

The Self-Adaptive Process

So far I've talked about adaptivity in the context of a project
adapting its software frequently to meet the changing requirements of
its customers. However there's another angle to adaptivity: that of
the process changing over time. A project that begins using an
adaptive process won't have the same process a year later. Over time,
the team will find what works for them, and alter the process to fit.

The first part of self-adaptivity is regular reviews of the
process. Usually you do these with every iteration. At the end of each
iteration, have a short meeting and ask yourself the following
questions (culled from Norm
Kerth)

What did we do well?

What have we learned?

What can we do better?

What puzzles us?

These questions will lead you to ideas to change the process
for the next iteration. In this way a process that starts off with
problems can improve as the project goes on, adapting better to the
team that uses it.

If self-adaptivity occurs within a project, it's even more
marked across an organization. To deepen the process of
self-adaptivity I suggest teams do a more formal review and major
project milestones following the project retrospective sessions
outlined by Norm Kerth. These retrospectives involve a 2-3 day offsite
meeting and a trained facilitator. Not only do they provide learning
for the team, they also provide learning for the whole organization.

A consequence of self-adaptivity is that you should never
expect to find a single corporate methodology. Instead each team
should not just choose their own process, but should also actively
tune their process as they proceed with the project. While both
published processes and the experience of other projects can act as an
inspiration and a baseline, the developers professional responsibility
is to adapt the process to the task at hand.

This self-adaptivity is most marked in ASD and Crystal. XP's
rigid rules seem to disallow it, but that is only a surface impression
since XP does encourage people to tune the process. The main
difference with XP is that its advocates suggest doing XP by the book
for several iterations before adapting it. In addition reviews are
neither emphasized, nor part of the process, although there are
suggestions that reviews should be made one of the XP practices.

The Methodologies

Several methodologies fit under this agile banner. While all of
them share many characteristics, there are also some significant
differences. I can't highlight all the points in this brief survey,
but at least I can point you to some places to look. I also can't
speak with significant experience about most of these. I've done quite
a lot of work based on XP, and seen RUP around in many guises, but
with most of the others my knowledge is primarily the less adequate
book knowledge.

XP (Extreme Programming)

Of all the agile methodologies, this is the one that has got
the most attention. Partly this is because of the remarkable ability
of the leaders of XP, in particular Kent Beck, to get attention. It's
also because of the ability of Kent Beck to attract people to the
approach, and to take a leading role in it. In some ways, however, the
popularity of XP has become a problem, as it has rather crowded out
the other methodologies and their valuable ideas.

The roots of XP lie in the Smalltalk community, and in
particular the close collaboration of Kent Beck and Ward Cunningham in
the late 1980's. Both of them refined their practices on numerous
projects during the early 90's, extending their ideas of a software
development approach that was both adaptive and people-oriented.

The crucial step from informal practice to a methodology
occurred in the spring of 1996. Kent was asked to review the progress
of the C3 payroll project for Chrysler. The project was being carried
out in Smalltalk by a contracting company, and was in trouble. Due to
the low quality of the code base, Kent recommended throwing out the
entire code base and starting from scratch. The project then restarted
under his leadership and since became the early flagship and training
ground for XP.

The first phase of C3 was very successful and went live in
early 1997. The project continued since and ran into difficulties
later, which resulted in the canceling of further development in 1999.
(which if nothing else proves that XP is no guarantee of success.)

XP begins with four values: Communication, Feedback,
Simplicity, and Courage. It then builds up to a dozen practices which
XP projects should follow. Many of these practices are old, tried and
tested techniques, yet often forgotten by many, including most
planned processes. As well as resurrecting these techniques, XP weaves
them into a synergistic whole where each one is reinforced by the
others.

One of the most striking, as well as initially appealing to
me, is its strong emphasis on testing. While all processes mention
testing, most do so with a pretty low emphasis. However XP puts
testing at the foundation of development, with every programmer
writing tests as they write their production code. The tests are
integrated into a continuous integration and build process which
yields a highly stable platform for future development.

On this platform XP builds an evolutionary design process
that relies on refactoring a simple base system with every iteration.
All design is centered around the current iteration with no design
done for anticipated future needs. The result is a design process that
is disciplined, yet startling, combining discipline with adaptivity in
a way that arguably makes it the most well developed of all the
adaptive methodologies.

XP has developed a wide leadership, many of them springing
from the seminal C3 project. As a result there's a lot of sources for
more information. Kent Beck wrote Extreme
Programming Explained the key manifesto of XP, which
explains the rationale behind the methodology and enough of an
explanation of it to tell folks if they are interested in pursuing it
further. In the last couple of years there's been an epidemic of
brightly colored XP books, most of which are pretty similar in that
they describe the whole process from the point of view of various
early adopters.

As well as books, there are a fair number of web resources.
To find a more structured approach to XP, it's best to start with two
sites from C3 alumni: Ron Jeffries's xProgramming.com and Don Wells's extremeProgramming.org. Much of the early advocacy and development of the XP
ideas occurred on Ward Cunningham's wiki web
collaborative writing environment. The wiki remains a
fascinating place to discover, although its rambling nature does lead
you into being sucked in. There is an active, and often interesting xp discussion egroup. One of the more
interesting "outside" views of XP is that of Mark Paulk, who is one of
the leaders of the CMM community - his paper looks at XP from a CMM perspective.

Cockburn's Crystal Family

Alistair Cockburn has been
working on methodology ever since he was tasked by IBM to write about
methodology in the early 90's. His approach is unlike most
methodologists, however. Instead of building on solely personal
experience to build a theory of how things should be done, he
supplements his direct experience with actively seeking to interview
projects to see how they work. Furthermore he isn't afraid to alter
his views based on his discoveries: all of which make him my favorite
methodologist.

His book, Surviving
Object-Oriented Projects, was his first piece of advice on
running projects, and remains my number one book recommendation for
running iterative projects. More recently Alistair has written an
overview book on agile software
development that looks at the underlying principles of these
kinds of methodologies.

Since that book he's explored agile methods further, coming
up with the Crystal family of
methodologies. It's a family because he believes that different
kinds of projects require different kinds of methodologies. He looks
into this variation along two axes: the number of people in the
project, and the consequences of errors. Each methodology fits into a
different part of the grid, so a 40 person project that can lose
discretionary money has a different methodology than a six person
life-critical project.

The Crystals share a human orientation with XP, but this
people-centeredness is done in a different way. Alistair considers
that people find it hard to follow a disciplined process, thus rather
than follow XP's high discipline, Alistair explores the least
disciplined methodology that could still succeed, consciously trading
off productivity for ease of execution. He thus considers that
although Crystal is less productive than XP, more people will be able
to follow it.

Alistair also puts a lot of weight in end of iteration
reviews, thus encouraging the process to be self-improving. His
assertion is that iterative development is there to find problems
early, and then to enable people to correct them. This places more
emphasis on people monitoring their process and tuning it as they
develop.

Open Source

You may be surprised by this heading. After all open source
is a style of software, not so much a process. However there is a
definite way of doing things in the open source community, and much of
their approach is as applicable to closed source projects as it is to
open source. In particular their process is geared to physically
distributed teams, which is important because most adaptive processes
stress co-located teams.

Most open source projects have one or more maintainers. A
maintainer is the only person who is allowed to commit a change into
the source code repository. However people other than the maintainer
may make changes to the code base. The key difference is that other
folks need to send their change to the maintainer, who then reviews it
and applies it to the code base. Usually these changes are made in the
form of patch files which make this process easier. The maintainer
thus is responsible for coordinating the patches and maintaining the
design cohesion of the software.

Different projects handle the maintainer role in different
ways. Some have one maintainer for the whole project, some divide into
modules and have a maintainer per module, some rotate the maintainer,
some have multiple maintainers on the same code, others have a
combination of these ideas. Most open source folks are part time, so
there is an issue on how well such a team coordinates for a full time
project.

A particular feature of open source development is that
debugging is highly parallelizable. So many people can be involved in
debugging. When they find a bug they can send the patch to the
maintainer. This is a good role for non-maintainers since most of the
time is spent finding the bug. It's also good for folks without strong
design skills.

The process for open-source isn't well written up as yet. The
most famous paper is Eric Raymond's The
Cathedral and the Bazar, which while an excellent
description is also rather brief. Karl Fogel's book on the CVS code repository also contains
several good chapters on open-source process that would be interesting
even to those who never want to do cvs update.

Highsmith's Adaptive Software Development

Jim Highsmith has
spent many years working with predictive methodologies. He developed
them, installed them, taught them, and has concluded that they are
profoundly flawed: particularly for modern businesses.

His recent book focuses on the
adaptive nature of new methodologies, with a particular emphasis on
applying ideas that originate in the world of complex adaptive systems
(commonly referred to as chaos theory.) It doesn't provide the kind of
detailed practices like the XP work does, but it does provide the
fundamental groundwork for why adaptive development is important and
the consequences at the deeper organizational and management levels.

At the heart of ASD are three non-linear, overlapping phases:
speculation, collaboration, and learning.

Highsmith views planning as a paradox in an adaptive
environment, since outcomes are naturally unpredictable. In
traditional planning, deviations from plans are mistakes that should
be corrected. In an adaptive environment, however, deviations guide us
towards the correct solution.

In this unpredictable environment you need people to
collaborate in a rich way in order to deal with the uncertainty.
Management attention is less about telling people what to do, and more
about encouraging communication so that people can come up with
creative answers themselves.

In predictive environments, learning is often discouraged.
You lay out things in advance and then follow that design.

In an adaptive environment, learning
challenges all stakeholders - developers and their customers - to
examine their assumptions and to use the results of each development
cycle to adapt the next.
--
[Highsmith]

As such learning is a continuous and important feature, one that
assumes that plans and designs must change as development proceeds.

The overriding, powerful, indivisible,
predominant benefit of the Adaptive Development Life Cycle is that it
forces us to confront the mental models that are at the root of our
self-delusion. It forces us to more realistically estimate our
ability.
--
[Highsmith]

With this emphasis, Highsmith's work focuses directly on to
foster the hard parts of adaptive development, in particular how to
foster collaboration and learning within the project. As such his book
helps provide ideas to foster these "soft" areas which makes a nice
complement to the grounded practice based approaches such as XP, FDD,
and Crystal.

Scrum

Scrum has been around for a while in object-oriented circles,
although I'll confess I'm not too au fait with its history or
development. Again it focuses on the fact that defined and repeatable
processes only work for tackling defined and repeatable problems with
defined and repeatable people in defined and repeatable environments.

Scrum divides a project into iterations (which they call
sprints) of 30 days. Before you begin a sprint you define the
functionality required for that sprint and then leave the team to
deliver it. The point is to stabilize the requirements during the
sprint.

However management does not disengage during the sprint.
Every day the team holds a short (fifteen minute) meeting, called a
scrum, where the team runs through what it will do in the next day. In
particular they surface to the management blocks: impediments to
progress that are getting in the way that management needs to resolve.
They also report on what's been done so management gets a daily update
of where the project is.

Scrum literature focuses mainly on the iterative planning and
tracking process. It's very close to the other agiles in many respects
and should work well with the coding practices from XP.

After a long time without a book, finally Ken Schwaber and
Mike Beedle have written the first scrum
book. Ken Schwaber also hosts controlChaos.com[missing reference] which is probably the best overview of
SCRUM. Jeff Sutherland has always has an active web site on object
technology issues and includes a section
on SCRUM. There's also a good overview of Scrum practices in
the PLoPD 4 book. Scrum has a yahoo
discussion list.

Feature Driven Development

Feature Driven Development (FDD) was developed by Jeff De
Luca and long time OO guru Peter Coad. Like the other adaptive
methodologies, it focuses on short iterations that deliver tangible
functionality. In FDD's case the iterations are two weeks long.

FDD has five processes. The first three are done at the
beginning of the project.

Develop an Overall Model

Build a Features List

Plan by Feature

The last two are done within each iteration.

Design by Feature

Build by Feature

Each process is broken down into tasks and is given
verification criteria

The developers come in two kinds: class owners and chief
programmers. The chief programmer are the most experienced developers.
They are assigned features to build. However they don't build them
alone. Instead the chief programmer identifies which classes are
involved in implementing the feature and gathers their class owners
together to form a feature team for developing that feature. The chief
programmer acts as the coordinator, lead designer, and mentor while
the class owners do much of the coding of the feature.

Until recently, documentation on FDD was very skimpy.
Finally there is a full book on FDD.
Jeff De Luca, the primary inventor. now has an FDD portal with
articles, blogs, and discussion boards. The original description was
in Peter Coad et al's UML in Color
book. His company, TogetherSoft, also does consulting and
training on FDD.

DSDM (Dynamic System Development Method)

DSDM started in Britain in 1994 as a
consortium of UK companies who wanted to build on the RAD and
iterative development. Starting with 17 founders it now boasts over a
thousand members and has grown outside its British roots. Being
developed by a consortium, it has a different flavor to many of the
other agile methods. It has a full time organization supporting it
with manuals, training courses, accreditation programs, and the like.
It also carries a price tag, which has limited my investigation of the
methodology. However Jennifer Stapleton has written a book which gives an overview of the
methodology.

Using the method begins with a feasibility and a business
study. The feasibility study considers whether DSDM is appropriate to
the project at hand. The business study is a short series of workshops
to understand the business area where the development takes place. It
also comes up with outline system architectures and project plan.

The rest of the process forms three interwoven cycles : the
functional model cycle produces analysis documentation and prototypes,
the design and build cycle engineers the system for operational use,
and the implementation cycle handles the deployment to operational
use.

DSDM has underlying principles that include active user
interaction, frequent deliveries, empowered teams, testing throughout
the cycle. Like other agile methods they use short timeboxed cycles of
between two and six weeks. There's an emphasis on high quality and
adaptivity towards changing requirements.

I haven't seen much evidence of its use outside the UK, but
DSDM is notable for having much of the infrastructure of more mature
traditional methodologies, while following the principles of the agile
methods approach. There does seem to be a question on whether its
materials encourage more of a process-orientation and more ceremony
than I would like.

The Manifesto for Agile Software Development

With so much similarity between these methods, there was a
fair bit of interest in some form of collaborative work. As such
representatives from each of these methodologies were invited to a two
day workshop at Snowbird Utah in February 2001. I tagged along,
without much expectations. After all when you put a bunch of
methodologists in room, civility is usually the best you can hope for.

As it turned out I was surprised. Everyone was conscious of
the fact that there was a lot of common ground, and this recognition
was much greater than the differences between the processes. So as
well as some useful contact making between the process leaders, there
was also the idea to issue a joint statement - a call to arms in favor
of more agile software processes. (We also agreed to use the term
"agile" to refer to our common ideas.)

The manifesto was just that, a publication which acted as a
rallying point for those who shared these basic ideas. One of
the fruits of the effort was to create a longer lived body, the
Agile Alliance. The Agile
Alliance is a non-profit organization that seeks to promote
knowledge and discussion of all the agile methods. Many of the
agilist leaders that I've mentioned here are members and leaders
of the Agile Alliance.

Context Driven Testing

From the beginning it's been software developers who have been
driving the agile community. However many other people are involved in
software development and are affected by this new movement. One
obvious such group is testers, who often live in a world very much
contained by waterfall thinking. With common guidelines that state
that the role of testing is to ensure conformance of software to
up-front written specifications, the role of testers in an agile
world is far from clear.

As it turns out, several people in the testing community have
been questioning much of mainstream testing thinking for quite a
while. This has led to a group known as context-driven testing. The
best description of this is the book Lessons Learned in Software Testing. This
community is also very active on the web, take a look at sites hosted
by Brian Marick (one of the authors
of the agile manifesto), Brett
Pettichord, James Bach,
and Cem Kaner.

Is RUP an agile method?

Whenever we start discussing methods in the OO arena, we
inevitably come up with the role of the Rational
Unified Process. The Unified Process was developed by Philippe
Kruchten, Ivar Jacobson and others at Rational as the process
complement to the UML. RUP is a process framework and as such can
accommodate a wide variety of processes. Indeed this is my main
criticism of RUP - since it can be anything it ends up being nothing.
I prefer a process that tells you what to do rather than provide
endless options.

As a result of this process framework mentality, RUP can be
used in a very traditional waterfall style or in an agile manner. So
as a result you can use RUP as a agile process, or as a heavyweight
process - it all depends on how you tailor it in your environment.

Craig Larman is a strong proponent of using the RUP in a
agile manner. His excellent introductory book
on OO development contains a process that's very much based on
his light RUP thinking. His view is that much of the recent push to
agile methods is nothing more than accepting mainstream OO development
that's been captured as RUP. One of the things that Craig does is
spend the first two or three days of a month long iteration with the
whole team using the UML to outline the design of the work to be done
during the iteration. This is not a blueprint that can't be deviated
from, but rather a sketch that gives people a perspective on how
things can be done over the iteration.

Another tack at agile RUP is Robert Martin's dX process. The dx process is a fully
compliant instance of RUP, that just happens to be identical to XP
(turn dX upside down to see the joke). dX is designed for folks that
have to use RUP, but want to use XP. As such it is both XP and RUP and
thus a good example of the agile use of RUP.

For me, one of the key things that needs to happen with RUP
is that the leaders of RUP in the industry need to emphasize their
approach to software development. More than once I have heard people
using RUP who are using a waterfall style development process. Due to
my contacts in the industry I know that Philippe Kruchten and his team
are firm believers in iterative development. Clarifying these
principles and encouraging agile instances of RUP such as Craig's and
Robert's work will have an important effect.

Other Sources

There are a number of other papers and discussions about this
theme of agile methods. While these may not be full methodologies,
they do offer insights into this growing field.

The Pattern Language of
Programming conferences has often contained material that touches
on this subject, if only because many of the folks interested in
patterns are also interested in more adaptive and humane methods. A
leading early paper was Jim Coplien's paper at PLoP1. Ward Cunningham's Episodes pattern
language appeared in PLoP2. Jim Coplein now
hosts the OrgPatterns site, a wiki
which collects together patterns for organizational patterns.

Should you go agile?

Using a agile method is not for everyone. There are a number of
things to bear in mind if you decide to follow this path. However I
certainly believe that these new methodologies are widely applicable
and should be used by more people than currently consider them.

In today's environment, the most common methodology is code and
fix. Applying more discipline than chaos will almost certainly help,
and the agile approach has the advantage that it is much less of a
step than using a heavyweight method. Here much of the advantage of
the agile methods is indeed their light weight. Simpler processes are
more likely to be followed when you are used to no process at all.

One of the biggest limitations to these new methodologies is
how they handle larger teams. Like many new approaches they tend to be
used first on a small scale rather than a larger scale. Also often
they've been created with an emphasis on small teams. Extreme
Programming explicitly says that it is designed for teams of up to
around twenty people. It's worth remembering that many software teams
could be reduced in size without reducing the overall productivity.

Other agile approaches are intended for larger sized teams. FDD
was originally designed around a fifty person project. ThoughtWorks
has used XP influenced projects with a team of around a 100 in three
continents. Scrum's been used to handle similar sized product teams.

Hopefully one message that's clear from this article is that
adaptive approaches are good when your requirements are uncertain or
volatile. If you don't have stable requirements, then you aren't in
the position to have a stable design and follow a planned process. In
these situations an adaptive process may be less comfortable, but it
will be more effective. Often the biggest barrier here is the
customer. In my view it's important for the customer to understand
that following a predictive process when requirements change is risky
to them just as much as it is to development.

If you are going to take the adaptive route, you need to trust
your developers and involve them in the decision. Adaptive processes
rely on you trusting your developers, so if you consider your
developers to be of low quality and motivation then you should use a
predictive approach.

So to summarize. The following factors suggest an adaptive
process

Uncertain or volatile requirements

Responsible and motivated developers

Customer who understands and will get involved.

These factors suggest a predictive process

A team of over a hundred

Fixed price, or more correctly a fixed scope,
contract

Acknowledgments

I've taken lots of ideas from people for this paper, more than
I could possibly list. For concrete suggestions I'd like to thank Marc
Balcer, Kent Beck, Alistair Cockburn, Ward Cunningham, Bill Kimmel,
and Frank Westphal.

Do remember that this is an evolving web paper and likely to
change whenever I have the inclination. I'll add a record of
significant changes, however minor changes will occur without comment.

Revision History

Here's a list of the major updates to this paper

April 2003: Revised several sections. Added section
on difficulty of measurement and context driven testing.

June 2002: Updated references

November 2001: Updated some recent references

March 2001: Updated to reflect the appearance of the Agile Alliance

November 2000: Updated section on ASD and added sections on DSDM and RUP

December 2000: Abridged version published in Software Development magazine under the title of "Put Your Process on a Diet"