Five tips for Agile measurement

Takeaway: Managers
and executives need a way to measure Agile development so they can make
informed decisions. Now there is finally some good guidance on how to
effectively measure Agile development.

Agile is here to stay. Once the radical
alternative to Waterfall development methods, these legacy methodologies
are being disrupted and replaced by Agile practices that improve
time-to-market, reduce development costs, and produce higher quality
software that better meets customer expectations. As the world demands
more software, development teams — from scrappy startups to big
corporations — are meeting the challenge with Agile.
But while Agile software development projects scale across the
enterprise, management is still searching for the best way to gain
deeper visibility into these projects. Large organizations cannot rely
on the subjective anecdotes of those closest to the work; they require
quantitative insight upon which to base business decisions.
Here are some quick tips to quantify the impact of the choices you make during and after your Agile transformation.

1. Start with desired outcomes, NOT what’s easy to measure.

Better measurement leads to better insights, which in turn leads to
better decisions and eventually better outcomes. Most people start by
measuring what’s easy. But measuring what’s easy can drive the wrong
behavior. Let’s take a look at this story about two NBA players.
In 2010, Monta Ellis, with the Golden State Warriors, was the 9th
highest scorer in the NBA. Carmelo Anthony, with the Denver Nuggets, was
the 8th highest scorer. Measuring individual scoring totals is easy.
You would assume that because they were prolific scorers, their teams
would win.
However, when they were in the game, their teams won less.
Scoring is a function of two other measures: 1) the number of shots and
2) the percentage of those shots that go in the basket. It turns out,
these two “stars” have low measures for #2, their shooting percentage.
The only reason they are high scorers is because they take more shots. They are literally stealing shots from their teammates who might have a better chance of scoring.
So, while the flow of learning goes from measures to outcomes, the
way we think about it should start with outcomes. That’s why we call
this ODIM.
better OUTCOMES ← better DECISIONS ← better INSIGHTS ← better MEASURES
The NBA players should focus on the outcome of winning more games
rather than being high scorer. If they used the overall likelihood of
the team scoring under various conditions as feedback, it would help
them make better game-time decisions to achieve the ultimate outcome of
winning. This brings us to our second tip.

2. Think of measurement as feedback, NOT levers.

Frequent feedback is the primary difference between Waterfall and
Agile development. Successful Agile projects incorporate short
iterations with fast feedback from customers. The key to effective Agile
measurement is to think of measurement in terms of feedback, not as the
traditional lever to motivate behavior. This often devolves into
keeping score, which is where the dark side of measurement starts —
avoid it.
There is a subtle, but important, distinction between “feedback” and
“lever.” Feedback is something you seek to improve your own performance.
Levers are used to influence others. The difference is more in how you
use the measure than the measure itself.
For example, healthy use of a burndown chart tells the team if they
are on track with their commitment so they can make adjustments in time.
The counterexample is a manager using burndown charts to red-flag
projects in trouble. While it may drive improvement, nobody wants the
red flag thrown at them, so the tendency is to keep the metric in the
green regardless of the reality of the situation.
You can’t make better informed decisions if the metrics you are using
to gain insight don’t accurately represent reality (see tip #1).
Rather, the manager could provide coaching on tools that the team can
use to improve their own performance — a subtle but critical difference.

3. A balanced measurement regime or none at all.

Balance in Agile measurement (Figure A) includes four cornerstones:

Do it fast.

Do it right.

Do it on time.

Keep doing it.

Figure A
Without balance of these four elements, it’s easy to focus on just one. For example, if we focus only on increasing productivity this will likely drive down quality and customer satisfaction.

4. Measure specific outcomes for software.

Productivity

Responsiveness

Quality

Customer Satisfaction

Predictability

Employee Engagement

These six outcomes are the elements of the Software Development Performance Index (SDPI),
used to quantify insights about development work and provide feedback
on how process and technology decisions impact the development team’s
performance. Know what to measure and focus on each individual element.

5. Listen to experts.

Agile has caught the attention of leading industry analysts,
asserting itself as a key part of application lifecycle management (ALM)
evaluations. These evaluations encompass more than just functionality
for developers; they assess commitment to the ALM market, ALM product
strategy, corporate strategy, and market presence. In fact, independent
research firm Forrester Research, Inc. recently evaluated
the most significant ALM software providers, treating Agile and Lean as
critical tests of an ALM vendor’s offering. The report also found that
businesses “can no longer accept historical gulfs between business and
application development and delivery teams as, increasingly, firms now
expect to manage application development and delivery as a business and
treat it as a competency.”

The Agile perspective for software development metrics

In the move to Agile, overall goals are largely the same as before:
to delight users with a quality product delivered in a predictable and
efficient manner. Even after your Agile transformation, you will largely
do the same “types” of things: analyze, design, code, test, release,
maintain, and, yes, measure.
It’s the perspective you take when doing these that is different with Agile.