Notes on Agile/Scrum

(These are my notes though they're writting in an inhabitually pedantic
style here. I'm writing as if to teach the reader what Agile/Scrum is.
More simply "notish" stuff comes along the deeper you read into this page.)

The point of Agile is to release early and release often. It is the opposite
of waterfall development which spends a great deal of time in design,
proof, planning and execution only to end up costly, missing a target that
sailed long before release and being a waste of everyone's effort.

The rationale for Agile is to release something, however small, immediately
based on the customer's wishes, something that all parties can see as either
beginning to meet the needs and requirements or not. Another way of saying
this is: fail early, fail often, correct immediately.

In order to do Agile well, you have to have teams, you have to have backlog,
and you have to have the ability to produce a working tested increment of
product at the end of each iteration.

Scrum is the best of Agile.

Scrum and the pigs and chickens metaphor

This famous cartoon explains breakfast pigs and chickens in the Scrum process. It's
a way to discern between roles in the Scrum/Agile world. Roles are key in Agile.
They are constant and form the basis of every decision and define the very basic
human relationships in Scrum.

For example, chickens have input on the product that will be developed, but pigs
decide how it's going to be done and the rate at which it can be accomplished.
Don't be put off by chicken being demeaning. The customer, vendor, executives
and other important people with vision are chickens in Scrum.

Formally speaking, the pigs are developers including development managers, the
produt owner/manager, who represents the chickens, and the Scrummaster, a sort
of master of ceremonies especially during Scrum stand-ups.

There is a rule that during stand-ups, pigs can talk, but not chickens. If chickens
wish to talk, they must wait for a "meeting after the meeting." There is no regularly
scheduled meeting after the meeting and, if institutionalized, it destroys the
effectiveness of the stand-up in not wasting time or taking the pigs away from
what they do best.

Who are pigs? The developers, their manager and the product manager (when he
shows up, but he's especially important during sprint planning. Chickens are
other stakeholders like customers, vendors, dotted-line managers, maybe interested
parties like executives. But, chickens are NOT allowed to talk!

Correspondingly, however, pigs are required to talk. They must attend every
stand-up.

Scrum is so much more

The principles of Scrum/Agile cannot be studied casually (like reading the few
notes on this page).

The Scrum stand-up

This is the most important meeting to pigs and to the essence of Scrum.

Pigs must attend stand-up. Chickens may attend, if they have enough discipline
to keep their mouths shut. In some organizations who pray at the Scrum altar, there
may be fines for pigs missing meetings or chickens opening their mouths.

What else?

The stand-up is all about accomplishing the meeting's objectives. These include
sync'ing up, making sure there's no disaster in the making and noting obstacles.

The stand-up answers three questions. Each pig states...

what he or she did since the last stand-up, ("Yesterday, I...")

what he or she expects to work on before the next stand-up ("Today, I...")

and whether or not there are any impediments. ("No blockers" or "I'm blocked on...")

These are absolutely, depending on how fastidious the organization is, the only
sounds that go on in the stand-up proper. In strict Scrum, nothing else may be
said during stand-up.

The stand-up happens in the same way, every day, in the same place, at the same
time, on time. All other meetings are subjugated to the Scrum stand-up in
importance.

The name implies that the meeting is so short and so succinct that a team need
not even sit down. Some (those fastidious Scrummers) even impose fines for sitting
down.

Other Scrum meetings and practices

In Scrum there are these meetings:

Stand-up

Release planning

Sprint planning

Sprint retrospective

(The stand-up is covered elsewhere.)

Release planning (not really Scrum)

Release planning is conducted by the product owner/manager. He maintains the
back-log, a list of epics and stories that will accomplish what's
needed to produce the product. Often, only team leads and development managers
attend.

Release planning is aberrant to Scrum, but a crutch in some organizations for
product planning. It has to be done sometime by someone whether or not it's
institutionalized and given this name.

Typically, the output from release planning is a list of epics or stories for
the product owner's back-log. Some degree of estimation, requiring cooperation
from pigs, can be done, but there is never any break-down deeper than the epic,
never deeper than the story level. Formal epics and stories need not come out
of it, but these must obviously be in place for a back-log to exist in time
for spring planning.

Release planning, because it's not Scrum, is a good opportunity for chickens
to get their licks in. They insist that the product owner see product
requirements their way and be able to plan accordingly.

Release planning happens ahead of beginning any sprints to work on the features
planned. This would not be every iteration, but more on a quarterly or yearly
basis depending on the size and scope of the product. (This is why it's not an
Agile concept.)

The product owner

In all of this planning, the product owner is the key not only to planning, but
to the ultimate success of the product. The product owner creates and maintains
the back-log, prioritizes it based on customer and management needs, and actively
promotes its accomplishment. His job isn't done once the back-log is full and
the sprint planning underway. He gives vision and order to the team. The
absence of a good product owner can literally destroy a products chances to see
the light of day. In many cases, the product owner is really a very savvy
developer with a strong streak of customer- and marketing sense, or an
extremely talented customer and marketing person who totally groks the developers'
side of the equation.

Sprint (iteration) planning

Sprint planning is conducted by the product owner. In some organizations, it
is a two-step process spread across two meetings, sometimes two days. This
doesn't mean it takes that long, just that there are two phases. Obviously,
if the iteration is two weeks long, spring planning must not waste more than
a few hours of it.

First, what's to be done in the sprint is estimated in terms of what can be
done based on historical team velocity. (Velocity is the observed
rate, in story points, at which a team typically works.) The availability of
the team members is scheduled (who's on vacation or out of the office, when,
etc.) and matched up against the story points to be accomplished. Stories
that can't be done are returned to the back-log for a later sprint.

The dynamic is that the product owner promotes as many stories as he can
reasonably get out of the team while the team pushes back to ensure they can
successfully complete them.

Second, team members take time personally to break down the planned stories
into tasks. A task is a subunit of work, never taking more than a few
hours (some organizations place a strict maximum like 4, 6, 8 hours, etc.) that
is required for most developers to complete. A task is not necessarily a
programming assignment. It can include research, learning some required new
technology, writing documentation, etc. Stories are broken down into 2, 4, 6
or even more tasks. Stories that cannot be decomposed into fewer than half a
dozen tasks might be epics and not stories.

When the sprint planning resumes, the stories are all broken down, ready to do
and (or) a sanity check is possible, to wit, the sprint as planned may have
been proven during break-down to be too much.

The longer an iteration is, say 30 days instead of a fortnight, the more likely
the two-part sprint planning will be needed.

Retrospective

The spring retrospective is a short meeting, usually at the end of one iteration
and prior to sprint planning, attended mandatorily by all pigs during which the
following questions are answered:

what went well?

what didn't go well?

what can we do better next time?

Bookkeeping

The foregoing makes obvious the need to remember what goes on during planning,
iterations, retrospectives, etc. Unless the number of stories successfully
completed and their points tabulated, it's impossible to gain an idea of a
team's velocity, something that takes several sprints (at least).

Obviously, knowing a team's velocity factors very usefully into estimation which
is essential to budget, resources, etc.

The results of the sprint retrospective must be noted for follow-up, usually under
the auspices of the Scrummaster.

Tools

This is where a good Agile tool comes in to play. This tool must be able to cope
with

(What Assembla does not do is most of what's above—barely tracking
the state of things with no hierarchy in the relationship between epics,
stories and tasks.)

Sprint cadence

What happens during a sprint is:

planning and buy-off,

testing underway (TDD: test first, code second!),

coding underway,

demonstration of finished tasks or stories,

the retrospective.

The pigs buy off on the sprint planning symbolically by expressing their
perception as to the likelihood of success, giving a fist of five for
full confidence, or a single (or no) finger when most pessimistic.

Then, they get to work, hold stand-ups daily. An important aspect of that work
includes test-driven development (TDD). This is an Agile concept that ensures:

unit tests for every testable unit of code,

reproducibility of coding assumptions, including notice that something's
broken,

training for those who didn't write the code by demonstrating what it does and
how it does it.

Scrum preconizes the term finished by saying what it means. Finished
is the final state of every task and story (it is to be hoped) by the end of
the iteration. Finished is defined as

test-complete

code-complete

documentation if needed

a demonstration capability

Demonstration is most important to chickens, but it's useful to pigs as well.

To the greatest extent possible, the effect of the work must be shown in the
light of how it contributes to the product. This proves to the developer,
manager and product owner that the story is finished, it shows chickens like
the customer, vendor, salesperson, marketer, etc. the finished product will
do.

Back-end effects, like database or server work, are shown if possible by the
side effects of product (UI) use. Often there is work whose demonstration is
too arcane to be done for chickens.

Most frequently, the demo is accomplished in a great-room setting showing off
the entire team's accomplishment for the iteration.

The epic...

...is simply a huge story that can be broken down into smaller stories that,
unlike the original story, do not defy decomponsition into tasks. The
epic is a relatively new concept to Scrum.

Writing stories

In Scrum, there's a peculiar way of expressing stories. These are couched in
the following language:

"As a _______________,

I need ______________,

so that _____________!"

For example, "As an iPad user, I need a button on this application, so that
I can save my work."

It's that simple. In the context of any good tool for managing stories, one
often elaborates by talking more about the stories, its requirements,
expectations or any other important information.

Typically, this expression is not applied to tasks, which are more telegraphic,
though it can be applied to epics.

Don't estimate stories during sprint planning

...because you know too much about them as compared to what you know about
stories that aren't likely to be done immediately. Instead, estimate them at
least a couple of months ahead of time so that the estimates have the same
weight and don't falsify velocity.

"So, if I estimate or reestimate the story in sprint planning, I can't estimate
it relatively to the rest of the backlog. I don't know that level of detail
about the rest of the stories. I'm now thinking at a different level of detail,
using a different kind of thinking than when we estimated the other stories.

"We know our estimates are wrong. We knew they were wrong when we estimated in
release planning, and we know they are still wrong when we get to sprint
planning. That's why we have velocity—an empirical measure that takes all
that wrongness into account."

Agile and management

As we're trying to do a project in a Scrum way, and as Scrum/Agile isn't
reassuring to management, I realize that I'm going to have to learn the part of
the Agile story that fixes this reservation. Specifically, "Agile is fine, when
do I get what?"

In the dark ages before Agile, one made estimates and commitments. These were
never exactly met, but no one really noticed. It was waterfall. That was how
the game was played. A commitment was made, everyone knew it would be wrong,
but it was expected anyway. Maybe management handicapped the commitment,
removing scope, lowering expectations, padding the schedule. It's been the
recipe for success since the pyramids were planned.

Early estimates are wrong. Arithmetic shows they will be wrong.
Program Evaluation Review Technique (PERT)'s law of large numbers
helps aggregate (read: fudge), but it will still be wrong. (When, in my
waterfall experience, were we ever right?)

The outside demands, availability, etc. of the resources, especially
people, will change including unplanned illness, attrition, personal
level of commitment. It's people stuff. It changes.

Customer needs and priorities will change. And they don't really even know
what they need or want in the first place. That's one of the things Agile
is all about: growing with the customer understanding, reflecting
educated needs live rather than in a cycle of work, disappointment,
renegotiation, rework, etc. Agile processes are designed to help
deliver what the customer actually needs, not what he asks for.
Waterfall processes encourage one to deliver the wrong thing, come
hell or high water, as close to on-time as possible.

Under waterfall, one committed to delivering a couple of pyramids. After
spending twice the time, double the budget, the first pyramid rolls off the
line and, guess what? sphinxes are now all the rage. Oops.

Unfortunately, the Pharaoh still wants a commitment to deliver a couple of
pyramids. How to remain true to Agile (not because it's a question of religion,
but because of the outcome that makes it the True Way), but mollify management.

One way is just to give the estimate. Not, "This will take two months."
Instead, "I expect, with 95% confidence, that this will take no more than two
months."

There's an uncertainty cone that develops as a result of predicting the time it
will take to produce the range of undefined tasks and even stories combined
with not being able to predict illness, attrition, etc. or customer lunacy
(because this is Agile and you're not leaving him out, therefore, he will
change what he wants as he sees the product develop because he'll begin to
realize what it is he really wants).

Because agile, one is willing to get smarter about the product over time. (And
not just the developer either.) Therefore, the uncertainty cone should shrink
over time.

But, a commitment is still required.

Backlog!

The backlog is an ordered, prioritized list of stories (and bugs). The backlog
represents what and the order in which stories will be executed. The product
manager will refine the backlog, changing the order and break-downs as the team
(and he, and also the customer) get smarter.

So, given the backlog and given iterations, it's easier to predict how long
because we commit to how much and how much is what's in the sprint this
go-around. So, it becomes not how long, but how much, and how long is the
iteration, so how much is (this much: what's on the backlog).

What will we finish?

The objective then is to move management from asking how long to asking how
much or rather what. Commit to a subset of what you predict you can complete
(in the iteration, or in the iterations planned).

Over time and with good care (i.e.: commitment to some rules of Scrum/Agile), a
team's velocity is known. Starting out a release with a backlog of 500
points (projected), one can predict completing n points worth of
stories, e.g.: a 5-member team, velocity average of 40 points per sprint, 8
sprints in the release timeframe equals 320 points completed. Beginning at the
top of the backlog and working down, one draws a cut-line 320 points worth of
stories down the backlog: that's how much will get done. That's a
prediction.

For a commitment, however, it depends on comfort. Maybe one is
comfortable committing to one-half (the 320 points), maybe more, maybe not even
that much (especially if the early requirements—read:
stories—themselves are not convincingly quantified or followed).
Similarly, measure down the backlog to determine what can be committed to.

Over the ensuing months, however, maybe things will go better, maybe worse.
Maybe the client will switch gears after seeing what's beginning to roll off
the assembly line.* One increases or decreases his scope of commitment. You
reach the point where you can say,

"We didn't know this at the start of the release, but X is really
important to our customers. We will be able to deliver X in addition to
what we have already committed. Without slipping the release date."

Making commitments in Agile isn't impossible. It just needs approaching
differently than in waterfall. In waterfall, commitments aren't ever ultimately
worth the medium in which they're made. With time, experience and a stable team,
commitments in Agile become a great deal more believable because they're based
on reality and shorter, more quantifiable periods of work.

* In particular, examine the boss' responses at the beginning and end of
I want to run an Agile
project, but client involvement, perhaps represented by the product
manager, is crucial.

There is a problem with commitments and the need for planning an entire
quarter. This is waterfall thinking.

Commitment is the usual goal, but really, it's not what's needed. What's needed
is delivery. The more often delivery happens, the better because the more that
can be changed. Points:

Even with small stories, estimates aren't perfect. Something might
(hopefully) get finished sooner. Agile makes it so that a new story is
pulled from the backlog and worked on. That's good.

For too-large stories, under-estimation is common.

If too much changes mid-quarter, it's "broken arrow." Do you want to be
able to change and realign or just fail? Do you wait until quarter's end
to realign?

Committing to a shorter cadence permits managing these problems. A novel idea
is not to commit to anything. Instead, use continuous planning.

Do planning with small stories, short iterations. A longer roadmap is okay, but
it's a wishlist and must not be committed to because it likely won't happen.
Why commit to something that's impossible?

Commit to one iteration at a time. This encourages everyone to...

Think small, short stories, managing work in progress and maintaining
momentum.

See interdependencies: the smaller the features, the clearer these will
be.

Plan smaller thing in less time reducing also the planning load for the
iteration. A full quarter or release planning is a huge investment in
time for something that, as noted, likely will not happen that way and
gradual replannings will take place anyway. So just count on doing the
"gradual" plannings.

Use deliverable planning (a list of features) in a rolling wave,
replanning as needed and as teams deliver.

Small planning means not gathering everyone together for long hours of
pointless, time-wasting meetings. Small planning means involving those you must
when you must, different ones at different stages.

Do a short kick-off with everyone, to be sure, so they all know each other and
feel like a big team. This is, however, different from a big planning session
where people are planning instead of delivering.

Scaled Agile Framework (SAFe)

I found this interesting in a heads-up sort of way. Most organizations suck at
implementing Agile because of a) over-willingness to get religious and build
empires and b) valuing tools and process over human intelligence and
interaction because they think they can control the former while the latter is
just herding cats. (Well, okay, it is herding cats, but tools and systems don't
build software, people do.)

I think SAFe is onto something though I'm no kind of crusader, just an
interested by-stander. This video introduction to SAFe is short and sweet. In
the first couple of minutes, it does a remarkable job of illustrating the
essence of Scrum, something people who think they know Scrum, but never did and
bastardized it in order to build their own religious empire around it, should
watch because they need to learn or be reminded of what Scrum really is.

"I want to run an Agile project." This will be funnier to watch if a)
you know yourself how agile works and b) you've experienced the
frustration of no-vision, un-agile organizations, processes and people. The
most hilarious conversation is the one with the architects. The
conversation about pairing is narrowly Agile XP and not a particularly
valid point in greater Agile.