(Reposted from Agile Summer Camp. The team will edit and improve that version, filling in the gaps in my memory and understanding. This is my rough draft.)

Organized by Chris Woodruff with the able assistance of Josh Holmes and Michael Eaton, Agile Summer Camp 2008 was a fantastic success. This is an Agile Summer Camp Diary, documenting bits and pieces of a fun, rich, informative weekend with a crowd of unwashed geeks. No text page (nor even sadukie'sgreat pictures) can capture the full experience of what you missed. You had to be there. Agile is all about the conversation.

Style Note: A common principle in many Agile processes is collective responsibility and collective ownership of the code and the message. That leads to a team that collectively commits to their results. In keeping with that principle, this diary emphasizes what we asked and discussed, more than what individuals contributed. Individual comments will be minimal, and "our" comments will dominate. Partly that's because my flawed memory can't always put a name to each line in my notes; but mostly it's because this was our conversation, not individuals dictating to listeners. There were actually very few points where we disagreed, and everyone learned from everyone. Sometimes this may result in statements written in an annoying passive voice. Sorry.

Introduction

Matt Werstler has the list of us who actually made it to the Camp. The Who's Coming list covers most of us, though some couldn't make it and some never got the chance to sign up but joined us anyway:

Josh Holmes

Chris Woodruff

Michael Eaton

Michael Letterle

Corey Haines

Sarah Dutkiewicz

John Hopkins

Martin L. Shoemaker

Gayle Craig

James Bender

Brandon Zylstra

Brandon Joyce

Matthew Werstler

Aaron Worsham

Duane Collicott

Steve Andrews

Lee signed up at the last minute, but I never caught her full name. She's a Project Manager from Proquest, and made an awesome contribution. We needed someone with a PM perspective. Dianne Marsh joined us briefly, with a little one in tow. Tim O'Connell and Dave Giard also joined us for part of the day.

Friday, Sept. 5: Arrival

With people arriving at all times through the night and most weary from the road, we chose not to try to organize any formal talks on Friday night. Numerous informal Agile discussions broke out, along with discussions on other topics, lots of good food, fire, socializing, drinking, music, and a great time.

Saturday, Sept. 7: Open Day

Agile Summer Camp was an Open Spaces event. As Josh explained, that meant impromptu but organized talks on topics selected by the attendees, as well as a small set of rules for keeping the talks relevant. Among these rules are:

Whoever attends are the right people. The attendees are the right people to discuss what interests the attendees.

Whatever is discussed is the right discussion.

The Law of Two Feet: if you're not interested in a topic, leave that area and find an interesting discussion. This is your time, and you're not obligated to waste it. You are obligated not to start a side topic in the same space, disrupting those who are interested.

So we nominated topics, voted, and began our discussions. The topics we selected were:

Agile 101

Given the wide range of Agile experience at the camp, it was good to start with an introduction to what Agile Development is (and isn't). We discussed the origins of Agile: how projects run according to the traditional Waterfall model had a high failure rate often related to the implementation of Waterfall itself; and so there was a push for more agile processes which would better fit for how software development really happens. At the same time, we emphasized how the idea that Agile "competes" with Waterfall can be misleading, and often leads to religious wars rather than working code. Agile is about a fix for chaos, and in fact can have a lot in common with Waterfall.

So if Agile isn't a competitor to Waterfall, what is it? It starts with a philosophy, as captured in the Agile Manifesto. We summarized the Manifesto at the Camp; but for copyright reasons, we include it in full here:____________________

We discussed the importance of "over" in the Manifesto. For instance, "working software over documentation" doesn't say that documentation is bad; but it recognizes that working software can be the best documentation, because it has concrete meaning to users.

We returned to the idea that Agile is an alternative to Waterfall, and how some Agile proponents will make bold claims like "Agile never works." Given that there are plenty of case studies of Waterfall projects that have worked, that approach divides the community and can give Agile a bad name. Agile simply shifts our focus to processes that emphasize the importance of people and the acceptance of change. An Agile process should be regularly reevaluated and updated to use more of what works and less of what doesn't. "Waterfall never works" is the sort of statement that arises from an emotional attachment, not a rational evaluation. Because developers are people, we appreciate developers who commit to the effort; but there's a line between having a stake in the effort and having an emotional attachment. The emotional attachment can blind us, emphasizing "our" process over our goal of processes that work. There's no one-size-fits-all process.

At the same time, most of us wouldn't have been at Agile Summer Camp - company and great weather notwithstanding - if we didn't see value in Agile Processes, value that we haven't always found in other processes. As Corey said, "Agile made it so I could stop lying." He didn't have faith in most of the non-Agile practices he had followed in his jobs, but the jobs required him to treat those processes as correct and predictive. With Agile practices, he believes; and the practices (if followed) won't let him make misleading statements, because those run counter to the processes themselves.

Next we discussed two examples of Agile processes, as we knew them from practicing them:____________________

Extreme Programming (XP)

This process includes (at least) the following elements:

Short iterations, usually 2 weeks.

Requirements defined as user stories, short snippets of work that a user needs. Stories are commonly written on index cards, encouraging the team to keep them brief. Emphasize "naked stories": small, simple stories, without implementation details.

Planning meetings where the stories are matched to a timeline based on estimated complexity. The emphasis is not on "What do we want to do by this date?", but on "Can we do this in this time frame?" or "What can we do in this time frame?"

Prior to the planning meeting, a complexity meeting, where the team "scores" each story for complexity. One approach is to assign points to each story - say 1, 3, or 5 - and write those on the cards.

In planning, insist that customers understand they can own two of the Three Pillars (a.k.a Iron Triangle): Time, Money, and Features (sometimes also referred to as Quality). If the client demands control of all three, Agile processes will fail. So will non-Agile processes. So will anything you can try.

Pair programming.

Test Driven Development.

Consistency in the process. Your goal is velocity: a measurable (and, you hope, increasing) rate of feature delivery.

A Daily Stand-Up meeting to identify what each person is doing and what's blocking them from doing it. The Daily Stand-Up is supposed to be very brief. (Hence the Stand-Up aspect: no one wants to stand for a long meeting.) Explain just what's going to happen, and what may be a problem. Take problem-solving off-line.

Continuous Integration. The entire project is built and tested from checked-in source frequently (preferably daily or even more frequently). Any build problems are found and solved as quickly as possible.

On-site client. Because XP emphasizes constant communication so that problems are found and fixed early, XP requires a knowledgeable customer representative to work with the team all day, every day. It's often hard to get a client to commit to this, but it has significant impact on the success of the project.

____________________

Scrum

This process shares many practices with XP (and vice versa), especially short iterations, the Daily Stand-Up, and user stories. Where XP is more developer focused with some emphasis on project management, Scrum is the reverse: largely about the role of management in enabling a team to solve problems. We spent some time discussing ScrumWorks, a particular Scrum tool that Lee uses in her work; but in keeping with the Agile value of people over tools, we emphasized Scrum as a whole, not just as one tool.

While Scrum defines many different roles in the process, we looked specifically at two roles:

Scrum-Master. This is an observer who doesn't participate directly in the process, but who rather measures progress and looks for problems and ways to improve.

Coach. This is an experienced Scrum practitioner who helps team members to learn how to use Scrum better.

____________________

We finished with further discussion of estimating in Agile projects.

One-Man Agile Development

Our concern in this discussion was simple: since so much of Agile is about communication within teams, how much of it applies to a one-man (or one-woman) development shop? Of course, Agile also involves customer communications. Those aspects apply no matter what your team size. But a related concern is Agile in a non-collocated situation: how do you manage meetings if you work out of your home office, not on the client's site? We discussed tools and practices for these, including "virtual" Stand-Ups with clients. Stand-Up meetings serve two goals: to provide visible status for the project, and to resolve roadblocks. A one-man team will have to resolve most roadblocks for himself, so Stand-Ups don't help there; but they still help to let a customer know your status. Customer Stand-Ups will perhaps be less frequent than Daily Stand-Ups, but shouldn't be ignored.

We also discussed how we can work with customers remotely to identify user stories, and use those to reduce complexity. Michael said he uses online bug-tracking software (modified a bit) to let his users create and file stories online; and then he reviews the stories with them to ensure he understands them correctly.

We diverged a bit from "Can this be done, and how?" to discuss the trust relationship we need with a client for this to work. They have to trust that we're working in an efficient and productive fashion. We have to trust that they will fulfill their responsibilities in defining requirements and priorities. If this trust is absent, it can doom a project, no matter the size or the process.

We also discussed how - IP constraints permitting - one-man shops might team for brief bursts of paired coding or mutual Stand-Up meetings to help each other identify problems we might help each other with. Josh is investigating the possibility of a "Community Stand-Up" conference call.

Complexity in Agile Projects

This discussion revolved around a complex and sometimes sensitive subject: what is Agile, anyway? Even though we were mostly in agreement with our Agile 101 definitions, some in the development industry can appear to think Agile means using specific languages and tools and processes. This impression is usually mistaken, due to reasons Cockburn explains in his discussion of the Three Levels of Listening: listeners who are new to a topic can hear black-and-white distinctions when experts discuss shades of grey. But the mistaken impressions can lead those new listeners into black-and-white habits that may lead them to think Agile has failed when they really haven't been Agile at all. (Some in the industry completely miss the point, thinking Agile just means, "Start writing code right away, and lots, and fast, with no time wasted on thinking!" We discussed that in our next session.)

And as one example of a statement that can mislead new listeners, some Agile proponents will insist at the start of a project - even before they know requirements - that they'll need certain tools and frameworks, regardless of the complexity those might add. When others ask how they know they'll need those tools, their answer is, "My experience says we always do." To a new listener, that can sound awfully absolute. We saw some irony in that rationale, because it directly contradicts the often-stated Agile mantra, YAGNI: You Ain't Gonna Need It. "Because I said so" or "I know best" thinking is precisely one of the habits that Agile was designed to counter. Until you see the tool need arise from the user requirements, YAGNI.

At the same time, we recognize the grey that the new listener might miss, and we understand the speaker's true meaning. There are a lot of reasons why you may lean toward a particular tool or framework. You may have strong intuition that says you'll need it. Or it may be a tool you know well and can comfortably work with quickly and efficiently. There's business value in familiarity if familiarity leads to speed or quality. Those might be reasons for exploring user stories and looking for the reasons you need the tools.

But the emphasis should be on business value, not "I just know". We discussed a Steve McConnell book where he described how each team member brings to the project a set of personal values for what constitutes "best" coding practices. Some may prefer performance, some brevity, some familiarity, some clarity, some maintainability, and more. McConnell says that it's important to bring these values to the table at the very start of a project, discuss them, and reach agreement on what are the best values for this project. If the deadline is very close, then maintainability and clarity and reuse may be less important than speed and familiarity. If the deadline is distant but cost is a concern, emphasizing clarity and maintainability can keep total costs lower. Each team member has to agree with the values for this project, even if those don't align with his or her personal values.

When people advocate for a given tool, that's usually because it strongly supports their development values; but again, those values may not fit the needs of the project. That contradicts the Manifesto. When new listeners hear something is "the one true answer", that isn't Agile; it's the sort of proscribed decision-making that Agile is supposed to solve.

Of course, just because new listeners might misinterpret is no reason to deprecate the experience of people who know and use good tools. The experts' experience means there's a strong chance that their initial instincts are correct. But there's a line between "probably" and "I just know," a line that new listeners might miss. We should prepare to use the best tools we know, but not pre-decide that we will need those tools.

We also discussed how important this issue is to the community as a whole. Do we have a responsibility to say "Yes, but..." in community settings when that may derail a discussion of good technologies? Again, Cockburn points out how that sort of "Yes, but..." discussion can be just as damaging to a new listener, leaving them with so many paths that there's no clear path at all. Until they gain their own experience, too many choices can be worse than no choice at all. Ultimately, we decided that people who are strong advocates of good tools have strong reasons for being advocates, and the community needs to hear what those are. Our responsibility to the community is not to disrupt that message with "Yes, but...", because the proponents are adding value to the community by educating people on the power of those tools. Rather, our responsibility is to do the exact same thing: be strong proponents of Agile and its emphasis on alternative approaches, with strong reasons why we see benefit. Don't muddy the message, add to it. And avoid any black-and-white, right-and-wrong discussions, because Agile tells us there's no right and wrong without context. Let the community hear all positions and decide for themselves. That's fundamental to Agile.

Selling Agile to Clients and Managers

This session started as How to Sell Agile to Clients, and quickly expanded to include managers. We found a wide range of experience in whether to sell Agile from top down or bottom up. As with all things Agile, either can work, and either can fail; it all depends on context. We also expanded to a related concern: how to sell what Agile really is, when people expect it means, "Don't stop to think! Just codecodeCODE!!!"

And that led to a major point of misconception: people think Agile Development means more productivity. It really doesn't (yet at the same time, it does). Agile Development means more quality through better requirements definition and better quality control, along with a sustainable and predictable and visible pace of development. Rework due to quality failures and requirements failures causes massive delays in software development; so better quality results in shorter schedules and better productivity over the true life of the project. And the visible pace of development means managers and customers will be less surprised by what they get and when they get it.
So it's important in selling Agile to emphasize what they won't get, and what they will get instead. They won't get lots of code quickly; but they will get some code sooner than they would with a pure Waterfall approach, and more high-quality code on a regular basis. They won't get everything they ask for on a predicted date at a predicted cost; but they will get a maximal amount of working, high-quality code on a fixed date; or they will get everything they ask for in the shortest reasonable schedule; or they will get the largest volume of high-quality code on a fixed date that their money will buy.

Another part of selling reluctant stakeholders is to emphasize again that Agile is not a "competitor" to Waterfall. In fact, Agile is very close to Waterfall, and some Agile processes are virtually identical to Waterfall. What changes primarily are the cycle times. Instead of doing a single long Waterfall cycle or even multiple long cycles, you do a much greater number of much shorter cycles. The software development process can be opqaue even to the developers on the team; and the opacity is far greater for managers and customers (as Spolsky discussed in The Iceberg Secret, Revealed). Short cycles increase visibility and thus give managers and customers better feedback and better control. The Agile approach is all about finding and improving requirements as early as possible, and finding defects as soon as possible, all of which help us to mitigate risk.

We emphasized that it's important to realize that the Agile message - including the message we use to sell Agile - must vary with the audience. Just as no one process fits all, no one sales strategy fits all. We also need to recognize that any process will likely have Champions and Enemies. Agile will appeal to Champions because they see the value in increased visibility and communication, while Enemies may see something unfamiliar or risky. Or Enemies may be experienced managers or team members who have lived through numerous "process improvement" efforts that never really improved any processes (and may have made things worse), leaving them justifiably cynical.

But in either case - even in the case of a Champion eager to say "yes" - your sales strategy should include educating them on the process. With Champions, you need to be sure they know they're getting visibility and feedback, not "lots of code really fast". With Enemies, you have to work harder, providing more data and explaining the risk mitigation benefits and showing industry data on the causes of project failures. And for both - and for the team as a whole - you need to educate them on their responsibilities in the process, and the critical importance of meeting those responsibilities. The leading cause of process failure is failure to follow the process. Cynical Enemies have reasons to be cynical, and you should acknowledge those reasons. You should encourage them to point out process failures - actual and potential - because Agile processes are supposed to constantly improve.

We discussed what to do when management or customers or team members just won't agree to Agile practices. One answer is to Go Guerilla or Stone Soup: bring in Agile practices anyway, a little at a time, and gently argue how they actually satisfy the requirements of your current processes. This isn't guaranteed to work by any means, but can often be a way to open people's minds to the possibilities of Agile Processes.

We emphasized the need to get management and customer involvement in some parts of the Agile Process. We also discussed how critical it is to get them to involve developers more. In particular, we discussed how requirements and estimates formed without strong developer input will usually fail.

We discussed the two biggest risks to Agile processes after everyone is sold on them: contractual obligations that no process can possibly meet, and scope creep. Contractual obligations, we agreed, are usually bad (there's a surprise) but usually unavoidable. We discussed these further during our final session on Planning with Deadlines. As for scope creep, we discussed how in Agile, there really is no such thing. The scope is whatever the client says is most important for the next interval, constrained by how much the team can actually accomplish in that time frame. In a side conversation, Corey described a project where his management came to him with a "crisis": system updates outside of their control were crashing the whole system. This had to be solved ASAP! On a more rigid project, they knew, this would cause the programmers to start complaining about feature creep, unexpected features, etc. But because Corey's team was Agile, they didn't complain at all. They said, "The next iteration starts in two days, and we're planning it now. Make this your number 1 priority, while we estimate the size." It ended up a size that fit within one iteration, and they released it without a hiccup, and without deviating from their process. Their process was designed, as Beck wrote, to Embrace Change, not control it.

That led again to a discussion of the Three Pillars, which led again to a discussion of how to manage expectations and secure customer involvement. Attendees emphasized that question: "What if customers just won't give us the feedback we need?" Martin gave his own personal answer (while acknowledging not everyone would be comfortable with it): intentionally look stupid. Find an area of confusion, flesh it out in the least likely way that's almost guaranteed to be wrong, and bring that to the customer. When the customer asks how you could be so stupid, politely point out all the times you needed clarification but didn't get it, and how that lack of clarification left you with the wrong answer. And then pull out and show other possible answers, ones you think are close to correct, and say, "I really need your help to get this right. Please help me." As Josh said, fail early. (That's a common principle in Agile processes as well as other successful processes: make and find your mistakes early, when it's easier to correct them.) But we agreed: if you're not getting the help you need, don't pull out a contract and demand they comply. Remember the Agile value of collaboration over contract. As Mike said multiple times (or maybe it was Corey?), when you have to refer to the contract, you've failed.

We briefly discussed the value of pair programming, along with the perceived value: managers and accountants may see it as "two programmers doing the job of one", and see it as wasteful. We discussed actual benefits we have experienced with paired programming, and argued that it's two programmers doing the job of two or even three, because code gets tested and reviewed more thoroughly. Pairing also improves communications and spreads cultural knowledge withing the team. Still, we agreed it might be better to split the pair at times, such as having one of the pair write tests while the other writes code to meet those tests.

We discussed perceived complexity vs. actual complexity. At least, that's what my notes say. I can't remember a single thing about this topic.

We discussed the danger of a Quiet Client, both in selling Agile practices and in running them. A client may be quiet because he's just not talkative; but he may be angry, or confused, or tired, or in disagreement, or distracted, and just not willing to speak out. Until he explains to you why he agrees with you, assume that he doesn't. Until then, your job is to probe - gently and politely - for why he's not speaking up. A Quiet Client == Trouble, or usually so.
We also discussed how selling Agile is easier if you have become a Trusted Advisor to the client. James (I think it was James?) brought up this theme multiple times over the weekend: if you have a strong trust relationship even outside the bounds of your current work, you'll have better and more productive conversations with your client. The client will let down walls of suspicion and reticence, and really tell you what he needs. Michael described his relationship with a long-standing client, and James held it up as an excellent example of the success that comes from being a Trusted Advisor.

Finally, we discussed worst-case scenarios. One is not really a worst-case, just pragmatic reality: Agile processes don't work for every project. We need to be aware of that and responsibly evaluate each project. If Agile isn't right for a given project, then the responsible thing to do is to not sell it, but to sell the best process instead.

And the other worst case is when Agile is right, and even the only way likely to succeed in your professional opinion, but the client can't be sold on it. In that case, we argued, walk away. Don't commit yourself to efforts you believe will fail: that's not really a commitment, and false commitment will only doom the project further. Explain your concerns, explain why you just can't commit to a project you don't believe in, and walk away. But then, in keeping with the Trusted Advisor theme, keep up contacts with the client. Don't press for status on the project, and certainly don't point out failures if you know they're happening. The client knows if he's failing or not, and any comment from you will only offend him. Just strive to be a Trusted Advisor. If he asks you to come in and "save" the current effort without changing anything, then politely explain why the results he has are the outputs of the process he's following, and won't change unless the process changes. And if he agrees to change, point out that time will be lost in the change as well. Don't promise what can't happen. Trusted Advisers don't do that. As Steve said (his recurring theme for the weekend), it's all about Relational Dynamics: how we relate to the customers (and each other) determines our success in understanding and meeting their needs.

Planning with Deadlines

I failed to take any notes from this session. Here's hoping someone else can fill in what I missed.

Closing Circle

At the end of the day, we reviewed our sessions and the lessons learned. As we did, we kept returning to some core themes: requirements, communications, embracing change, managing expectations, visibility, and complexity. And we said how ultimately success is determined by business value delivered, which depends on requirements communicated. Agile Development is about that, not about particular languages or tools or even practices. We said that in the end, there are only four absolutely mandatory practices in Agile Development:

Listen to the customer.

Turn what you heard into a concrete expression of some form.

Express what you learned to the customer.

Listen again, and repeat.

It's all about communications. Everything else derives from what gets communicated.

Sunday, Sept. 8: Many Partings

Chris had family obligations and had to leave us Saturday night, as did many others. But the diehards among us spent another night in the snore-filled cabin and woke to clean-up under the very diligent supervision of Mike (who also did more than his share of the clean-up work). By 9:30, we were all on the road to our homes, with only one question left unanswered...