Tuesday, December 15, 2009

This is my third installment writing about our agile adoption. In the first posting I discussed how we got started and how things started to pan out for us after a few months at it. In the second I moved on to look at a few of the practices we began trying after figuring out the basics. In this posting I'm going to share how we started to go beyond the notion of adopting a general notion of agile and selected Scrum along with some of the things that led to.

Ever since I started learning about agile I've become a voracious reader again. Blog posts, articles, books. I tore through many in the first months and continue to do so still. At some point I'll blog on what I found to be the most helpful material. For now though, what I want to talk about is how all this reading quickly deepened my understanding of the various ideas sitting under the general agile umbrella. In particular I was struck by the dominance of Scrum as a framework for agile project management. Although there was mention of XP and Lean (aka Kanban to some) and to a lesser extent I would stumble across RUP or Crystal...it was Scrum that seemed to have got center stage. And the more I've learned about it the more it seems clear why. It's a pretty light framework with minimal mandatory dogmatic things prescribed for you to follow. It makes a helluva lot of sense and it's not "hard" to internalize and figure out how to use it. At least not conceptually. Of course I think it takes a lot of effort to consistently employ it. If you're not convinced that Scrum is simple, try reading one of the short simple explanations of Scrum or even the official Scrum Guide. Really not that hard eh? Of course there is more to it than that. Applying this framework and these principles to an actual project leads to lots of decisions. Many times you need to "try" stuff and see what occurs. Scrum is great for that with feedback loops very obviously built into it.

All this reading about scrum started to change the language we were using. In the past we always talked about iterations, now they were sprints. People were getting comfortable with points and velocity -- much less talk of hours and days. We framed things in terms of stories now. Our former project managers were scrum masters. Our status meetings were daily stand ups. We didn't have a lessons learned or post mortem meeting, we had retrospectives. We didn't have iteration kick-off meetings, we had sprint planning meetings.

This change in language happened first just inside the immediate team, but eventually we were talking to our customers about velocity, story points and so on. I can't quite articulate why, but I found this to be immensely cool. I think maybe it was because we had started to get away from the "dammit I want all my stuff done by February" type conversations and on to much more productive and healthy "here's where the team is, here's what they look like they can do by February, maybe we should alter priorities of what's in the backlog so X, Y and Z get done."

This sea change in language was deeper than just terminology though. It spread in part through a kind of osmosis, but also through people's interest being piqued, through training (a number of people attended CSM courses) and through people sharing articles, doing their own research and so on. Again for me this was just the coolest thing to see, an organic growth in learning and interest from people.

Up until around this point we still had a rather large shadow hanging over what we were doing: stories weren't shared between software engineers and QA staff on the team. A set of stories would be developed during a sprint and, at the end, a build given over to the QA folks for them to test during the subsequent sprint. Any defects found resulted in a bug fix and new build. We were in effect doing mini-waterfalls, or "scrummerfalls".

We knew we wanted to get out of that pattern, and as we embarked on a new product release cycle we went all in on doing shared stories. Specifically now, a story was shared by the software and QA engineers, and it needed to emerge from the sprint completely done: developed and tested; bug free and potentially shippable. But there was more to "done" than this. We work in a business subject to FDA regulations. We can (and do) get audited by our clients and the FDA. They expect to see that we follow a rigorous software development process, that we have SOPs and follow them. That we follow what they believe to be best practices such as traceability from requirements to testing, that we unit test, peer review code and so on. To help with this notion of "done" -- meaning a feature was (potentially) ready to ship -- we decided to come up with a checklist that would serve as our "definition of done." This basically contained items like code the story, develop tests for the story, peer review of code, unit tests, execute manual tests, fix any defects, update any related documentation etc.

As an aside...I was quite humbled to learn recently that we (that is me and my peers) did a piss poor job of getting this definition of done idea out to the team. I realize now that we dropped it on them already figured out, affording them no opportunity to discuss and agree what should be on there. Very bad and lesson learned for me there: people are going to feel like it's a mandate from their managers rather than a tool to help them. And really it should be a tool to help them -- something that makes clear all the good work they need to be allowed to do to develop software professionally; a guard against manager types and customers pressuring people to cut corners and build up "debt" for the future that we'll have to then rush to address or skip altogether.

This move to shared stories that need to be really done and potentially shippable at the end of the sprint had several more knock on effects:

the order in which we implemented and tested stories was now important -- previously we just cared which sprint they were done in, now people needed to know what order they would get done in during a sprint

we were clearly going to be releasing multiple builds during a sprint as stories were implemented and ready for testing

we needed, more than ever, to keep our build "clean" -- ready to tag and build a version that could be put into the testing environment

much greater shared understanding and transparency between software engineers and QA engineers was necessary too

the whole team shared estimation with planning poker

In addition to the above, a number of other things started to happen. We explored FitNesse (don't let the dodgy logo and generally retro look put you off) as a means to adding acceptance tests. That helped a lot with figuring out how far our unit tests should go. In the past there was some pretty heavyweight stuff there which was really integration or acceptance tests masquerading as unit tests. Thinking in terms of acceptance criteria or tests I believe is also going to help people do a better job of focusing on requirements.

With two product teams now running Scrum and a third already exhibiting many agile behaviors we decided to try holding a "Scrum of Scrums" meeting. We had tried in our pre-agile past many times to do *something* to get cross-team communication going. That had always typically failed in one way or another. Part of that, I think, was due to us having 2/3rds of the product development done by something less that a full-time, well defined team. This led to situation where when you got people together you needed too many and nobody was terrible interested in anybody elses situation. Now with stable teams we can have one or two members represent each team *and* there are only three streams of product development rather than a dozen. It still took us quite a while to figure out how to do the Scrum of Scrums, there's not a lot of concrete info out there, although talking with other folks provided some ideas. Right now we still have some ways to go, and may even change our focus in the future, but I believe for now we've found a good approach which I will write about at some later point.

After all of this change, what were we getting out if it? Well I'm pleased to say some pretty good stuff. Two key things were product releases on the promised date. In fact we were done a little earlier, which never hurts. But in addition to that, whilst we clearly could see some people struggling with the quantity and rapidity of changes introduced, most were thriving. There was a renewed spark in people's eyes -- they were starting to own things more, to realize they could influence things more. And that is clearly leading to a better working environment for everyone.

I'm confident that as we continue we will have more and more success. We will keep delivering what's needed on time with high quality. We won't have frenzied periods at the end of product release cycles burning people out on death marches. We will have predictability; we'll know team's capacities and we'll be able to plan well based on that. We'll use more automation. We'll see stronger relationships amongst team members. Our customers will be happier, more involved and satisfied with what they get. They'll trust us to deliver what they need.

In a bit I'm going to write more more part in this series of posts on our agile adoption. It's going to be the forward looking piece, talking about what I think we need to focus on next. Top of that list for me right now is figuring out how to capture requirements in a way that satisfies our regulatory obligations and is compatible with our Scrum approach to product development. Additionally I believe we should be focusing on more XP technical practices, test automation and really getting the Product Owner role accepted by the business.

Monday, December 14, 2009

The other day I retweeted what I thought was an interesting article from the Harvard Business Review about meetings. The idea was that to avoid those days of back-to-back meetings by scheduling 50, rather than 60 minute long meetings. In this way the day is punctuated with 10 minute interludes that allow one to catch up a little bit.

I thought it was a great idea, but in response someone asked "how does that apply to meetings to discuss 'issues' where an email with the answers would do?" This is a good question, and I can't help but notice that there's a lot of gratuitous meetings. I wanted to comment on my thoughts and ideas about organizing meetings optimally for an agile team. Absent the full opportunity to try out some of these ideas it is more theory that proven methodology. Nonetheless it's based on my own observations and a fair bit of discussion and background reading...so it's not completely off the wall.

Personally, and I've read others feel the same (can't remember who though!), just inviting me to a meeting is a huge mental burden. Compared to emailing me, IM-ing me or even stopping by my desk I find it considerably more invasive. Any one of those other low key approaches and I can probably be done in minutes without really feeling to imposed upon. If needs be I can ignore/deflect the interruption too.

But you send me a meeting invite and it's an entirely different proposition. As the allotted time approaches it tends to creep into my mind...especially when that "15 minutes to go" Outlook remind chimes in. I wind down in anticipation, wander off to the meeting room (or dial in to the call) -- we do the smalltalk as we wait for everyone to assemble, then we're done...then I go make some tea or something. Finally I sit back down at my desk and try and get back into whatever I was doing before. The cost of the meeting in terms of flow is high. This is especially true for some: see Paul Graham's "Maker's Schedule, Manager's Schedule" for a good explanation of why this is.

All this preamble brings me to my first premise of meetings:I'm only interested in really necessary ones. It's far, far too easy for people to adopt a default mode for addressing things by calling meetings. Only recently I and two of my colleagues were invited to a meeting to discuss a bug by the people that found it. Of course the bug didn't really need a meeting. All salient information about the bug could be communicated via the bug report and clarification handled through informal discussion. The meeting was all about impressing upon us the severity of the bug and just how much we needed to pull our finger out and fix it pronto. I'd rather be told that straight so we're all clear about people's expectations.

I digress; moving on...so if we're only to have "really necessary" meetings -- which for me right now is those recommended by Scrum and probably some requirements workshop/story writing/estimation type things -- then how do we deal with all that other necessary communication that must happen on a product development team? How do we get clarity on various issues, or thrash out design decisions, etc.? My answer would be to just go grab the colleague(s) you need and get to it.

That notion may strike fear into some people: "Eeek no, leave me alone, if you must talk to me book some time, I don't want to be interrupted like that without warning."

But there are ways to handle this. In the Paul Graham article above he suggests one approach. I think it's spot on for all the meetings people like myself and my management peers want to have with people on product development teams, namely try and put them at the very beginning or very end of the day. That way people are left with a big hunk of uninterrupted time to do their thing.

But for the intra-team communication that must happen, I think this would be too rigid. Something I believe is more suited here is the Pomodoro Technique. At it's very simplest you can understand this to work as follows:

set a timer for 25 minutes and focus on whatever your next work task is without stopping until the end

at the end of the 25 minutes take a ~5 minute break

repeat

at the end of four of these "pomodoros" take a longer break

It's more subtle than this, but the key additional points that I see are:

don't permit interruptions when the timer is ticking: ignore email, IM, phone and people walking up to you (gesture pointedly at the ticking timer...)

deal with any important interruptions during the breaks, e.g. go see what Bob wanted when he wandered over looking for you

If Bob is also using the Pomodoro Technique he may either have decided to wait until you are free or started up another pomodoro (25 minute work session) of his own. Obviously if he's done the latter whatever he wanted wasn't that important...or maybe he IMed/emailed his question. If he's waited then maybe he's looking to work on something with you: it could even be done as a shared pomodoro.

I think this approach would be a very effective way of handling interrupts without ever forcing people to wait an unreasonable amount of time for access to another team member. I say "I think" because I've not had chance to put this fully into practice in a team with everyone agreeing to work this way. I've use pomodoro's myself as a simple way to just focus on getting stuff done. Procrastination is harder when using them. And it stops me from "allowing" myself to be interrupted. Based on that evidence alone I feel pretty good about how well it could work for a team. Organized this way I they would still be able to have frequent interactions as needed. But they wouldn't be filling up their calendar with "meetings" just to talk to one another.

Of course one key enabler for this approach is probably co-location, maybe even ideally a shared team room/workspace where everybody sits. But even without that I believe it would be a good approach.

There's also a psychological benefit I believe to not having a calendar full of nasty white blocks of meetings. You wouldn't feel so much as though you had been subjected to a day of meetings, unable to "get anything done". Rather, you'd feel like you had been working all day long, some of that time collaboratively with your colleagues as you and they found it helpful.

Therefore, my second premise of meetings is: handling issues via informal discussion doesn't mean you need to be unpredictably interrupted all day long.

Of course there are meetings that have to happen. Sprint planning meetings, retrospectives, requirements workshops etc. And for these I believe it's key to adhere to a few principles to enjoy success.

First of all, any real meeting needs an agenda, preferably with each item on it timeboxed. I've been known to reject invitations or at least tersely question the sender in the absence of agendas. Now obviously you can't go around doing that to everyone...but for a group of peers running a Scrum product development team everyone is free to call out colleagues requesting meetings that are all too vague.

Additionally it's important that any meeting is well facilitated: who's keeping notes? Who's ensuring we stay with the agenda? Who helps encourage the less talkative members of the team to voice their opinions? Some of these things can be shared amongst a good team, but having an identified facilitator makes for a good experience I believe.

Given how invasive meetings are they should be of a reasonable duration. I particularly liked that idea from the HBR of stopping 10 minutes early to avoid filling up entire hours. Multi-hour meetings (like spring planning) should have built in break times and they should be kept to.

Lastly, good meetings need respect from and for all participants. In particular I see the following as key:

give people and their ideas and contributions a fair hearing

invite people who haven't yet made contribution to volunteer their opinion

be there and start on time -- lateness is a lack of respect unless there is a good reason

quality phone communications (rooms with poor hardware or acoustics plain suck for those on the other end of the phone)

make the effort to be clear for those not physically present (i.e. those on the phone)

use a webex if appropriate -- it helps those not in the room follow along

Thus my third and final premise of meetings is: if we're going to have a meeting, let's bloody do it well.

Yeah, I'm biased about the not physically present stuff...I do so many meetings over the phone :-)