Blogs

Agility@Scale: Strategies for Scaling Agile Software Development

Scott is a Senior Consulting Partner of Scott W. Ambler and Associates, working with organizations around the world to help them to improve their software processes. He provides training, coaching, and mentoring in disciplined agile and lean strategies at both the project and organizational level. He is the founder of the Agile Modeling (AM), Agile Data (AD), Disciplined Agile Delivery (DAD), and Enterprise Unified Process (EUP) methodologies. Scott is the (co-)author of 19 books, including Disciplined Agile Delivery, Refactoring Databases, Agile Modeling, Agile Database Techniques, The Object Primer 3rd Edition, and The Enterprise Unified Process. Scott is a senior contributing editor with Dr. Dobb's Journal and his company home page is ScottWAmbler.com

In November 2011 Paul Gorans, the Accelerated Solution Delivery (ASD) practice lead in IBM GBS, and I ran an agile adoption survey. The survey explored a range of issue, including the factors that appear to be associated with the success and failure of agile project teams. Paul wrote up his thoughts in his Agile State of the Art Survey article on ibm.com and I did the same for Dr Dobb's Journal in Agile Success Factors. This blog posting summarizes the results of the survey.

There is a fair bit of rhetoric surrounding agile methods, some of which we subscribe to and some of which we don’t. We’d like to briefly examine the rhetoric which we’ve found to be the most misleading for people trying to be effective at adopting agile techniques. The following list is in the format X but Y, where X is the rhetoric and Y is the strategy promoted by the Disciplined Agile Delivery (DAD) process framework. This includes:

Requirements evolve throughout the lifecycle BUT the scope should still be agreed to at the beginning of the project. There has to be an initial vision for a project, a vision which your stakeholders should help define and then agree to, and to come to that vision you will need to perform some initial requirements envisioning. A list of high level features is part of this initial vision. Yes, the details are very likely to evolve over time but the fundamental goals of your project and scope of your effort needs to be defined early in your project. In a very small minority of situations you may not be able to get the right people together, either physically or virtually, to define the initial vision – this should be seen as a significant project risk.

Simple designs are best BUT the architecture should be thought out early in the lifecycle. Too many developers interpret the advice to focus on simple designs to mean that they should build everything from scratch. Yet more often than not the simplest design is to take advantage of what is already there, and the best way to do that is to work closely with people who understand your existing technical infrastructure. Investing in a little bit of architectural envisioning early in the lifecycle enables your team to identify existing enterprise assets that you can leverage, to identify your architectural options, and to select what appears to be the best option available to you. The details will still emerge over time, and some decisions will be deferred until a later date when it’s more appropriate to make them, but the bottom line is that disciplined agilists think before they act.

Teams should be self organizing BUT they are still constrained (and enhanced) by your organizational ecosystem. Intellectual workers, including IT professionals, are most effective when they have a say in what work they do and how they do it. IT professionals can improve their productivity by following common conventions, leveraging and building out a common “dev-ops” infrastructure, building towards a common vision, and by working to common business and technical visions. In short, disciplined agile professionals are "enterprise aware".

Delivery teams don’t need prescriptive process definitions BUT they do need some high-level guidance to help organize their work. Individual IT professionals are typically highly-skilled and highly-educated people often with years of experience, and teams of such people clearly have a wide range of knowledge. As a result of this knowledge it is incredibly rare for such people to read detailed procedures for how to do their work. However, they often still require some high-level advice to help them to organize their work effectively. Teams can often benefit from techniques and patterns used by other teams and this knowledge sharing should be encouraged.

IT professionals know what to do BUT they’re still not process experts. A decade ago the strategy was to provide detailed process advice to teams, but recently the pendulum has swung the other way to provide little or no defined process at all. Over the last few years there’s been a trend within the agile community to advise teams to define their own process so that it’s tailored to their own unique situation. While this clearly strokes people’s egos, it’s relatively poor advice for several reasons. First, although every team is in a unique situation there is significant commonality so having at least a high-level process framework from which to start makes sense. Second, although these teams have a wide range of knowledge it might not be complete, nor consistent, nor is it clear what the trade-offs are of combining all the really good techniques that people know about. There is significant benefit in having a flexible process framework such as DAD which shows how everything fits together.

IT professionals should validate their own work to the best of their ability BUT they likely aren’t testing experts so therefore need help picking up the appropriate skills. The mantra in the agile community is to test often and test early, and better yet to test first. As a result agile teams have adopted a “whole team” approach where the development team does its own testing. This works when there are people on the team with sufficient testing skills and more importantly can transfer those skills to others. Minimally you will need to embed testers into your delivery teams, but you should also consider explicit training and mentoring of everyone on the team in testing and quality skills. You may find my agile testing and quality strategies article to be an interesting read.

Disciplined agile teams work in an iterative manner BUT still follow a lifecycle which is serial over time. On any given day people on a DAD project team may be performing analysis, testing, design, programming, deployment, or a myriad of other activities and iterating back and forth between them. But, the DAD lifecycle includes three distinct phases which are performed in order. So, DAD is both iterative in the small but serial in the large.

There is a distinct rhythm, or cadence, at different levels of the agile process. We call this the agile 3C rhythm, for coordinate, collaborate, and conclude (which is sometimes called stabilize). The agile 3C rhythm occurs at three levels in Disciplined Agile Delivery (DAD):

Day. A typical day begins with a short coordination meeting, called a Scrum meeting in the Scrum method. After the daily coordination meeting the team collaborates throughout most of the day to perform their work. The day concludes with a working build, hopefully you had several working builds throughout the day, which depending on your situation may require a bit of stabilization work to achieve.

Iteration. DAD construction iterations begin with an iteration planning session (coordinate) where the team identifies a detailed task list of what needs to be done that iteration. Note that iteration modeling is often part of this effort. Throughout the iteration they collaborate to perform the implementation work. They conclude the iteration by producing a potentially consumable solution, a demo of that solution to key stakeholders, and a retrospective to identify potential improvements in the way that they work.

94% of teams which are claiming to be agile are providing value to stakeholders on a regular basis.

87% of teams which are claiming to be agile are validating their own work.

95% of teams which are claiming to be agile are working closely with stakeholders.

56% of teams which are claiming to be agile are self organizing.

88% of teams which are claiming to be agile are improving the process that they follow throughout the lifecycle.

Teams which are claiming to be agile often aren't. 53% of "agile teams" meet the five criteria, although 72% meet all but the self-organization criteria.

Teams which are moving towards agile but aren't there yet are reasonably close. 39% of those teams meet all five criteria and 63% meet all but self-organization.

I believe that there are several important implications:

Whenever someone claims to be on an agile team you may want to explore that claim a bit deeper.

The low level of self organization may be an indicator of cultural challenges with organizations in that their project managers aren't giving up sufficient control. The Agility at Scale survey in November 2009 found that 59% of respondents who indicated that their organization hadn't adopted agile techniques yet that a rigid culture was hampering their efforts. The IT Governance and Project Management survey in July 2009 discovered that "questionable behaviors", many of which were ethically questionable (I'm being polite), were far too common within IT project management.

Although "agile teams" may not be as agile as they claim, they're still doing better than traditional V-model teams, as revealed (again) by the 2010 IT Project Success survey.

If there was some sort of consensus within the agile community as to the criteria for determining whether a team is agile, I highly suspect that the agileness ratings would increase over time. What gets measured often improves.

However, how agile you are isn't anywhere near as important as getting better at what you're doing. So perhaps I'm barking up the wrong tree on this issue. ;-)

When you are first adopting agile techniques in your organization a common strategy is to run one or more pilot projects. When organizing these projects you typically do as much as you can to make them successful, such as finding:

Projects where the stakeholders are willing to actively work with you.

IT people who are flexible, willing to try new things, and willing to collaborate with one another.

Finding a project which is of medium complexity (therefore it's "real" in the sense that it's significant to your organization) but not one where it can make or break your organization (therefore it's safe to experiment with).

In North America we refer to this as "cherry picking" because you're picking the cherry/best situation that you can find.

Some thoughts:

Being agile may not have been the primary determinant of success. You set up an environment where you have a good relationship with your stakeholders, where you have good people who want to work together, and the project is challenging but not impossible. Oh, and by the way you adopted a few agile techniques as well. Sounds to me that situation you could have adopted a few not-so-agile techniques instead and still succeed. Although my various project success surveys, see my IT surveys page for details, have shown time and again that agile project teams are more successful than traditional project teams I haven't been able to tease out (yet) whether this success is attributable to agile or just attributable to improved project initiation efforts.

When adopting agile/lean widely across your organization, you can't cherry pick any more. For the past few years I've been working with IT organizations that are in the process of adopting agile/lean strategies across their entire organization, not just across a few pilot projects. What these organizations are finding is that they need to find ways to adopt agile where the business isn't as willing to work with IT, where some of the people aren't so flexible or collaborative, where some of the people are narrowly specialized and not as willing to expand their skills, or where the project exhibits scaling factors which motivates you to tailor your agile approach. It's harder to succeed with agile in these situations because they're not as "cherry" as what you've experienced previously. Luckily, if you've been successful previously then you now have some agile experienced people, you have successes to reference, and you've likely overcome some problems even in the cherry situations which you have learned from. So, your cherry successes will hopefully improve your ability to succeed even in "non cherry" situations.

You need to work smarter, not harder. If the source of your success was actually from improved project initiation practices and not from agile, then recognize that and act accordingly. Realistically part of your success was from that and part was from agile, and the organizations that adopt a measured improvement approach potentially have the data to determine which practices lead to success and which didn't. Without the metrics you're effectively flying blind when it comes to deciding how to improve. There is clearly a mandate for smarter work practices within IT, within your organization as a whole for that matter.

If you want to gain more insight into some of the issues that you'll face when adopting agile across your organization, I suspect that you'll find my recent paper Scaling Agile: An Executive Guide to be interesting. I've got a more detailed paper in the works, so stay tuned to this blog.

An imporant step in scaling your agile strategy is to adopt a Disciplined Agile Delivery (DAD) approach instead of one which is just focused on agile construction. One aspect of adopting a DAD approach it to mature your focus from just producing software to instead providing a solution which meets the needs of its stakeholders within the appropriate economic, cultural, and technical constraints. The fundamental observation is that as IT professionals we do far more than just develop software. Yes, this is clearly important, but in addressing the needs of our stakeholders we will often:

Provide new or upgraded hardware

Change the business/operational processes which stakeholders follow

Change the organizational structure in which our stakeholders work

Update supporting documentation

And yes, develop high-quality software

Although delivery of high-quality, working software is important it is even more important that we deliver high-quality working solutions to our stakeholders. Minimally IT professionals should have the skills and desire to produce good software, but what they really need are the skills and desire to provide good solutions. We need strong technical skills, but we also need strong "soft skills" such as user interface design and process design to name just two.

The shift to a solution-oriented focus from a software-oriented focus requires your agile teams to address some of the software-oriented prejudices which crept into the Agile Manifesto. The people who wrote the manifesto (which I fully endorse) were for the most part software developers, consultants, and in many cases both. It is little wonder that this group would allow a bias towards software development creep into the language of their manifesto.

An inhibitor that I run into again and again within organizations that are still in the process of adopting agile development techniques is something that I call the "We're Special" anti-pattern. The people involved believe that their situation is special, that some unique factor in their environment makes it all but impossible to adopt agile techniques, and therefore they need to continue to work in the manner that they've always worked, regardless of the obvious inefficiencies of doing so.

An organization suffers from this agile adoption anti-pattern when they start giving domain-based or technology-based excuses for why they can't become more agile. For example, I've heard bank employees claim that "Agile works fine for building web sites, but we're building financial systems therefore agile won't work for us", telecom employees claim "Agile works fine for building financial systems, but we're building embedded systems therefore agile won't work for us", and government employees claim "Agile works fine for embedded systems, but we're building web sites therefore agile won't work for us." Needless to say I often struggle to not roll my eyes.

The reality is that the business domain that you're working in doesn't dictate your ability to adopt agile strategies. I've seen very successful agile projects in banks, insurance companies, manufacturing companies, retail companies, pharmacueticals (yes, life critical applications), telecoms, and government agencies. I've also met people working in those domains claim that they're special because of the inherent challenges of the domain.

Similarly, technology isn't an issue. I've seen project teams that were successful at applying agile approaches using Java, VB, COBOL, C, Linux, Windows, System Z, on PCs, and so on. Granted, some technology platforms suffer from a plethora of "agile tooling", PL/1 comes to mind and I'm sure that there's a few more niche platforms where this is the case, but with a little online searching it's often possible to find good open source tools out there (or what's stopping you from starting such a project?).

The primary issues around agile adoption are cultural in nature. Can you become more flexible in your thinking? Can you become more disciplined (agile requires greater levels of discipline than traditional approaches)? Can you build a collaborative environment with your business stakeholders? Can you move away from bureaucratic processes to ones which focus on adding real value? Can you invest in your IT staff to give them modern development skills required for test-driven development (TDD), continuous integration, and agile database techniques (to name a few)? Addressing the "people issues", the cultural issues, is the hard part of moving towards agile.

If you're looking for valid excuses for why your organization can't move to agile, here's some valid adoption inhibitors that I see in organizations all the time:

Our project management office (PMO) has been trained and certified in traditional strategies and struggles to come to grips with agile project management techniques

We don't have the funding to train, educate, and mentor people in agile techniques

Middle management is threatened by agile strategies because their role clearly needs to change

Senior technical staff, in particular our architects, don't accept the need to roll up their sleeves and be actively involved on project teams

Our IT governance effort is not itself being governed effectively and is all but out of control, focusing on bureaucracy instead of enabling development teams to succeed

Our data management group insists on working in a serial and documentation heavy manner

Our QA/testing group insists on detailed requirement specifications

Our staff is overly specialized, resulting in numerous hand-offs between analysts, developers, architects, testers, and so on

The above list is just the tip of the iceberg. The point is that the real problems that you face are cultural in nature, not domain-based nor technology-based. It is possible to overcome these inhibitors to success, but you need to understand that you're suffering from them, what the implications are, and how to overcome them. This is one of the facets of the Health Assessment portion of our new Measured Capability Improvement Framework (MCIF) service, the goal of which is to help organizations improve their internal IT processes. Although MCIF isn't specifically about becoming more agile, the reality is that there are a lot of great agile practices out there, and some of them are good options for your organization. Assuming of course you get over your misconception that you're special for some reason and instead accept the need that you've got some hard slogging ahead of you to improve your IT game.

The popular Agile literature can often seam naive when it comes to how Agilists work with project stakeholders:- Extreme Programming (XP) has a practice called On-Site Customer where one or more people work closely with your team to provide information and to make decisions in a timely manner.- Scrum has the role of Product Owner who is the one single person that the development team goes to for decisions about requirements. - Agile Modeling (AM) has the practice of Active Stakeholder Participation which extends On-Site Customer to get the stakeholder(s) actively involved with the modeling effort through the use of inclusive tools and techniques.

These are great strategies for small, co-located teams doing straightforward development, but they quickly fall apart at scale. This occurs for several reasons:1. Stakeholders are a diverse group. Your stakeholders include end users, business management, project funders, enterprise architects, operations staff, support staff, other system development teams, and many others. Different people have different, and often contradictory, requirements and they certainly have different priorities. It's questionable whether a single person, or a handful of persons, can adequately represent this diverse group.2. One person becomes a bottleneck. Even with a small co-located team this is a problem, let alone one that is geographically distributed or one that is very large. There's no way that a single person can be available 24/7 in a responsive manner to support distributed teams.3. It's a difficult role. The Product Owner/Customer (POC) is responsible for representing the business to the development team. They're making important decisions on a regular basis, decisions which they'll be held accountable for.4. One person becomes a serious project risk. Not only is it questionable whether a single person can fairly represent all stakeholders, even if they could what happens if you lose that person? They effectively become a single point of failure for your team.

To scale this role, consider the following strategies:1. Recognize the true scope of the POC role. Not only are they stakeholder proxies they also are a development team representative to the stakeholder community as a whole. As stakeholder proxies they'll make decisions and prioritize the work, they'll run requirements elicitation sessions, they'll negotiate priorities, and they'll put the development team in contact with stakeholders who have expertise in specific aspects of the domain. As team representatives they'll often demo the current version of the system to other stakeholders, communicate the status of the project to people, and respond to various requests for information from the stakeholders.2. Have multiple people in it. A single POC works well for small, co-located teams developing simple software. At scale you'll soon discover that you need multiple people in this role so that they don't become a bottleneck. For distributed teams it's common to see each subteam have one or more POCs who are managed by a primary/chief POC. The primary POC typically works on the coordinating team with the chief architect (I'll talk about this role in a future blog posting) and the program manager (also a topic for a future blog posting).3. Train them in business analysis skills. The person(s) in the POC role need good business analysis skills. If fact, it's common for people who were formerly BAs for traditional teams to step into the POC role, particularly with BAs who originally come from the business side of your organization. This strategy has its advantages and disadvantages. As a BA they've likely got solid business knowledge but their instincts may motivate them to take a documentation-driven approach to providing information to the development team instead of a collaboration-based approach. Be careful.4. Consider the full system development lifecycle. There's far more to the POC role than supporting the development team during Construction iterations. During "Iteration 0", the Inception phase for an Agile RUP project or the warm-up phase for an Eclipse Way project, the POC(s) will often lead the initial requirements envisioning efforts. The product backlog, or better yet your work item list, needs to come from somewhere after all. During the release iteration(s), the Transition phase for RUP or the End-Game phase for Eclipse Way, the POC(s) will focus on communicating the upcoming release to the stakeholder community, will be actively involved with any final user acceptance testing (UAT), and may even be involved with training end users.

In my January 2008 column in Dr Dobb's Journal, posted at http://www.ddj.com/architect/204801134 , I provide detailed advice about how to scale the way that you work with stakeholders on Agile projects by applying the practices of Agile Model Driven Development (AMDD). There's no magic solution, you just need to choose to organize yourself effectively. The good news is that you can easily work with stakeholders at scale.[Read More]