Blogs

Agility@Scale: Strategies for Scaling Agile Software Development

Scott is a Senior Consulting Partner of Scott W. Ambler and Associates, working with organizations around the world to help them to improve their software processes. He provides training, coaching, and mentoring in disciplined agile and lean strategies at both the project and organizational level. He is the founder of the Agile Modeling (AM), Agile Data (AD), Disciplined Agile Delivery (DAD), and Enterprise Unified Process (EUP) methodologies. Scott is the (co-)author of 19 books, including Disciplined Agile Delivery, Refactoring Databases, Agile Modeling, Agile Database Techniques, The Object Primer 3rd Edition, and The Enterprise Unified Process. Scott is a senior contributing editor with Dr. Dobb's Journal and his company home page is ScottWAmbler.com

Agility@Scale: Strategies for Scaling Agile Software Development

Agile requirements activities are evolutionary (iterative and incremental) and highly collaborative in nature. Initially requirements are explored at a high level via requirements envisioning at the beginning of the project and the details are explored on a just-in-time (JIT) basis via iteration modeling and model storming activities. The way that you perform these agile practices, and the extent to which you do so, depends on the situation in which a project team finds itself. The Agile Scaling Model (ASM) is a contextual framework for effective adoption and tailoring of agile practices to meet the unique challenges faced by a system delivery team of any size. To see how this works, let's apply the concepts of the ASM to see how we would scale our agile approach to requirements.

First, let's consider how a small, co-located team would work. The first two categories of the ASM are core agile development and disciplined agile delivery, the focus of both are small co-located teams in a fairly straightforward situation. In these situations simple techniques such as user stories written on index cards and sketches on whiteboards work very well, so the best advice that I can give is to stick with them. Some teams will take a test-driven development (TDD) approach where they capture their requirements and design in the form of executable specifications, although this sort of strategy isn't as common as it should be (yet!), likely because of the greater skill and discipline that it requires. Traditionalists often balk at this approach, believing that they need to document the requirements in some manner. But, for a small co-located team working in a collaborative manner, requirements documentation proves to be little more than busy work, often doing nothing more than justifying the existence of a business analyst who hasn't made the jump to agile yet. Don't get me wrong, there are good reasons to write some requirements documentation, and we'll see this in a minute, but you should always question any request for written specifications and try to find more effective ways to address the actual goal(s) motivating the request. Never forget that written documentation is the least effective communication option available to you.

Although inclusive tools such as whiteboards and paper work well for requirements, for development activities you will need electronic tools. You will either put together an environment from point-specific tools or adopt something more sophisticated such as IBM Rational Team Concert (RTC) which is already fully integrated and instrumented. RTC is a commercial tool, but luckily you can download a 10-license environment free of charge, which is just perfect for a small team. Larger teams, of course, will need to purchase licenses. One of the things that a disciplined agile delivery approach adds to core agile development is it addresses the full delivery life cycle, which is important because it explicitly includes pre-construction activities such as requirements envisioning. The first step in scaling agile techniques is to adopt a full delivery life cycle which covers the full range of activities required to initiate a project, produce the solution, and then release to solution to your end users.

More interesting is the third category of the ASM, Agility@Scale, and how its eight agile scaling factors affect the way that you tailor your process and tooling strategy. Let's explore how each one could potentially affect your agile requirements strategy:

Geographical distribution. The majority of agile teams are distributed in some manner -- some people are working in cubicles or private offices, on different floors, in different buildings, or even in different countries -- and when this happens your communication and coordination risks goes up. To counter this risk you will need to perform a bit more requirements envisioning up front to help ensure that everyone is working to the same vision, although this doesn't imply that you need to write detailed requirements speculations which would dramatically increase the risk to your project. Remember, agilists do just barely enough modeling and are prepared to iteratively elicit the details when they need to do so. The more distributed the team is the more likely they will need to adopt software-based requirements modeling tools such as IBM Rational Requirements Composer (RRC) which supports streamlined, agile requirements elicitation throughout the delivery life cycle. Index cards and whiteboards are great, but they're difficult to see if you're outside the room where they're posted. I've written a fair bit about distributed agile development in this blog.

Team size.Some organizations, including IBM, are successfully applying agile techniques with teams of hundreds of people. A team of one hundred people will naturally work much differently than a team of ten people, or of one thousand people. Large teams are organized into collections of smaller teams, and the requirements for the overall project must be divvied up somehow between those teams. The implications are that as the team size grows you will need to invest a bit more time in initial requirements envisioning, and in initial architecture envisioning for that matter; you will need to use more sophisticated tools; and may need to use more sophisticated modeling techniques such as use cases and functional user interface prototypes. See large agile teams for more advice.

Compliance requirement.When regulatory issues – such as Sarbanes Oxley, ISO 9000, or FDA CFR 21 – are applicable you are likely going to be required to capture requirements specifications in some manner and to enact traceability between those requirements. However, I highly recommend that you read the actual regulations yourself and don't let bureaucrats interpret them for you (doesn't it always seem that their interpretation always results in an onerous, documentation heavy solution?) because I have yet to run into a regulation which required you to work in an ineffective manner. Managing your requirements as work items in RTC can often more than meet your regulatory requirements for documentation and traceability, although you may want to consider a tool such as IBM Rational RequisitePro for complex regulatory situations.

Domain complexity.The manner in which you elicit requirements for a data entry application or an informational web site will likely be much simpler than for a bio-chemical process monitoring or air traffic control system. More complex domains will require greater emphasis on exploration and experimentation, including but not limited to prototyping, modeling, and simulation. Although user stories may be effective as a primary requirements artifact in simple domains, in more complex domains you are likely to find that you need to drive your requirements effort with more sophisticated modeling techniques.

Organization distribution.Sometimes a project team includes members from different divisions, different partner companies, or from external services firms.In these cases, particularly where the work is strictly organized between the various organizations (perhaps for security concerns), you may need a more sophisticated approach to managing the requirements. RTC enables you to organize the requirements between teams, and then to automatically track progress in real time via the RTC project dashboard.

Technical complexity.The technical complexity of a solution can vary widely, from a single platform silo application to a multi-platform application working with legacy systems and data to a full-blown systems engineering effort. Complex technical domains, just like complex business domains, require more complex strategies for requirements elicitation and management. The requirements for your legacy systems are likely to have been captured using tools and techniques appropriate for that platform, for example the requirements for your COBOL application may have been captured using data flow diagrams and data models, whereas the requirements for your Java legacy application where captured using UML diagrams. The subteam working on the COBOL system might be using IBM Rational Application Developer (RAD) and RTC for Z whereas the Java subteam may use Eclipse with RTC. Because systems engineering projects can stretch on for years, particularly when the hardware is being developed in parallel to the software, sophisticated tooling such as IBM Rational DOORS is often used in these situations. For more information about systems engineering, see the IBM Rational Harmony process.

Organizational complexity. Your approach to requirements elicitation and management will be affected by a host of organizational complexities, including your corporate culture. When the culture is flexible and collaborative you can be very agile in your approach to requirements, but as it becomes more rigid you become more constrained in what is considered acceptable and thus take on greater project risk. For example, many organizations still struggle with their approach to funding projects, often demanding that the project team provides an "accurate" estimate up front to which they will be held to. This in turn motivates risky behavior on the part of the development, including a "big requirements up front (BRUF)" approach where a detailed requirements speculation is developed early in the project. This is just one example of how questionable corporate culture can impact the way in which an agile team works.

Enterprise discipline.Some organizations have enterprise-level disciplines, such as enterprise architecture, enterprise business modeling, strategic reuse, and portfolio management in place. These disciplines can easily be agile and from what I can tell the more successful efforts appear to lean more towards the agile end of the spectrum rather than the traditional end.Having an enterprise business modeling effort underway will affect your project-level requirements strategy -- you'll be able to leverage existing models, have access to people who understand the domain at an enterprise level, and will likely need to map your project efforts back to your enterprise models. The enterprise modelers will likely be using tools such as IBM Rational System Architect or IBM Websphere Business Modeler.

It is important to note that the way that you tailor the agile practices that you follow, and the tools that you use, will reflect the situation that you find yourself in. In other words, you need to right size your process and the Agile Scaling Model (ASM) provides the context to help you do so. As you saw above, in simpler situations you will use the simpler tools and techniques which are commonly promoted within the core agile development community. But, when things become a bit more complex and one or more of the scaling factors applies you need to modify your approach -- just don't forget that you should strive to be as agile as you can be given the situation that you find yourself in.

In 2009 I wrote a white paper entitled The Agile Scaling Model (ASM): Adapting Agile Methods for Complex Environments for IBM Rational. Apparently it's been taken down, which I think is unfortunate as it contains some interesting ideas that your organization may be able to benefit from.

The original white paper addresses several key issues:

It provides and explains a definition for disciplined agile delivery. A more up to date discussion of DAD can be found on the Disciplined Agile Delivery site.

It describes criteria to determine is a team is agile. I've explored this issue via several surveys over the years since then. See the January 2013 How Agile Are You? results.

It describes the ASM, which distinguishes between core agile development techniques, disciplined agile delivery strategies, and agility at scale. The ASM was superceded in early 2013 by the Software Development Context Framework (SDCF). Perhaps this is why the ASM paper was taken down??

It overviews the eight scaling factors which a delivery team may face, scaling factors which motivate changes in the process that you will follow and the tools that you will adopt. The SDCF provides my recent thoughts regarding scaling factors. I have also run various IT Surveys over the years exploring how well organizations fare at scaling agile.

On Tuesday, Dec 1, 2009 Philippe Kruchten, Bruce MacIsaac, and myself participated on two virtual panels about the future of the Unified Process (we did two to support callers from around the globe) for the Global Rational User's Group (GRUG). During the panel sessions we discussed a bit of the history of the Unified Process, some of the misconceptions people have with it, some of the common mistakes people made implementing it (instantiating it to be documentation heavy and/or serial) due to those misconceptions, how it can be very agile if you choose to instantiate it that way, the OpenUP, the AUP, how UP relates to the IBM Practices, and other topics.

Hope you find it interesting. As I've written in the past, the RUP can be as agile as you want to make it. Furthermore, there are a lot of really good ideas in the RUP that the agile community can and should choose to mine, although sadly I see far too many teams doing things the hard way and reinventing the process wheel on their own. I hope they're enjoying themselves, because it clearly isn't a very efficient way for them to go about process improvement.