Why Lean And Agile Go Together

The command-and-control hierarchical organization of large companies is rapidly giving way to a more biological metaphor, one in which a corporate nervous system senses important signals from the market and supply chain and rapidly reacts. At the largest scale, this trend is being driven by principles of Lean manufacturing, which envisions the enterprise as a massive event-driven system, waiting to create products when customer orders arrive instead of building products in advance and holding them in inventory.

Software development has been similarly transformed by Agile development methods. In this arena, the command-and-control ethos is represented by the Waterfall development methodology, which follows a seemingly reasonable pattern of gathering requirements, designing a product, and building and testing. The problem is that requirements are almost always wrong. The Waterfall method does not allow for mid-course corrections, and the beginning-to-end cycle may span a year or even many years.

Agile posits a huge payoff for being highly suspicious of software requirements. In Agile methods, instead of building the whole product, you build the smallest possible useful part and give it to users, who tell you what is right and what is wrong. Agile development is an evolutionary conversation in which incremental steps of two to four weeks lead to feedback that allows requirements to be tested and adjusted. In addition, some functionality is available much sooner.

Quality also increases in Agile projects because using a working system exposes defects right away instead of leaving them to a final testing phase. Many other aspects of Agile are beneficial but beyond the scope of this discussion.

Agile's popularity has led to a problem: How do you apply it at scale? How can a large project run using Agile methods? One team working on one project in an Agile way is not hard to envision. But what about running 10 or 20 teams, each working on part of a product? How does the list for each iteration get sorted out and synchronized? How does the result of each iteration become integrated into the larger whole?

I've been discussing this topic for a few months with Ryan Martens, CTO of Rally Software, a leading vendor of software and training services to support adoption of Agile development. In addition, I saw a fascinating presentation about tracking Agile projects at the New York CTO Club last week in which Bruce Eckfeldt, the founder of Cyrus Innovation, an Agile training company, detailed various methods of tracking progress of Agile projects.

What surprises me is that most advanced practitioners of Agile now use more vocabulary from Lean manufacturing than from Agile. In essence, as a practical matter, good ideas from Agile are being absorbed into a new approach to software development that is more Lean than anything else. Someone else can name this phenomenon, but Lean and Agile are merging.

The fundamental recommendations of Agile methods are focused on creating a rapid feedback loop between the users supplying the requirements and the technologists transforming them into a solution. Agile methods also suggest a variety of engineering practices that increase the quality and stability of code.

Agile has much less to say about how to connect the work of many different teams, and that's where Lean has a huge impact. In a Lean manufacturing system, the work is broken into a set of value streams triggered by demand signals. The output of one value stream leads to others. Value streams may be executed sequentially or in parallel as needed. Eventually, everything is combined into the product. The suppliers for materials needed are alerted through a system of just-in-time replenishment of parts and components called Kanban.

Large organizations that are using Agile are applying a value stream approach in which various development teams are organized in sequential and parallel streams of work so that at the end of each iteration, you get a new version of the product. Sometimes different rhythms of iterations occur, with some work taking place in faster increments followed by the integration of everything every fourth or fifth iteration.

"Large Agile teams manage dependencies through multiple and iterative synchronization points, but they also work to remove them through better design process like Lean's set-based and concurrent engineering approaches," Martens says. "We've come a long way, but I see another decade of major improvement coming from the application of Lean principles to software development."

For example, SAP has been using SCRUM and other Agile methodologies for several years at the team level. Herbert Illgner, COO Business Solutions and Technology at SAP, who has been involved with the effort, says that team empowerment and faster feedback cycles with customers are two significant benefits. Illgner added that SAP is expanding application of Agile methods to the entire product creation process using a Lean framework that includes empowered cross-functional teams, continuous improvement process and managers as support and teachers. "We are implementing a large scale SCRUM for software development teams that includes several thousand developers," said Illgner. "We expect to reduce planning overhead, synchronization and integration efforts significantly as we move from individuals and individual timelines to crossfunctional teams and time boxed iterations. This is a radical change compared to how software has traditionally been developed and requires significant commitment and support especially from top management."

The Lean concept of Kaizen also has a strong influence on the way Agile is being practiced, filling a gap relating to continuous improvement. The evolution of Agile is primarily focused on evolving the product toward a better fit with requirements. In Agile, both the product and the requirements are refined as more is known through experience. Kaizen, a continuous improvement method used in Lean, focuses on the development process itself. When Kaizen is practiced in an Agile project, the participants not only suggest ways to improve the fit between the product and the requirements but also offer ways to improve the process being used, something usually not emphasized in Agile methods. Eckfeldt described the use of Kaizen snakes and project thermometers to capture process improvement feedback.

"The power of Agile is that it's self-adaptive," Eckfeldt says. "It teaches software teams how to deliver value to customers and how to improve themselves using techniques like Kaizen, allowing them to deal with unique and changing constraints and environmental factors."

In practice, Agile seems to be changing for the better by adopting Lean thinking in a large way. Rally says that its customers get to market 50% faster and are 25% more productive when they employ a hybrid of Lean and Agile development methods. Given the way that Agile fits in to the Lean framework, it wouldn't surprise me if before too long Agile is considered a branch of Lean practice tailored for the software industry.

Dan Woods is chief technology officer and editor of Evolved Technologist, a research firm focused on the needs of CTOs and CIOs. He consults for many of the companies he writes about. For more information, go to evolvedtechnologist.com.