Lean Software Development in a Nutshell

Abstract

Much of the work done in software development groups (IT or product) is the result of delays and interruptions. This sentence sounds funny, doesn’t it? But on reflection, let’s see where a lot of our time (work) is spent:

Reworking requirements

Building software to satisfy missed or misunderstood requirements

Finding bugs

Thrashing while trying to integrate two components that were built semi-independently of each other

What causes most of this work? I’d suggest delays. In the above cases, respectively:

Delays in the time from getting requirements until using them

Delays caused by having to do big batches of work until getting feedback on it

The delay from writing the bug until it is discovered (fixing is fast if this delay is short, finding gets exponentially longer the longer this delay is)

The delay from testing the synchronization of the two components

Not to mention inefficiencies create by decisions made at the wrong time (too early means a delay from when the decision was made until it should have been made, too late means a delay from when it should have been made to when it was made).

One hears a lot about batch size in lean because batches of work are the easiest things to spot. But it is actually the delays and interruptions we are trying to eliminate when we manage flow. Managing batches does this because the underlying cause of batches (too much work in progress – WIP) is the same underlying cause of delays and interruptions.

Lean’s mantra of “delivering fast” is often misunderstood to mean “go fast.” It doesn’t mean that. It means “avoid delays and interruptions in order to get there (delivery) fast.” You avoid delays and interruptions by attending to flow.

How to Read this Article

It is best to read this article with a fresh mind. If I had a Men-In-Black neurolyser I would use it on you to wipe out what you’ve heard about both Lean and Agile. There is a lot of nonsense out there about what each of these mean. On the Lean side we’ve heard many say:

Lean is about being more efficient

Lean is about needing fewer people so you can let some go

Lean is about eliminating waste

Lean is based on manufacturing methods and is best learned from Toyota

On the Agile side we’re heard many say:

Agile requires you to release every 2-4 weeks

Agile says to not do up-front design

Agile says to not look ahead for planning

Agile says not to document anything

Agile says teams get to do what they want

Agile only works for software

Agile does not have a place for management

<flash> <blink> Please forget you ever heard any of these. J

Where We Are

In order to make a correction in direction it is important to first understand where one is and sometimes how one got there (so we won’t keep coming back here). Most software organizations are currently overloaded with work. They have too many, too large, projects that are poorly prioritized being worked on at the same time. Most of these projects are bigger than they should be for several reasons. A prevailing one is that when the stakeholder’s project comes up to be done, everything that is anticipated as being needed is asked for because it is unclear when there will be a second chance (and the change process to do this is very painful). So everything is asked for - we call this “the getting it while the getting is good syndrome.”

The result is that people work on multiple things. It is not unusual for key people to be working on six or seven projects. When we investigate organizations that claim their folks work on one project, we discover that, while they are working on only one project, they are likely working on six or seven things within that project. The resulting delays/interruptions in their workflow are very similar to folks working on multiple projects.

Working on multiple projects also causes many interruptions. When information is needed, the person who has it is often not immediately available – creating a delay for the person who needed it.in addition to the interruption of the person who has the information. Critical people are typically interrupted the most, meaning your best people are reduced the most in efficiency. There is another side effect of all of this: the development group gets so jammed executives can’t wait for their pet projects to be done so they interrupt entire projects by interjecting new ones.

It’s like in the picture to the right. We know we’re overloaded but we’ll still jam in one or two more things into the spots available. In the hope we’ll get more done we slow everything else down. The root cause of our problem is our focus on efficiency and utilization. Lean talks about eliminating waste, but sometimes we can’t see the waste. In software, writing and fixing bugs looks the same as writing software without bugs. It’s hard to tell what’s happening when it is going on. Software is different than manufacturing; therefore, we need something different to look at to improve our methods.

People often think the downside of all of these projects is task-switching. We know task-switching is bad, but what can we do? The reality is that task switching is only a (small) part of the problem. The bigger problem is that delays and interruptions themselves create the additional work I mentioned at the start of this article.

We are in an endless cycle of having too many things to work on which causes delays, and interruptions which vastly expands time to market which tends to have people interrupt us and to ask for more than they absolutely need because they are afraid there won’t be a another chance. This cycle just makes things worse and worse. It cannot be fixed with the current orthodoxy of achieving high utilization, but instead is a downward spiral. Adding work to an already overloaded team just makes it worse.

The Lean Mindset

Lean is based on two fundamental principles – first, one must take a systemic view to improvement and second, one must do this by looking at the flow of work though the system. The first is a shift from managing people to improving the systems within which people work. The second is a shift from focusing on productivity directly to removing delays in workflow to increase productivity.

Systems Thinking: The foundation of Lean

Systems thinking means that you recognize that a system is not merely a collection of its pieces but rather includes the way they work together. That is, one must take a holistic view when trying to improve the system since affecting one part of the system will have an impact on the other. Lean takes this a step further by incorporating Deming’s belief that the system within which people work has a very profound impact on the quality of the work these people do. This does not mean to imply that people don’t matter. It does mean, however, that the system within which people work is as significant as the people. One way to look at it is the belief that putting good people into a bad system will result in poor results. It isn’t systems that make people great, however. Putting poor people into good systems won’t get good results either. But the attitude is that most people are competent and motivated and that if you are not getting results you want you need to improve the system in which the people work instead of focusing on how to get the people to be more motivated.

Lean focuses on getting good people to create great systems to create spectacular results.

A focus on time

Lean’s focus on time is the main insight that extended Deming’s system thinking and turned our focus away from utilization. Lean suggested a focus on doing things at the right time to improve time eliminate waste and get products out the door quickly. Instead of looking at the cost of each step we look at the cost of delaying the product. Think of the development of a product from its start to its finish. We call this “concept to consumption.” There are times we work on developing the product and times people are interrupted in this workflow and work on something else. The more delays and interruptions in this workflow the long it takes to get the product out the door. This is another way of looking at “time to market” an old concept, of course. Lean software’s twist on this was that these delays and interruptions create additional work for us to do. Therefore, by attending to removing delays and interruptions, we can deliver faster because we have both fewer delays and less work to do. We mentioned a few of these delays at the start of this article.

In other words, a project that could be built by a focused team in one-month will take more than two months if they work on it half-time. Working on multiple projects not only extends the time required but adds work to be done. The extra time is not due to task-switching, but due to new work being literally created from the delays of working on multiple projects.

Lean therefore suggests that removing delays and interruptions in the workflow increases productivity, lowers cost and improves quality. Lean suggests that it is these delays and interruptions that are the source of waste and poor quality. Therefore, if we can eliminate these delays, we can improve our quality, lower our cost, raise our productivity and delivery product quicker - not by going faster, but by removing interruptions and delays. This is a different focus than most companies have.

The Source of Working on Too Many Things

The main cause of working on too many things are:

Long (annual) planning cycles

Larger than necessary projects

Doing development in large batches predicated by phase gate methods

Specialized team members who have to help out on many projects

Validation of what has been built in large testing cycles at the end which results in much being worked on at one time – which results in late feedback

Implementing Lean Software Development

Lean Software Development tells us we need to:

Limit work to capacity to remove delays and lower interruptions

Attend to the timing of the workflow with methods such as acceptance test-driven development, test-driven development and continuous integration

Use time-to-market (which Lean calls "cycle time") as the ultimate measure of progress

The solution is to focus on the most important functionality, get it completed and then go on to the next most valuable work. The question remains, of course, how to do this. Lean Software Development tells us to look at our workflow and attend to the cost of delays. The image of our work flowing through the system gives rise to the name “value stream.” That is, it is the flow of value from “concept to consumption.”

This is accomplished by working on smaller pieces of work. This, in and of itself, has a very beneficial effect. By necessity, timeframes shorten when smaller pieces are worked on. Eli Goldratt put it this way, “often reducing batch size is all it takes to bring a system back into control”.

To have a value stream which can deliver products quickly we must do the following:

Identify the most valuable work to be done

Avoid building what’s not needed

Provide work to teams in a manner that allows them to work together with little delay

Have the workflow be in the proper order

Evolve robust system and application architectures using emergent design

Manage the workflow so all of these actions are appropriately load balanced throughout the value stream

A brief look at each one

Identify the most valuable work to be done. This first requires the use of Minimum Marketable Features (MMFs) (or their equivalent) in identifying the right sized, most valuable, capabilities to be delivered. These can then be prioritized prior to entering the development process.

Avoid building what’s not needed. One often sees functionality built in large blocks. This is often done to create economies of scale – particularly in the test environment. Unfortunately, this also often results in functionality being built that was not fully understood or needed. Furthermore, it is often difficult to take advantage of what has been learned during this process and therefore new ideas, that are often more valuable than what was originally planned, have to be postponed for the next release.

Provide work to teams in a manner that allows them to work together with little delay. The biggest shift to Lean-Agile requires that teams pull work from their backlogs of work to be done when they are ready. This enables them to achieve stability in their development methods by avoiding them from working beyond their capacity. When more than one team is required to work together, these backlogs must be populated in a coordinated fashion so teams work in a coordinated fashion.

Have the workflow be in the proper order. Recent advances in Agile engineering practices (TDD, ATDD, BDD, continuous integration) suggest that using tests as a validation method greatly increases quality and helps avoid misunderstandings of requirements. While not originating from Lean, they are very consistent with the Lean concept of reducing delays – particularly from initial communication of a requirement until its validation. Lean teams must incorporate these ideas to avoid delays and unnecessary rework.

Evolve robust system and application architectures using emergent design. Architectures have always needed to evolve. Attempts at creating an architecture that will not need to change have at best put off the inevitable. It is better for an architecture to be able to change than it is for it to be able to withstand change. Fortunately, the use of design patterns, refactoring, and test-driven development can be used to create these architectures.

Manage the workflow so all of these actions are appropriately load balanced throughout the value stream. It is essential to manage the work in progress (WIP) throughout the work flow. Exceeding the capacity of the teams at different stages of work will result in queues, delays and interruptions.

Lean, Management, and Visibility of Work

While there is often debate about whether you can manage something you can’t measure, everyone agrees that you can’t manage something you can’t see. Lean suggests that we map out our workflow in what are called visual controls – that is, visual representations of our work that enable us to manage the work. As a practical matter, these value stream maps can be created as Kanban boards (the short video Mapping a Value Stream to a Kanban Board).

These workflow representations (Kanban boards) can be used to create a context within which all parts of the value stream engage. This avoids local optimizations from hurting true productivity. The ultimate measure used by management in lean implementations is value delivered by time. Management’s role is to help create the bigger picture needed for teams to work together and to remove any impediments in the overall workflow.

Push vs. Pull

The essence of effective Lean-Agile development is to deliver the most important capabilities in the timeliest manner. To enable developer groups to respond quickly, they must work without slowing themselves down with delays and interruptions. This means they need to accept work at their own pace. This approach is called ‘pull’ because they pull work from a backlog of prioritized work. This enables a more stable rate of productivity. This, in turn, creates greater predictability of the rate at which they accomplish their work. On the other hand, when work is given to them with set schedules, the set schedule typically requires more things being worked on than should be. This contributes to delays and interruptions causing additional work. This instability in the development process causes unpredictability of what they can accomplish.

Not the Whole Story

This, of course, is not the whole story of Lean. Lean provides insights into product prioritization, management, and learning.

Summary

Much of the work done in software development is not value added work but is work caused by delays and interruptions. Working from old requirements, re-working requirements, the time to find bugs, thrashing due to integration, implementing misunderstood requirements often account for more than half of the work done by a development team. Delays in, and interruptions to, our workflow is mostly due to working on too many, too large projects. We must start at the beginning of the value stream by doing proper product portfolio management. We must create visibility throughout the value stream so we can improve the efficiency of the work being done. We must manage the workflow across teams when more than one team is involved. We must shift from focusing on isolated optimizations of efficiency to a holistic view of the entire workflow.

Table 1 shows a summary of the cause of delays and the actions to counter them.