Barrier To Agile Development

Why don’t more companies and teams use agile development techniques? We know some teams just aren’t aware of them – although that list is getting shorter every year. The benefits of iterative development over waterfall development are pretty well established. I don’t believe I’ve seen a study that shows that waterfall is more effective. Do people refuse to believe in the data? Or maybe they are unable to believe.

His most recent article talks about the non-negotiability of software quality when pressed for time. Although the comments thread seems to be going off on a tangent about the iron triangle of project negotiations (scope/content, cost, and time), the article is pushing a different idea. The idea that time already spent is a sunk cost.

The idea of sunk costs, also know as “don’t throw good money after bad” is difficult for most people at a psychological level. We often feel like making just a little more investment of time, work or cash will turn an ailing effort around. Unfortunately, a traditional waterfall or phased-based approach to project work increases the psychological pressure to continue working, even when the project is “bad”.

Mishkin Berteig

The point that Mishkin seems to be driving at is that by using an agile process, you dramatically reduce the sunk cost psychological issue. One thing Alan Cooper said about agile processes that really stuck with me is that agile processes are optimized to deal with change. His comment was in the context of a left-handed compliment, that agilists presume that change is inevitable because it is impossible to know in advance what needs to be done. That is an interesting debate, and even mentioning it here is likely to cause our discussion to go off on a tangent.

Taking those two ideas together – it makes sense that people who anticipate and plan for change will not incur a psychological penalty for changing their solution in response to changes in requirements. It makes for good support to Mishkin’s argument. Waterfall processes make change harder because of sunk costs. But there’s more.

Cognitive Dissonance

OK, now the picture of a brain scan might make more sense for this article.

Cognitive dissonance is a psychological term which describes the uncomfortable tension that may result from having two conflicting thoughts at the same time, or from engaging in behavior that conflicts with one’s beliefs.

The reason that people feel uncomfortable about sunk costs is rational. Making decisions based on sunk costs is irrational. Here’s how cognitive dissonance can play out in a waterfall project:

We know we’re good at what we do. We gathered requirements and we’re implementing a solution based on those requirements – and we’ve developed a test plan to validate that we actually did what we set out to do. In short – we rock.

We’ve invested a lot of time implementing what we’ve done so far.

We’ve just been told that what the customer really needs is something else.

If we accept that the customer needs “something else”, then that means that we wasted a bunch of time and money doing the wrong thing. The delays (time is not negotiable – see Mishkin, we get it) may doom this product to failure even if we do fix it. This of course means that we are a lousy product team.

Cognitive dissonance is introduced between items 1 and 4 in the list above. We can not both rock and be lousy. These two ideas are mutually exclusive. In Mistakes Were Made, But Not By Me (linked in the ‘Recommended Reading’ box at the end of this article), Carol Tavris and Elliot Aronson describe this situation and its global ramifications in a way that will enlighten and frighten you. A quick quote from Dr Cathy Goodwin that sums up the central idea behind the book:

Why do people refuse to admit mistakes – so deeply that they transform their own brains? They’re not kidding themselves: they really believe what they have to believe to justify their original thought.

And that’s what we have here. Cognitive dissonance prevents people from accepting that change is important in a waterfall project, because they have conspicuously avoided change for the life of the project. It almost seems like they go out of their way to make sure that they don’t accidentally uncover changes until the project is complete. And the higher the waterfall, the more vested interest, the more likely it is that change is needed, and the greater the cognitive dissonance.

Cognitive Dissonance Prevents Adoption of Agile Development

Not only does the notion of cognitive dissonance explain why people freak out about sunk costs, but it explains why waterfall projects are so terrifically bad when they fail.

Now take this tactical assessment and apply it to the longer term.

We’re good at software development.

We run waterfall projects, we have check-gates, and other processes to assure that we get the job done right.

Sure, many of our projects fail. And maybe our “successes” are not as big as we initially planned them to be.

Agile development would make this better. The CHAOS report, many other companies, and many thought leaders in the space all seem to agree. We don’t do agile. We’re really not good at software development.

We’re introducing cognitive dissonance again. We can’t be good at software development if our process choice was the wrong one.

Catch and Release

There’s a catch to cognitive dissonance. And this is the striking part. It isn’t the case that rational people see the data and don’t like the conclusions to which it leads. Cognitive dissonance is a subconscious process. People are not aware. In an interview on NPR’s Science Friday podcast, Dr. Aronson referenced studies with brain scans that demonstrated the dissonance affect.

How big is this affect? In the podcast, Dr. Aronson refers to multiple cases where prosecutors discover that innocent people are incarcerated (and will continue to be incarcerated). They have, for example, DNA evidence that the convicted person could not have committed the crime. And those prosecutors would rather (subconsciously) let the innocent person rot in jail for the rest of his sentence than accept that he was wrongly convicted. That’s powerful stuff!

The pleasure centers of the brain of an affected individual are rewarded when they resolve the dissonance by ignoring or discounting the “conflicting idea.” We subvert the possibility of being wrong, at a subconscious level, and our brains reward us for it. That’s the catch.

So what do we do? Our only hope is to consciously accept the premise that our old ideas might be bad. That we might in fact, not be good at software development (or whatever). By making the change to acceptance of the new idea – and perhaps even experiencing contrition about it – we also will satisfy the pleasure centers of our brains. That’s the release.

Summary

Cognitive dissonance also exists as a barrier to adopting agile processes. People experience cognitive dissonance when exposed to data that what they’ve been doing with waterfall processes would be improved by changing to agile processes. These people resist changing how they run their projects as much as they resist the acknowledgment of changing needs during their projects.

Thanks Craig for your continued reading and comments – they are good and appreciated.

I’ve been on projects where the client insisted on “complete delivery” – where people express a position that “if it doesn’t do everything I need, it isn’t worth using.” In those cases, the final, formal release will be “everything.” And I’ve delivered on those projects both as waterfall and as incremental development projects.

I’ve been much more successful when managing internal deliveries / beta releases, etc. Along the way. By getting partial releases, and eliciting customer feedback – with a willingness to change the objectives along the way, I’ve been able to operate as if it were an agile project. Even if the end customer is not realizing benefits from the incremental releases, the feedback cycle still provides the “do the right thing” benefits of agile projects.

“We’ve just been told that what the customer really needs is something else.”

That is the key point to everything we discuss on this topic….
1) Did the customer just not know what they wanted? If so, what did they say they wanted? Just make something up?
2) Did the customer really want one thing, but said they wanted something else? Good lord, why?
3) Did they really want what they first said, but now the need has changed due to a change in the business environment?

In the case of (1), of course it would be better if the customer just said they weren’t sure what they needed (often because they are entering a new business line), so an exploratory/prototyping approach is the way to go.

In the case of (3), this is a real situation to be managed. It has become the most well-known situation because many past projects were too long nd delivered in a big bang. Everyone should know by know that this a recipe for failure. Whether you call them iterations or phases or whatever, a project has to deliver something usable on regular basis, no longer than 3 months apart.

If you are dealing with (2), go home and get out the Glenlivet…

In the end, I am getting a little bored with the “one methodology is better than another” discussions. No one approach works in all cases. I think it is very dependent on the people doing the work; if you are lucky enough to have ‘generalizing specialists’, go Agile. Most people, however, are more specialists than generalists, so agile may not work in that case. Division of work into specialized tasks has been around along time, for a reason.

So, the issue here is not agile or waterfall, it is effectiveness. Mis-using resources in the flawed belief that one approach is best will breed failure as well. Remember, the customer has other options too. As a Business Analyst, it has been several years since I worked on brand-new development, it has been maintenance of existing systems or purchased packages. Agile or Waterfall makes no difference in these projects.

Cognitive dissonance is an interesting theory as a barrier to agile development but I don’t think it’s the largest. My personal belief is that new methods (of any type, not just agile) are not adopted because they are NEW. Change is hard. Change is risky. Who is signing up to lead the charge and bear that risk?

As the senior project manager recently hired at a small company I’m mid process of moving us to more agile methods. Because I’ve initiated the move my internal reputation and probably my job are on the line. That’s a lot of risk I’m taking on that many others wouldn’t, and I think that’s the largest barrier to agile adoption.

The reason I’m doing this is my own experience with quality management early in my career and the parallels I see with agile methods. Done the right way for the right reasons it works and done half way or in the wrong place it fails spectacularly, but regardless the cultural change is hard. There are huge challenges just in exploring the cloud of terms, branded processes, and tools. Which are Agile, which are charlatans?

I’m trying to bring agile to my company because I’m confident in my ability to manage risk and change and I believe agile has a place in our business.

Something I’ve not seen anywhere: You can divide agile into agile project management and agile development. They’re good compliments but aren’t dependent on each other. I’m introducing agile PM (scrum) because that’s my domain but I’m not going to ask developers to start doing pair programming or test-driven development.

@sehlhorst on Twitter

Who Should Read Tyner Blain?

These articles are written primarily for product managers. Everyone trying to create great products can find something of use to them here. Hopefully they are helping you with thinking, doing, and learning. Welcome aboard!