In Defense of Agile Development (and Their Ilk) September 21, 2011

In my previous post I asked the question “why doesn’t Agile work?”. I’m not sure the nuance of the question came over correctly.

I’d just like to highlight that the question I asked was “Why does agile not work”. It was not “Why is Agile rubbish“. I’ve said a few times in the past couple of weeks that I like the ideology of Agile and I am (and have been for years and years) a strong proponent of prototyping, cyclic development, test driven design and many other things that are part of the Agile or XP methodologies.

That distinction in the title is a really important distinction and one I’d hoped I’d made clear in my post. Looking back at my post though, I think it is clear I failed :-(. I highlighted reasons why I think Agile does not work and in my head I was thinking “if we avoid these, Agile could work” – but when you write something down it does not matter what is in your head if it does not reach the paper.

I’m actually frustrated that in the last few years I have not seen Agile really succeed and also that this must be the normal situation, going on the response you get when the topic of Agile comes up with fellow technicians and comments on my own blog.

However, on that post about Agile two people who’s opinion I deeply respect came back at me to say “Agile does work!”. Cary Millsap, who many of you will have heard of as the “Method R” guy and the person behind Oracle Flexible Architecture. And Mike Cox, who most of you won’t have heard of but Mike taught me a lot about sensible development back in the 90’s. He’s one of the best developers I have ever had the pleasure of working with and I know he has had great success with Agile and RED. I’m not sure if they read my post as “Agile is Rubbish” or they are, like me, simply frustrated that it can work but so often does not.

So I’ve been thinking about this a lot this weekend and I was helped by Cary’s paper on the topic that he mentioned in his comment. I’d highly recommend downloading it as it is an excellent description of not only why Agile can help but describes how and some of the pitfalls {I’d started my own post on that, but go read Cary’s}. I should add, you can see Cary present his case for Agile at the UKOUG conference this year.

So where does this bring me to? Well, I think “Is Agile good or bad” has become almost an “IT religion” topic, people love it or loath it and it is based on what they have seen of the methodology in real life. No, that’s wrong, it is based on what they have seen that has been labelled with that methodology in real life. Or worse, it is based on anecdotal opinion of those around them. The thing is, if you look at what XP is supposed to consist of or what Agile Programming is supposed to consist of, most of us would agree that a great deal of it makes sense in many situations. I’d disagree with some of the details in Cary’s paper but overall I’m in strong agreement. Sadly, What Agile and XP is supposed to be is not well matched by what you see on the ground in most cases. So even if these methodologies are right for the situation, what has been implemented is not the methodology but probably more a slap-dash process that simply jettisons documentation, design and proper testing. This whole thread sprung from my lamenting the demise of database design and several of the comments highlighted that the introduction of Agile seemed to equate, at least in part, with the demise of design. As MIke and Cary say, and as I think anyone who has successfully utilized Agile would say, Design is an integral part of Agile and XP methodology.

Agile can and does work. But many things can and do work, such as taking regular exercise to keep healthy or regularly maintaining your house to keep it weathertight. Like Agile, both take effort but the overall benefit is greater than the cost. And like Agile, do it wrong and you can make things worse. If your window frames are starting to rot and you just slap a new layer of top-coat on them all you will do is seal in the damp and rot and hide the problem – until the glass falls out. Going for a regular 5 mile run is good for you – but not if you are 10 stone (60KG) overweight and have not run in years. A 5 mile run is also not a good idea if you want to be a long-jumper. Right training (methodology) for the right aim. Also, just like keeping healthy, house maintenance or anything that takes effort but works, proponents tend towards extremism – probably as a reaction to the constant {perceived} pig-headedness of critics or the failure of people to just do what now seems so sensible to them {think reformed smokers}. I’ll have to buy Cary and Mike pints to make up for that jibe now, and promise them it was not aimed at them personally…

Sadly, the reality is, Agile does not work 90% of the time it is tried. So, does that mean Agile is actually rubbish? Or at least, not fit for purpose, because many companies are not able to use it? Companies are there to achieve something and the IT systems are part of achieving that something. If Agile cannot aid that IT department then Agile is the wrong way for that department and company.

*sigh* I’ve gone on and on about this and still not got to my own main point, which is this.

– Can we identify reasons for Agile and XP Failing.
– Having identified the Reasons, can we fix them in simple ways?
– Can we create some simple guidelines as to when a project should be more Agile and when it should be more Up-Front design.

I’d love to know people’s opinions on those three points above.

Like this:

LikeLoading...

Related

“Can we create some simple guidelines as to when a project should be more Agile and when it should be more Up-Front design.”

Is the software going to control a system which can kill people if it goes wrong? I’m thinking of nuclear power plants, avionics, hospital life-support machines and the like. Then I’d be very, very, very worried if anyone suggested that Agile methods should be used. Not because Agile is inherently flawed, but because, as you pointed out, Agile does not work 90% of the time it is tried.

If the software is going to drive a social networking web site, then frankly, who gives a damn if there are bugs in it? Go ahead and use Agile if you want, because nobody is going to die if the web site falls over.

David, bugs and agile don’t go together, would’t expect to see any more or less bugs than I would with any other alternative methodology.

If an agile project goes live without adequate testing then it’s just another project that has not been adequately tested, again agile doesn’t lessen the need to test nor would I expect any drop in quality.

I think the problem that Martin is making is that on paper agile should be great but in practice often doesn’t deliver, will followup with answers to Martin’s points.

I’ll also put together a list of why waterfall fails everytime even when viewed as a success, Im sure not one could possible disagree with me on that one :-)

In an ideal world, that’s true. But one of the failings of “so-called Agile” that I’ve witnessed personally is that many developers don’t do testing correctly. They pay lip service to test-driven development, but the reality is that key tests are either forgotten, or implemented incorrectly, or in the worst cases, fudged so that they pass because the developer is under huge pressure from his/her manager to deliver the goods by the end of the sprint.

“I think the problem that Martin is making is that on paper agile should be great but in practice often doesn’t deliver”

And no-one could possibly disagree with him on that.

“I’ll also put together a list of why waterfall fails everytime even when viewed as a success”

Consider the computer system on board the Space Shuttle. It controlled every aspect of the launch and re-entry — the two most dangerous periods in every Shuttle flight. It did so using less memory and computing power than the most basic of today’s mobile phones. The software contained 400,000 lines of code. In 30 years of Shuttle flights, only a handful of software errors were discovered during a mission, and no crew was ever endangered by a bug in the software. Given that the software was originally written in the early 1970s, long before Agile and many of the other iterative software development methodologies, it’s a fair bet that the waterfall approach was used. I’d call that a success, wouldn’t you?

I agree with your underlying diagnosis about why Agile does not work. But if the underlying message is “do it right and it will work” then don’t you think that is applicable to anything and everything? for e.g. even the much maligned Waterfall methodology? Your main point is exactly what worries me the most. If, as you say, about 90% of the agile projects do not succeed, then I would question in the first place if it was really necessary to “invent” and “promote” agile in the first place?
I see it exactly same as the birth of new religions. The concepts of ancient religion are misinterpreted/misimplemented and then someone thinks a “new” religion would solve all the problems of mankind. But along the road, even new “religion” appears to contain flaws. That means people end up with more than one religion with “flaws”.
Personally, I think even Waterfall methodology, when used in a right manner, can resolve the issues. I would rather “break” a large project into multiple phases and apply Waterfall to each phase and it would succeed.

Exactly. Absolutely exactly. And yes, maybe the issue is that Agile works brilliantly but is just too ‘difficult’ for a typical organisation. You’ve summed up well what I was attempting to get to.

As for Waterfall methodology, of course it is the one method you can safely point to as being flawed and useless. Everyone knows it is both very old and utterly useless. Poppycock! It is a good methodology and has it’s place. We all use it all the time! It gets used a lot as the unit of work in eg Agile :-) You want to add a new screen, you get the requirements, design it, develop it, test it and release it. At each pool of the waterfall you assess if you need to step back. Classic waterfall. (OK, in how I would use Agile I’d write my tests as a step before “develop it”)

As for religion, well I’m going to steer clear of that one! Other than to observe that what is implemented so often fails to apply the original rules. JUST like methodologies.

When i first read the Agile Manifesto, my first thought was that it was a counter-argument against the threats of outsourcing/offshoring. Close business involvement and no heavy specs being picked over by opposing forces like it was the Treaty of Versailles.

So my first, absolute requirement is, are you going to be delivering updates to Prod every two-to-six weeks. If not, then it is not an Agile project. If you are, then waterfall won’t work for you.

The daily, constant, face-to-face involvement between the business/users and the developers is the other critical item.

Any project, whatever methodology, will fail if the demands are unrealistic. I think that is the key for any project. But a waterfall will fail in a different way (cost/time overruns) from Agile (buggy functionality, no documentation)