For sharing my view on things related to technoology

Waterfall guy in an Agile World

After working for considerable period of time with waterfall I recently joined Xebia. I believe it is one of the few organizations which live and breathe agile. From what I’ve seen and learnt so far I’ll say Agile Rocks!!! To me agile is simple no nonsense methodology which is all about common sense. I do not claim to be an expert on Agile or even waterfall, I just want to share what I’ve learned and understood so far working as a guy on the ground.

If somebody asked me to describe core idea of agile in fewest words I would say “frequent inspection”. Everything else in agile methodology is laid around this simple yet powerful notion. I’ve worked on lot of waterfall projects and fortunately none of them failed, all of them had a varied amount of success. For some it was smooth ride to production and others barely made it through. From last so many years I have heard and read about abysmal success record of IT projects and have often wondered why do so many of them fail? I have always held this opinion that building software is just like building anything else… like maybe building a housing project or maybe some piece of furniture. I can already see heads nodding in disagreement! I understand these analogies may not be perfect or a right fit for all kind of software projects but I still feel on a certain level its valid for significant number of them (or at least for the ones that I worked on!). It sounds almost logical that in order to build something, analysis of what needs to be built is required (consider projects in which users have a fairly good idea of what they want). The abstract requirements discovered from analysis are taken to designing table where an effort is made to propose a domain specific solution. Once that’s done and accepted actual work starts to provide something tangible that can be delivered, deliveries go through cycles of construction/inspection/rework and then the final delivery is done. Of course the events described in sequence above do not have such strict boundaries, but it can generally be acknowledged as high level framework of building and delivering goods.

This high level framework which I believe inspired waterfall methodology has for years worked successfully in various industries for building and delivering goods but has been a disappointment for software industry. Why this tested and tried logical process would not yield the same kind of results when it comes to software? When I started working with Agile I sort of got answers to these questions and I’ll share those shortly, but before that I’ll like to point out two subtle differences that exist between process of construction in software and other industries for example lets say building construction.

Prospects of inspection by end users

Ability to accommodate change

Prospects of inspection by end users
For a building construction project end users may not fully comprehend or visualize the architectural plans but they can very well understand the progress by inspecting the construction site. They can see the limits of land by looking at outer walls, they may look at bricks laid for foundation to judge the room areas, they can see when walls are erected and plastered, location of stair case etc. With the ability to inspect they can catch anomalies early and request for change. Now compare this with a software project using similar methodology, at each stage of build nothing tangible that can be inspected by end user is available. End user mostly do not understand the design documents, they do not comprehend software design, code or component interactions. The artifacts produced at various stages of software construction do not make any sense to them, since they do not understand they cannot inspect or suggest change! The opportunity of inspection is provided towards the end of the project and often it’s too late by that time to change.

Ability to accommodate change
The notion of being able to change something significantly after it’s fully built is very powerful, software artifacts offer this rare ability to its builders (developers/designers) like no other. Can you imagine plight of construction project where teams unknowingly/knowingly did not lay down water supply pipes in washrooms? The ramification of such a blunder would be huge! Builders of a construction project cannot afford such mistakes because a mistake like this is not easy to fix, this very fact forces them to be more rigorous. Software developers on other hand often believe that they do not need to be too rigorous because they can change at any stage. The ability to change later, time constraints and hassle of clarification with end user often drive developers/designers to take an easy way out. They code with what they know making assumptions, because they can fix it later if assumption was not up to the mark. This behavior can add quite substantially to project debt. Also, it’s certainly not true that consequences of changes to software would be less severe than that of a construction project, but perceived ability and ease with which it can be done can subconsciously drive procrastination in minds of some individuals.

I understand that ability to accommodate change is one of the biggest strengths of software and if done correctly can provide software construction an edge over others. Agile in fact embraces change and uses the ability of software to change as a tool to deliver. It would be worth noting that point that I made above with regard to software ‘rigor’ should not be considered in violation of agile principal “Do the simplest thing that could possibly work”. By being rigorous I am no way suggesting that one should make baseless assumptions and try to accommodate everything, all I am saying that ability to change should not be misused to defer legitimate stuff.

Enter agile!
I think Agile was born with core idea of enabling frequent inspections of software artifact by end user, everything else in various agile methodologies was layered around this notion. Agile makes sure that artifacts are always in a form that can be inspected by end user, this enables early feedback and failure is detected early. Waterfall is considered to be change averse because by the time project is ready for feedback from the end user it is so deep into the build that accommodating change is not easy and in certain cases not even possible without considerable time and effort.

Agile also provides a solution to second problem, though the lack of rigor may still exist but most of the issues introduced are detected early and fixed, unlike waterfall where they keep accumulating making a huge unmanageable lump of project debt. Since Agile is able to recognize and accommodate for these two subtle differences the chances of success with Agile are much better than waterfall.

After a long stint of working with waterfall I recently joined my current organization Xebia Architects which I believe is one of the few organizations which live and breathe agile. From what I’ve seen and learnt so far I’ll say I love agile! To me agile is simple no nonsense methodology which is all about common sense. I do not claim to be an expert on Agile or even waterfall, I just want to blog what I’ve learned and understood so far working as a guy on the ground.

If somebody asked me to describe core idea of agile in fewest words I would say “frequent inspection”. Everything else in agile methodology is laid around this simple yet powerful notion. I’ve worked on lot of waterfall project and fortunately none of them failed, all of them had a varied amount of success. For some it was smooth ride to production and others barely made it through. From last so many years I have heard and read about abysmal success record of IT projects and have often wondered why do so many of them just fail? We in software industry must be doing something wrong.

I guess I have always held this opinion that building software is just like building anything else like maybe building a housing project or maybe some piece of furniture. I can already see heads nodding in disagreement! I understand these analogies may not be perfect or a right fit for all kind of software projects but I still feel on a certain level its valid for significant number of them (or at least for the ones that I worked on!). It sounds almost logical that in order to build something, analysis of what needs to be built is required, also it would be nice to assume that in kind of project I am referring to users have a fairly good idea of what they want. The abstract requirements discovered from analysis are taken to designing table where an effort is made to propose a domain specific solution. Once that’s done and accepted actual work starts to provide something tangible that can be delivered, deliveries go through cycles of construction/inspection/rework and then the final delivery is done. Of course the events described in sequence above do not have such strict boundaries, but it can generally be acknowledged as high level framework of building and delivering goods.

This high level framework which I believe inspired waterfall methodology has for years worked successfully in various industries for building and delivering goods but has been a disappointment for software industry. Why this tested and tried logical process would not yield the same kind of results when it comes to software? When I started working with Agile I sort of got answers to these questions and I’ll share those shortly but before that I’ll like to point out two subtle differences that exist between process of construction in software and other industries for example building construction.

1)Prospects of inspection by end users

2)Ability to accommodate change

Prospects of inspection by end users

For a building construction project end users may not fully comprehend or visualize the architectural plans but they can very well understand the progress by inspecting the construction site. They can see the limits of land by looking at outer walls, they look at bricks laid for foundation to judge the room areas, they can see when walls are erected and plastered, location of stair case etc. With the ability to inspect they can catch anomalies early and request for change. Now compare this with a software project using similar methodology, at each stage of build nothing tangible that can be inspected by end user is available. End user mostly do not understand the design documents, they do not comprehend software design, code or component interactions. The artifacts produced at various stages of software construction do not make any sense to them since they do not understand they cannot inspect or suggest change! The opportunity of inspection is provided towards the end of the project and often it’s too late by that time to change.

Ability to accommodate change

The notion of being able to change something significantly after it’s built is very powerful, software artifacts offer this rare ability to its builders (developers/designer) like no other. Can you imagine plight of construction project where teams unknowingly/knowingly did not lay down water supply pipes in master bedroom? The ramification of such a blunder would be huge! Builders of a construction project cannot afford such mistakes because a mistake like this is not easy to fix, this very fact forces them to be more rigorous. Software developers on other hand often believe that they do not need to be too rigorous because they can change at any stage. The ability to change later, time constraints and hassle of clarification with end user often drive developers/designers to take an easy way out. They code with what they know on assumption, because they can fix it later if assumption was not up to the mark. This behavior can add quite substantially to “construction debt” of the project. Also, it’s certainly not true that consequences of changes to software would be less than that of a construction project, but ability and ease with which it can be done subconsciously can drive procrastination in minds of some individuals.

I understand that ability to accommodate change is one of the biggest strengths of software and if done correctly can provide software construction an edge over others. Agile in fact embraces change and uses the ability of software to change as a tool to deliver. It would be worth noting that point that I made above with regard to software ‘rigor’ should not be considered in violation of agile principal “Do the simplest thing that could possibly work”. By being rigorous I am no way suggesting that one should make baseless assumptions and try to accommodate everything, all I am saying that ability to change should not be misused to defer legitimate stuff.

Enter agile!

I think idea of Agile was born with core idea of enabling frequent inspection of software artifact by end user and everything else in various agile methodologies was layered around this notion. Agile makes sure that artifacts are always in a form that can be inspected by end user, this enables early feedback and failure is detected early. Waterfall is considered to be change averse because by the time we get feedback from the end user we are so deep into the build that accommodating change is not easy and in some cases not even possible without considerable time and effort.

Agile also provides a solution to second problem, though the lack of rigor may still exist but most of the issues introduced are detected early and fixed, unlike waterfall where they keep accumulating making a huge unmanageable lump of project debt.