What happens when you don’t have enough science: talking about software development methodologies

23Mar

Yusuf Arslan has recently asked an important question: Is Agile and Scrum really better than Waterfall? I think that’s a perfectly valid question and admit that he touched important aspects along the way. What piqued my curiosity was a particular section of his entry:

There are also serious allegations that promoting software development methods is just one big money-making exercise for a group of consultants. Certification has created a small army of consultants and trainers who are constantly busy training and coaching a bigger army of certified Scrum practitioners.

Being probably a valid criticism aside, I think there’s an interesting point here. I wonder whether people who bring that criticism to the table realize that the situation about programming technologies and platforms is very similar. Think about one of the most popular technology platform for developing enterprise software, or one of its rivals: If you are, like me, older than 30 and have been involved with computing at least during the last 15 years, then you will very easily remember how much branding, selling, training, certification, etc. swirled around these (and still do!). And here we are not talking about some fuzzy software development methodologies but cold, hard, concrete programming languages that run on cold, hard, physical devices that are supposedly open to rigorous physical experimentation and statistical data analysis. Does anyone sincerely believe that the majority of people and companies made their choices completely free from bias, without being effected by the huge marketing machineries of the companies producing those technology platforms? And if not, then what’s the point of saying, e.g. promoting such and such software development methods is nothing but a money-making exercise for consultants? The world of professional software development is about making money (including training people for your software), and unless you are talking about the simplest of benchmarks, everything is more or less based on speculation, and thus interpretation.

Until we have high quality data gathered from replicable controlled experiments, or carefully observed software projects (sound like an utopia), and then do some statistical analyses on them to arrive at some clear conclusions, and then at some laws by aggregating many of such studies, that is, not merely do science, but also good, high quality science, we will be more or less speculating about the ‘best practices’, software development methodologies, what works and what doesn’t. At least, being honest about that can be the first step to arrive at a more reasonable and scientific level of discussion on these matters. Until then, I think the best we will ever get will be the nicely written anecdotal case studies such as “Lean from the Trenches: Managing Large-Scale Projects with Kanban” (one of the nicest things about that book is that the author is very well aware of the fact that we are still far from talking about hard, cold, absolute scientific laws when it comes to software development methodologies).

I really appreciate that you have given a thorough response on my article. It is not my intention to denigrate development methods. I only hope that everyone gains from exchanging thoughts and knowledge and your reaction is exactly what causes this.

I actually started to write my blog article with the title “Scientific research about Scrum and Agile” where I wanted to sum up papers. Although I did encountered some research. I became increasingly less enthusiastic and changed the title. I would be more then happy to write an new article, proofing me wrong with links to great researches and papers. There are certainly some research reports on the subject. I have collected these empirical studies, but none of them are really proofing a significance. If I have collected more of them, I will write an other blog entry.

A good analogy is worth 3 hours of discussion. I enjoyed your comparison with more concrete subjects like the choice of a programming language and tooling. But saying that B is also wrong, makes seem that the wrong of A is less severe. I also think, that these both problems are linked: we maybe need to take software development as a whole even more seriously. Designing experiments (like double blind controlled experiment) or extracting quantitative data from complex datasets or field experiments should be more common. After this phase, we can discuss about epistemology and the empirical limits of science :)

I think there is nothing wrong with making money. (To be honest, I love to make money and to spend it.) But how the money is made is very important. I think it is not ethical to suggest that B is better than A, without any proof that it is and selling B with the promise that it will heal A. Burden of proof is on the person asserting the claim.

I think that there is nothing wrong with random or subjective choices. We are doing it all the time. But personally I would find it cool if we had more comparative studies why B is better than A.

Not only I agree with most of what you say, but I have also found the bibliography section of your blog entery very interesting, especially the report from K.U. Leuven on avionics, ITEA projects and agile software development methodologies. It is very important that scientists try to improve our understanding of the current matters, but as we both apparently agree, even Einstein cannot come up with solid laws without either a) having some high quality data b) relying on previous scientifc laws, which, in turn, relied on solid evidence.

I also agree that there is nothing principally wrong having random, subjective choices. The problem starts when people say ‘this is the best, this is the way to go, this is the silver bullet’ without relying on enough scientific study. Even in relatively rigourous fields that make use of controlled experiments and data sets such as medical research, neuroscience, psychology, etc. there can be fierce discussions about the conclusions, and we are talking about the discussion of software development methodologies, wherein creating high quality data sets is much more difficult due to the closed nature of many commercial projects.

If we had a few thousand, or even a hundred projects on GitHub that also documented their requirements, planning, milestones, whether the project was delivered on time and within budget, then we would have a starting point for a better discussion I believe.

There are many projects, products were developed by using Waterfall. They made a lot of money :) when people used Waterfall. When companies started to loose money, they invented Sprial. I mean, Mr. Boehm was TRW company employee.
I suggest you read http://tra-spacepark.org/docs/TRW_History.pdf

Thanks Emre, for elaborating on Yusuf’s article (I just left a comment there as well). You guys definitely have an interesting discussion going on here.

As you mentioned in your comment, Emre, I think one of the main problems is that people blindly latch onto these methodologies as a buzzword and proclaim that they have found the “silver bullet” which will solve all our problems.

I have worked on several large and small projects, using elements from both Waterfall and Agile methodologies. As it turns out, mixing things up to fit the company culture and the personalities who work there is usually the best way to go; the pragmatic way. The problem with that is that, it’s very hard to compare yourself with how other people work, because they do things differently (even if they are all “using Agile” or “using Waterfall” so to speak).

It’s very worthwhile to read about empirical software development — Laurent Bossavit has done some amazing work in debunking some commonly held tropes in the field. Check out “The Leprechauns of Software Engineering” some time.