Thoughts about software development

The pitfalls of Test-Driven Development

A few days ago, David Heinemeier Hansson posted a very negative article on Test-Driven Development (TDD) which generated quite a bit of noise. This prompted Kent Beck to respond with a Facebook post which I found fairly weak because it failed to address most of the points that David made in his blog post.

I have never been convinced by TDD myself and I have expressed my opinions on the subject repeatedly in the past (here and here for example) so I can’t say I’m unhappy to see this false idol finally being questioned seriously.

I actually started voicing my opinion on the subject in my book in 2007, so I thought I’d reproduce the text from this book here for context (with a few changes).

The Pitfalls of Test-Driven Development

I basically have two objections to Test-Driven Development (TDD).

It promotes microdesign over macrodesign.

It’s hard to apply in practice.

Let’s go over these points one by one.

TDD Promotes Microdesign over Macrodesign

Imagine that you ask a famous builder and architect to construct a sky scraper. After a month, that person comes back to you and says

“The first floor is done. It looks gorgeous; all the apartments are in perfect, livable condition. The bathrooms have marble floors and beautiful mirrors, the hallways are carpeted and decorated with the best art.”

“However,” the builder adds, “I just realized that the walls I built won’t be able to support a second floor, so I need to take everything down and rebuild with stronger walls. Once I’m done, I guarantee that the first two floors will look great.”

This is what some premises of Test-Driven Development encourage, especially aggravated by the mantra “Do the simplest thing that could possibly work,” which I often hear from Extreme Programming proponents. It’s a nice thought but one that tends to lead to very myopic designs and, worst of all, to a lot of churn as you constantly revisit and refactor the choices you made initially so they can encompass the next milestone that you purposefully ignored because you were too busy applying another widespread principle known as “You aren’t going to need it” (YAGNI).

Focusing exclusively on Test-Driven Development tends to make programmers disregard the practice of large or medium scale design, just because it no is longer “the simplest thing that could possibly work”. Sometimes it does pay off to start including provisions in your code for future work and extensions, such as empty or lightweight classes, listeners, hooks, or factories, even though at the moment you are, for example, using only one implementation of a certain interface.

Another factor to take into consideration is whether the code you are writing is for a closed application (a client or a Web application) or a library (to be used by developers or included in a framework). Obviously, developers of the latter type of software have a much higher incentive to empower their users as much as possible, or their library will probably never gain any acceptance because it doesn’t give users enough extensibility. Test-Driven Development cripples library development because its principles are at odds with the very concept of designing libraries: think of things that users are going to need.

Software is a very iterative process, and throwing away entire portions of code is not only common but encouraged. When I start working on an idea from scratch, I fully expect to throw out and completely rewrite the first if not the first two versions of my code. With that in mind, why bother writing tests for this temporary code? I much prefer writing the code without any tests while my understanding of the problem evolves and matures, and only when I reach what I consider the first decent implementation of the idea is it time to write tests.

At any rate, test-driven developers and pragmatist testers are trying to achieve the same goal: write the best tests possible. Ideally, whenever you write tests, you want to make sure that these tests will remain valid no matter how the code underneath changes. Identifying such tests is difficult, though, and the ability to do so probably comes only with experience, so
consider this a warning against testing silver bullets.

Yes, Test-Driven Development can lead to more robust software, but it can also lead to needless churn and a tendency to over-refactor that can negatively impact your software, your design, and your deadlines.

TDD Is Hard to Apply

Test-Driven Development reading material that I have seen over the years tends to focus on very simple problems:

A scorecard for bowling

A simple container (Stack or List)

A Money class

A templating system

TDD works wonders on these examples, and the articles describing this practice usually do a good job of showing why and how.

What these articles don’t do, though, is help programmers dealing with very complex code bases perform Test-Driven Development. In the real world, programmers deal with code bases comprised of millions of lines of code. They also have to work with source code that not only was never
designed to be tested in the first place but also interacts with legacy systems (often not written in Java), user interfaces, graphics, or code that outputs on all kinds of hardware devices, processes running under very stringent real time, memory, network or performance constraints, faulty hardware, and so on.

Notice that none of the examples from the TDD reading materials fall in any of this category, and because I have yet to see a concrete illustration of how to use Test-Driven Development to test a back-end system interacting with a 20-year-old mainframe validating credit card transactions, I certainly share the perplexity of developers who like the idea of Test-Driven
Development but can’t find any reasonable way to apply it to their day jobs.

TestNG itself is a very good candidate for Test-Driven Development: It doesn’t have any graphics, it provides a rich programmatic API that makes it easy to probe in various ways, and its output is highly deterministic and very easy to query. On top of that, it’s an open source project that is not subject to any deadlines except for the whims of its developers.

Despite all these qualities, I estimate that less than 5% of the tests validating TestNG have been written in a TDD fashion for the simple reason than code written with TDD was not necessarily of higher quality than if it had been delivered “tests last.” It was also not clear at all that code produced with TDD ended up being better designed.

No matter what TDD advocates keep saying, code produced this way is not intrinsically better than traditionally tested code. And looking back, it actually was a little harder to produce, if only because of the friction created by dealing with code that didn’t compile and tests that didn’t pass for quite a while.

Extracting the Good from Test-Driven Development

The goal of any testing practice is to produce tests. Even though I am firmly convinced that code produced with TDD is not necessarily better than code produced the traditional way, it is still much better than code produced without any tests. And this is the number one lesson I’d like everybody to keep in mind: how you create your tests is much less important than writing tests in the first place.

Another good quality of Test-Driven Development is that it forces you to think of the exit criteria that your code has to meet before you even start coding. I certainly applaud this focus on concrete results, and I encourage any professional developer to do the same. I simply argue that there are other ways to phrase these criteria than writing tests first, and sometimes even a simple text file with a list of goals is a very decent way to get started. Just make sure that, by the time you are done with an initial version, you have written tests for every single item on your list.

42 Comments (and 3 trackbacks)

I agree with pretty much everything here. While TDD is supposed to be a staple of agile development, I understand that it lets you validate your changes quickly by giving you a coherent test suite (in theory), but in practice it makes code way more complicated to design and write, as you point out.

Also, testing UIs still being a huge PITA, I’ve never seen (or even heard of) UI-related code being developed in this manner.

I read dhh’s article last week and I had the chance to see the hangout between Martin, Fowler and dhh himself.

dhh was more about the way he programs. He prefers pen and paper and probably wants to explore deeper the requirements and this pretty complies with your vision of why a library (let’s call RoR a library for the sake of the argument) is not the right candidate for TDD.

I have to say though that TDD really fits when you are working with Scrum in a “standard” company. Usually our priorities change at each sprint so the YAGNI applies quite well in our situation and even they don’t change we create stories (for new feature) that you could do in less the three working days so it’s pretty impossible to have the need to refactor everything at once.

That said TDD does not fit everything. If I’m developing a sorting algorithm for a tree, I won’t use TDD because probably I need to implement a well known algorithm so I’ll use tests just for acceptance.

What I’m trying to say is that we are engineers and we are professionals, so, as long as you can ship what you are asked, you can use alternatives to TDD.

I will stick with TDD because I found it to be really functional for the type of programming I do.

It is a tool for a specific purpose. But then people end up falling in love with the tool. For a guy carrying a hammer, everything looks like a nail. It becomes a religion instead of a tool. That is how it goes wrong.

In the early 90s, there were 3 stars in the horizon, Grady Booch, Ivar Jacobson & Jim Rumbaugh. These were considered the gurus of Object Orient Methodology. Each was peddling his own brand of symbols, path, DOs and DONTs, etc. The old waterfall based SDLC was under attack by these luminaries. There were some lesser known ones like Shlaer Mellor and Fusion methodology too.

In the first few years of my career, I bounced through all these competing schools of thought. Then I got really tired, and started using common sense. Somewhere in late 90s, the 3 musketeers decided to settle their differences and came together to create the next great methodology religion. The UML! By then I had become quite suspicious of such dogmatic approaches, and decided not to join anymore methodology religions and cults.

When a methodology makes people think “following process” is the actual output [irrespective of the real output], it has crossed over to the cult zone. [play the twilight zone theme here]

The Agile, Scrum, TDD & XP came much later. But I have become agnostic, so I keep away and, stay happy and content using common sense.

Although I agree with some of the examples I also think that TDD is a very useful tool while developing. The same with riding a bike. If you never learn how to ride a bike, you will never be able to ride it. You may have strong opinions about other people riding a bike. Its dangerous and a menace to traffic etc. etc. but you wont ever grasp why people want to ride a bike.

The same is applicable to TDD. If you have never done it in fully, you wont ever be able to do it.

Secondly,m in my opinion, development and design are two fundamentally different things. You can design small parts of code based upon your own experience and what you can influence but not the full system.

The example of the building is a poor one. It advocates that a construction worker should design a bridge. In my opinion the two do not mix and are not what TDD is about. You have the design of the entire bridge but while building you come across a problem, the river bank is not able to support the projected weight. How we know this? We have tried it out and studied the result of the trial. We need to adapt a small part and invent a solution on the spot.

Morale of the example. TDD is a tool any developer should be able to wield and not the solution to all problems known to mankind.

I somehow completely agree with the article even though i favoured TDD over anything since i started agile development, so it means we are better off with simpler extensible design of our software without focusing too much on TDD.

one of my colleague missed his deadline because he was too much in love with TDD and ultimately he visited every piece of code to furnish his test cases over and over again, although his code was good but the software was missing key components that should have been part of it. just because he spent too much time on TDD.

I completely agree with you : the benefits of TDD are very debatable and i thank you for writing about it, like others have done before.

A few years ago, during an interview for a project, someone asked me if i knew what TDD is and what i thought of it. I answered that I knew what it is and that I did not think much of it. The guy was very much into TDD so the rest of the interview was about coding together a solution for a small problem in Eclipse using a TDD approach.
At the end of the interview, i was still not convinced by TDD and I had to pretend that i liked the TDD approach because he was very much into it and because at that time almost everybody was talking about it in a very positive way.
But these days people are expressing their dislike of the TDD approach and I am quite happy about it. I do not need to hide anymore.
Spending time trying to first find a test case in order to come with a robust solution in the end is a waste of time in my opinion.

I agree with both posts only because I have experienced that over engineering and complexity…it keeps people in a job at the end of the day.

The problem is that many enterprise development scenarios use TDD, almost religiously and will crucify you if you dare to speak up against their mantra – that’s good but it doesn’t fit every model and I can’t help but think that it keeps people in their positions and jobs because the whole package is so complicated (SCRUM, TDD etc).

I much prefer the Agile approach of a smaller team where we don’t have the luxury of time hence not enough time to scaffold up the various tests first or during development time so we either test last or in some cases throw it into an accepted Alpha stage for the customer to feedback on.

Agile here means being adaptive and using what works, not repeating verbatim some mantra because its what everyone else is doing.

However that is not to say that we shouldn’t be coding efficiently. Our code should be engineered in such a way that it lends towards TDD and the first actual testing and developer does is in the head as we bring to life an often evolving vision in our minds (well most of us, the rest are just automatons who are perfect for the whole SCRUM Master, Slave Coder mentality haha).

I don’t know if it is the same for others, but for non-trivial problems where good design is required, I always end up doing a lot of code rework to get the solution right whether I do TDD or not. So for me the question is not about whether I have to do rework, but the conditions under which that rework is done. And I would infinitely prefer to do the rework with tests to help me know when my thinking is not precise enough.

I don’t feel this article is actually against TDD at all. It is actually against the idea of “writing the simplest thing that could work and refactoring mercilessly”. The latter is often advocated by those who use TDD, but it is not IMO inherent to TDD itself. I regularly use TDD for difficult and historic applications but take a more forward looking approach; the difference is that because I have to write tests for the any additional hooks I add, that it forces me to think more critically about how necessary they really are.

Regarding YAGNI, there are really two categories of design decision:
- The fairly mechanical ones, such as whether you use a class factory or whether a class triggers events.
- The ones with much more consequence, such as anything that touches on the mental model you create of your problem domain.

If you want to meaningfully comment on YAGNI you can’t limit your examples to the first category.

I don’t believe the first point is really TDD’s fault, Object Oriented Design quality depends on experience, because experience tells the designer that principles are ‘the more what you’d call “guidelines” than actual rules’ and which you apply and which don’t and in which degree you apply them changes for every project.

But in my experience the second hits the spot. The initial design/code for any real project is going to change, at least 50% before you have the first release. Any time I tried to write tests for that classes, had to stop, because while writing the tests I found so many things to change in the design, that the tests lost their meaning.

From my point of view, you only can write tests on advance when you are doing something you have done before, but I do not call that “development”.

Well, it is true, but you can still not abandon TDD because of some misusage. It is a lifesaver if you have to deal with calculations like in invoicing and accounting apps. I can think of many other crucial use cases where TDD can help to survive an app through several years. The most important point is given in the last sentence. You should test smart and not every idea that pops in your mind.
I would like to add that testing can improve your API design. Again, only if you use it the right way.

DHH’s article was a breath of fresh air in the claustrophobic testing space. I’ve been using coverage testing since the mid 1980′s and prefer to do that when there is time and budget.

The real problem with any form of testing is that management will rarely provide the resources to do it. The best tests are written by professional testers who didn’t write the code, but most teams do not have them.

I agree and disagree. I strongly believe that TDD has a place. But I have been around long enough to agree that it is just too difficult to put into practice EVERYWHERE. My toughest attempts: (GUI, DB, and Older Systems Integration).

While a library should maybe not be DESIGNED using TDD, it certainly should have a Test Harness, and support full-on Regression/Validation tests.

I come at this from 30+ yrs of software experience, where one of my early jobs was making a change to an insurance rating system that had NO testing, and NO source control. You could not be certain that “something” changed. And the OLDER the code was (10yrs since someone touched it, the less convinced you were it would compile to the SAME “exe” that was currently in production). Trust me. I wish I had a TDD test harness to run it through back then!

But I do have a problem with minimalist designs… I prefer a Metaphor approach to the big picture, and minimalist on the little stuff…

I strongly disagree with section “TDD Promotes Microdesign over Macrodesign”. What you have written about is obviously a problem, but it doesn’t have any direct connection with TDD. If I was an architect, I would implement walls first and test it with the help of 1stFloorMock and 2ndFloorMock. Then I would continue with implementing floors.

The Testing should help programmer to make code fine and safe.
But in TDD Tester drives the programmer.

So TDD should enter into picture for a well matured software. I mean, either by algorithm point of view or customer point of view, the software development should have reached some satisfactory level. Then TDD helps.

But complete software development if you want to do with TDD, then you are daily blaming the programmer.

Because, a bug posted by you means you are pointing some fault in code for which programmer is responsible.

Finally, a testing team posts bugs. If you want buld complete software with bugs posted, then it will be a maintanance nightmare.

My original and continuing reaction to the idea of TDD is that it’s main value is to make the programmer confront the requirements of the unit being developed before doing the development, and to develop the unit with the requirements in mind. Included in that benefit is recognizing that the requirements may need to be developed before the unit should be started. The secondary value is to provide confirmation that changes haven’t broken the code.

TDD is going to be more efficient and useful when applied by an experienced developer who structures the code to support the testing, who understands what’s worth spending time creating tests for, and who knows from experience – theirs or someone else’s – how to start with a toolshed and end up with an industrial building.

It’s my understanding that TDD means Test Driven “Development”, not Test Driven “Design”. This argument feels as if the author wants them to happen simultaneously, rather than TDD being a separate reaction to design. Design is a whole separate thought pattern and activity. Development begins when the design is penned down. It is the developer’s responsibility to create the application that brings the design into reality.

Maybe my understanding is because I see design as not concept and idea. Using the “build a skyscraper” example, design is a comprehensive blueprint of construction. It is specifications and measurements of what is required. Deviation from blueprint will result in future tragedy.

I always saw TDD as a methodology that provides confirmation of intended results. Create tests that compare expectations (design) with reality, then raise a flag when the two are not the same. If the test results identify a failure of design, then it is back to the drawing board to determine why the design failed and to make the appropriate adjustments to all related aspects of the design. Its not a process of redesigning tests to compensate for incorrect design.

I’m also unable to understand how TDD is not suited for a library; TDD should be perfect for it. A library is something that should be well designed prior to development. Having a well developed test suite that nit-picks the library of functions is critical to future enhancement of the library. I’ve learned that all too well, all too many times.

TDD when used with Agile, is meant to provide assurances if the project stops at any point, that the code that DOES exist will work effectively.

Sure, future changes will require rework of previous code. And in a waterfall methodology, you’d save that rework effort.

But the practical truth of the matter, is that requirements ALWAYS change (well, 99% of the time… the remaining 1% belongs to NASA, and no, their specs and requirements don’t change), and it’s better for the BUSINESS for the coded solution to be responsive to those changes. The small investments in rework often pale in comparison to the huge gains in early deployments and scope change costs.

TDD has some definite merits, although I disagree with Beck’s idea that you should turn the knob on all good ideas to maximum.

I had a major impact on design at a well-known company about ten years ago by going through their code and asking the team, “How are you going to test this function?” In a lot of cases, the answer was, “I don’t know.” They had gotten into the habit of passing around a lot of very complex parameters.

Getting them to think ahead of time about testing made a lot of difference in the quality of the code — even without the discipline of composing the tests first.

Why is it that every so often some new ideas come around, the industry creates a hype and some tooling, tries to make everybody not part of the hype feel bad about their professional qualifications and then – as fast as it was built up it’ll be broke down by mostly the same crowd or the crowd that never got the message that a concept is a concept not a recipe for success.

I wholeheartedly agree with the statements, that TDD does not work for everything, in every environment or for everybody, just as SCRUM, Agile, KanBan, XP and all the other good ideas and concepts of the past 30+ years of my career, but they ALL WORK somewhere, for something and for somebody.

I’m so fed up with “preachers for the New and Only, but I look at the ideas and concepts they try to “promote” and have found that everything has something for somebody…”

I have seen way too many professionals that maneuvered themselves into a technological corner subscribing to the pure beliefs of the “NEW and BETTER” instead for trying it and finding out, if it works for them and is worth adding to their tool belt.

The often proclaimed “Evangelists” turn frequently out to be “one trick ponies” – One to rule them all…..

I’d rather hire at any time a well balanced professional with a “rich tool belt” than the usually overrated “Specialist” that can do pretty much one thing great but nothing else….

TDD works, if used as a tool and concept for the right use case. We use it for parts of our system that stick around for a longer time and warrant the effort, such as libraries, controls and components.

TDD becomes like every other concept borderline worthless and almost counterproductive when we try to use it on everything and with everybody. We have more success stories with TDD than the opposite, but we use it on a fraction of our code base and we use it, when it makes sense and not religiously AND I would never force a team member to use it…. it’s a tool and a concept, nothing more, nothing less and as professionals we should be able to use the tool that works for the case and FOR US…..

Efficiency can’t be prescribed or dictated as long as we work with human beings…
Ideas and concepts in our profession are usually not “new or revolutionary”. Chances are somebody has thought about them and tried to promote them before.
Looking back on the last 35 years it sure feels like it.

I will always promote diversity of skills over “specialization”, the secret of efficiency is to pick the right tool for the job, which includes that you look at those new ideas and concepts and adopt them as you go and need.

The worst reaction is usually to blindly subscribe to them or condemn them due to other peoples statements or trends. Think for yourself, see for yourself and make your own decisions……

TDD can work great when you have clear requirements at hand, for example creating a component with a defined set of functionalities and a planned longer shelf life. Writing tests first is great for those cases, because you know the expected outcome to the level of detail necessary.

Writing a test first for a trivial database call that you have done a million times and that will “vanish”, with the application in a few weeks is counterproductive, especially when you have other “old” tools available to test the results of your work.

We often write some tests for our components after the components are developed to make future changes and maintenance easier and more predictable. We also write tests for existing components that we are trying to refactor due to speed issues or upcoming change requests. I still consider that TDD, even though the “purists” will probably not agree with me.

First, many programmers have created too many confusing models within the programming technology.
They are seeing programming as an end in itself, meanwhile it is only a means to the end.
The analogy given by Otaku, Cedric using “a famous builder and architect” does not apply.

Many programmers do code-copying instead of actually programming. It is important to know that programming is an art.

The issue surrounding TDD among some programmers is nothing more than the money-bags who pay the pipers and the self-righteous programmers who think that one method solves all.

Contrary to David Heinemeier Hansson’s view, every system requires both unit and system testings. When you are developing a software device that actually controls a physical device, this is where you see the interplay.

Programming is more like composing a musical piece and then singing it. How often does a musician role out new releases? Every programming issue has its solution approach.

Architecture Impact : Most of the time, you design an architecture to fit performances and features requirements. Testability has NEVER been a end-user requirement. Quality is a user requirement, but not Testability. And on many projects I’ve been working on, testability has a negative impact on architecture. So I think that the value of testability is equals to the lack of value or development team….Think about your car / Have you ever ask to you reseller for the testability of each component ? No. You need a good car. Point.

* over quality : Here, some of you speak about TDD with Agile…Agile is usually added value centric. But, have you ever think about the value of test ? Actually, a test has a negative value for a product (cost, but has no effect) UNTIL it fails. Then the value goes to zero. The value become positives after severals failures during application lifecycle.
The question is now : how tests will never failed ? Using TDD, most of tests will always succeeded, excepts small parts of them, of if your developer team is definitely the wrong one. Then, your development cost will increase without any extra feature that you will be able to sell to your customer…

Subsidiary question : What about beginning quality by developer training and coaching ?

TDD is a development technique and not methodology. As such there are no defined parameters about how much or how little up-front design is required. This article seems to assume that TDD prescribes emergent design, without any up-front design. I’ve never found that to be the case. One could argue that a methodology like XP should favor more up-front design, and suffers by not having as much as waterfall. Those sorts of choices should depend more on the team, the product and the market they want to deliver it to.

“TDD Is Hard to Apply”? Not if the design is good. The problems that Beck and others have used are simple… true, but what books successfully explain similar principles with complex examples? What’s more, part of the point of TDD is to use it as a tool to guide you in decomposing a complex problem into more simple ones.

Just like you shouldn’t listen to anyone who says they have the one true way, you should also not listen to absolute statements like “Don’t test first”.

I agree with you on your points. The programmer needs to balance the business needs against the quality of his system. If the business needs the software to be delivered in a timely manner at a low cost, some of the quality testing needs to be sacrificed. If you have the luxury of time to get your product out, then you can add better quality checks, even TDD and make your software as complex as needed.

TDD is just a technique, and like all things in software its useful in some cases and harmful in others. Once you learn how to do it, you need to learn when it applies and when it does not. Theres no silver bullet… http://en.wikipedia.org/wiki/No_Silver_Bullet

btw, you can basically swap “TDD” in my above statement for most any other practice in software

Most of the arguments in the article have been addressed by the “Specification by Example” ideas.

I disagree TDD means I have to follow a specific methodology during development. All it says I believe is to let tests guide you, which is possible when you know what the end result should look and work like. If you don’t you can still explore ideas and tests are a great help then as well, if you want to use them.

My previous boss also likes TDD based from the good things he reads from the internet. Our project is in a very tight deadline and he wants me to implement TDD. The first thing i told him is to extend the deadline. I have a couple of experiments with tdd and i find it more time consuming to think of test first to drive the design. Maybe my mind isn’t wired for this. What i usually do is to code first then test it via unit testing then refactor. I can work much faster this way.

In my opinion, tdd is extreme. The advantage is that it enforces you to do unit tests. If you want to do tdd, do it with a little bit of common sense. If you know that a method is too simple and you are very confident of behavior like adding two numbers, just go straight and do the code then test it with unit testing. No sense to waste time to do ” red green refactor ” for very easy items.

Don V. Nielsen wrote:
> It’s my understanding that TDD means Test Driven “Development”, not Test Driven “Design”. This argument feels as if the author wants them to happen simultaneously, rather than TDD being a separate reaction to design.

This is not Cedric’s claim, but actually what TDD proponents say: that tests ARE design and should drive it, and also that they should come FIRST.

A lot of people defending TDD are actually defending different and orthogonal things such as unit and regression testing. Nobody I know is arguing against unit test. Unit testing != TDD. TDD is — if we believe its more vocal proponents — not about testing but about *design*. THIS is what a lot of us are reacting against, and what the infamous Sudoku debacle exposed as bullshit.