Monday, May 5, 2014

Testing, Factoring, Flow

This is a post (mostly) defending Test Driven Development. I'm a little surprised at myself for writing this. I've gotten pretty weary of testing at various times in my career. And yet, here I am, a proponent of TDD.

As far as I know there are basically two claimed benefits of TDD. Note that there are a lot more benefits to having a suite of regression tests, which is something you can have regardless of whether or not you do TDD. Some of the articles I have read defending TDD have focused on the benefits of automated tests, which in my mind is a separate issue. Lots of people write regression tests who do not use TDD. It is possible that TDD users write more tests than people who write their tests after the code is working. This seems plausible to me, but I have no way of verifying it, so let's leave it out of the discussion for now. I'm more interested in TDD as a technique for generating code than a technique for generating tests.

One caveat I should put up front here is that I am not that familiar with the primary sources on TDD. I have not read much of Kent Beck's work, in particular. I have done quite a lot of TDD and have at times been successful at it, which is my only real credential here.

The two claims are that TDD results in simpler, better-designed programs, and that TDD makes a programmer more productive. The first claim is dubious on the face of it. A weaker form would be to say TDD produces programs that are easily tested, but I think this claim is even more dubious. In the recent controversies around TDD I haven't really heard anything about the latter claim, which is what inspired me to write this. I find that TDD really can, under certain circumstances, make me much more productive. I'm going to take the claim of productivity first, and then circle back around to the claims about design.

Flow

Nearly any task is made easier by being able to see what you are doing. In programming, you will be able to get your code working much faster if you are able to run your code frequently while you are writing it. This is often described as a feedback loop, where you perform repeated cycles of editing the code and then running it to see if it does what you expect. There are multiple ways to construct this kind of feedback loop. One simple way is to test the program manually. Often this is sufficient. If you are styling a web page, for example, you can simply refresh the page in a browser. If you are writing a command-line tool you may just keep a terminal open and keep running the test command to see the output. All of this is obvious and natural to anyone who has done much programming at all.

Another thing you have probably experienced at different times is what is sometimes described as flow. The idea is that if you are performing a task with clear goals and instantaneous feedback, you can enter a state of intense focus that simultaneously feels good and allows you to be extremely productive. Without getting too deep into the psychological theory, this certainly sounds like the kind of thing I have experienced when programming. In fact, that kind of experience is what made me want to make a career in programming. And many of the times I have achieved this kind of focus have been when I was using TDD.

While there are lots of ways to establish a feedback loop, the advantage of TDD is that it can theoretically work on any kind of code. Manual testing works well for UIs, and in my experience it is easier to achieve flow this way than trying to write UI tests. However, I work on a lot of code that is not UI code, and here manual testing is much more difficult, and writing tests should be relatively easy. In this kind of situation I find TDD to be an extremely valuable technique.

In order for this to really work, I have found that I need three ingredients:

The tests have to be pretty easy to write. If you are spending hours at a time wrestling with getting a test to do what you want it to, you are never going to achieve flow. You are just going to make yourself hate testing.

The tests need to run fast. To achieve flow you want to ideally have instantaneous feedback. In my experience even smallish differences (1 second vs. 2 seconds) make a significant difference on my level of focus.

The test cases need to build on each other incrementally. This isn't a feature of TDD, but rather a skill you need to learn to be effective at TDD.

It is probably worth saying a little more about the speed thing. A lot of people talk about ways to make their tests run faster. Sometimes when I hear myself talking about test speed it just sounds spoiled. Why spend time optimizing code that won't ever be run by an end user? The answer is that fast tests boost developer engagement in a way that produces both productivity and morale. I don't care about how fast the whole regression suite is. A properly configured CI server can completely eliminate this issue. But even with the ability to run a targeted subset of the tests, the test run is often too slow for me to achieve flow. I see making test runs near-instantaneous to be an extremely worthy goal, especially at the framework level. (Making manual testing of UI code similarly fast is also a worthygoal.)

Factoring

As to whether TDD results in better design, I think this claim is probably to nebulous as stated to really be evaluated. For one thing, we will need to agree on what is a good design before we can determine whether TDD produces good designs. (One data point we do have is that TDD seems to produce the kinds of designs that David Hanson dislikes, which I guess I count as a win for TDD. :p)

I suppose a simplistic kind of argument would be that good design requires decoupling, and starting with tests will force this decoupling. However, there is a limit to how much decoupling is actually a benefit before we begin to drown under the weight of indirection. Neither is the answer simply to have the right amount of coupling. We must make wise decisions about how to divide up responsibilities. The right abstractions can leave us with a codebase that is easy to reason about, and to extend. It is exceedingly common for the wrong set of abstractions to leave us with something very different.

For sake of discussion, let me offer two simple principles that I believe lead to the right kinds of abstractions. First, separate pure functions from mutable state. Second, separate mechanism from policy. What's interesting is that both of these accord with testability. They make for better designed software, and they also make for more testable software. So this lends some credence to the idea that TDD can lead to better design.

I do have some skepticism about this, though. For one thing, it seems like there could possibly be counterexamples. If you could demonstrate a design that was easier to test, but was actually a worse design for some other reason, that would greatly reduce the power of this argument. Of course some people sincerely believe this about, say, service layers. But those people are wrong. :)

My other bit of skepticism is about whether TDD can actually lead someone to principles like the ones I mentioned without them already knowing about them. But it does seem that if someone is persistent in pursuing TDD-flow, they will have to seek out these kinds of solutions. Either they will find a better design, or they will fail to enjoy TDD. It may be that TDD functions more as an indicator of whether a design is good or not than as a technique for discovering a better design.

Conclusion

I have found TDD to be an extremely valuable technique when flow can be achieved. Unfortunately it has often been impossible for me to achieve flow because of poor framework support. In particular, a framework with good support for regression testing (such as Rails) can still have pretty poor support for what I have called TDD-flow. I have also experienced the burden of trying to do TDD when the ingredients of flow are not available. In general I have found TDD to not be worthwhile under these circumstances, but I do think it is worthwhile to change things if possible so that TDD works better. In cases where flow can be achieved through some other feedback loop (manual testing) I don't tend to do a lot of TDD. I think in this case it depends a lot on how much you value the regression tests you would generate if you were to drive UI development through tests. (Whether systemic tests have no value is a discussion for another time.)

I feel like TDD has been somewhat oversold on the benefit of generating good designs and massively undersold on the benefit of enhancing programmer focus and productivity. This may just be a trick of my memory or an accident of what I have read. At any rate, it is something of a paradox that one person would derisively declare that TDD is dead at the same time that another person would use TDD to feel focused and energized, fully involved in the task at hand, or, in a word: alive.