January 2, 2010

Test-Driven Development versus Design By Contract

“Currently I’m making my mind up on DbC and the synergy with TDD and other paradigms again, so stay tuned for further blog posts on this topic.”

I am also thinking about how I prefer to combine Design By Contract with Test Driven Development. I am comfortable with both. And I use them both. So far, I seem to prefer to start with a test to drive the design, then add contracts later after the design stabilizes a bit. I have not given much thought yet to the use of Pex. That could change the development process even further. I do intend to investigate it after watching the excellent video from PDC 2009 http://microsoftpdc.com/Sessions/VTL01 “Code Contracts and Pex: Power Charge Your Assertions and Unit Tests”. I prefer this for the reasons that people like TDD:

writing the test first, ensures testability

writing the test as your first example of a call ensures a simple API for when your production code calls the new software

writing test first guarantees code coverage. Not just code coverage, but scenario relevant code coverage. This ensures your code has been tested for the anticipated scenarios.

a related item is that TDD ensures you have an automated test. And automated tests are powerful documentation for developers to understand code. They can be written after the code is done. But if you intend to have a test anyway, why would you not write it first?

But once my classes begin to take shape, I don’t mind adding contracts to clarify my intent, even if I’ve not written tests yet to capture the rules represented by them. This might offend a TDD purist. But I do a cost/risk assessment and compare the cost of writing a test with the risk that I will regret not enjoying one of the benefits of TDD listed above.

Regarding #1 – TDD ensures testability

I can’t recall ever adding a contract and then thinking “Gee, because I didn’t write a test first, I have made a mess that is not testable.” But I am certainly sensitive to this. If I ever added a contract and then thought “Gee, that might hinder testability,” then I would reconsider in that case. Or, if I ever learn this lesson the hard way, with pain and hindsight, then I will revise my thinking. To me, the overall class design and pattern affect testability, and I don’t believe that contracts have a dramatic influence on those.

Regarding #2 – Ensure friendly API calls

Contracts don’t usually change the signature or pattern. They just set limits and clarify how the pattern you have designed should work. As I said before, I prefer to use TDD to establish the initial design to satisfy the happy path. I may even use more TDD to elaborate on prominent alternate paths. But at that point, the design is fairly stable.

Regarding #3 – TDD ensures code coverage and automated tests as examples of using the code

Now, this one gets my attention and starts to get interesting. Consider a method like this

Line 2, Contract.Requires(databaseConnectionString != null), is an internal matter. It reflects something this method needs to work correctly. It corresponds to technical considerations only. If you call this method with a null databaseConnectionString, you have made an error. I want you to know that as soon as possible. I don’t feel the need to write an automated test for this. But I DO feel the need to add the precondition.

Line 3, Contract.Requires(amount > 0,…), appears similar to line 2, but is very different and should be treated differently. Why?

Line 3 is an external matter. It expresses a business rule that should come from customer requirements. It should be validated by a customer in some fashion. It should have a written test to ensure that we properly handle the customer’s needs. It should have test code that illustrates the approved way to behave when amount <= 0. This is appropriate under our software development methods in my company because we use these automated tests as a communication tool between me and the tester who works on our team. She helps review tests and guide our testing in accordance with the specifications. In theory, if we would learn to rely on the documentation of the contracts, we might be able to use that as evidence. But we still would be missing the unit test as a demonstration of how the caller should address this situation.

In this case, the proper way for code to call this method is as follows. In accordance with the principles of contracts, the caller must satisfy all preconditions. If the caller has responsibility for acting on the alternate path, then it should do something like this. Notice that I provide a fake method “DisplayMessage” in my test class to guide other programmers as to what our intent is in this situation. I could be persuaded to leave out the test case for this exception path if we can address the fundamental benefits of validating requirements and guiding developers in the use of the code.

Another consideration is related to code coverage metrics. If we set check in policies or have management reviews of code coverage to ensure adequate testing is occuring, then making exceptions in cases like this may interfere with those metrics and require us to try other approaches to ensure adequate testing. That would indicate a limitation in the way we measure “adequate testing”, not a defect in the practice of relying on contracts in place of tests. However, we don’t have such a system in our company today, so it’s a moot point for now.