2004 (28)

Writing the tests for FluentPath

Writing the tests for FluentPath is a challenge. The library is a wrapper around a legacy API (System.IO) that wasn’t designed to be easily testable.

If it were more testable, the sensible testing methodology would be to tell System.IO to act against a mock file system, which would enable me to verify that my code is doing the expected file system operations without having to manipulate the actual, physical file system: what we are testing here is FluentPath, not System.IO.

Unfortunately, that is not an option as nothing in System.IO enables us to plug a mock file system in. As a consequence, we are left with few options. A few people have suggested me to abstract my calls to System.IO away so that I could tell FluentPath – not System.IO – to use a mock instead of the real thing.

That in turn is getting a little silly: FluentPath already is a thin abstraction around System.IO, so layering another abstraction between them would double the test surface while bringing little or no value. I would have to test that new abstraction layer, and that would bring us back to square one.

Unless I’m missing something, the only option I have here is to bite the bullet and test against the real file system. Of course, the tests that do that can hardly be called unit tests. They are more integration tests as they don’t only test bits of my code. They really test the successful integration of my code with the underlying System.IO.

In order to write such tests, the techniques of BDD work particularly well as they enable you to express scenarios in natural language, from which test code is generated. Integration tests are being better expressed as scenarios orchestrating a few basic behaviors, so this is a nice fit.

The Orchard team has been successfully using SpecFlow for integration tests for a while and I thought it was pretty cool so that’s what I decided to use.

Consider for example the following scenario:

Scenario: Change extension
Given a clean test directory
When I change the extension of bar\notes.txt to foo
Then bar\notes.txt should not exist
And bar\notes.foo should exist

This is human readable and tells you everything you need to know about what you’re testing, but it is also executable code.

What happens when SpecFlow compiles this scenario is that it executes a bunch of regular expressions that identify the known Given (set-up phases), When (actions) and Then (result assertions) to identify the code to run, which is then translated into calls into the appropriate methods. Nothing magical. Here is the code generated by SpecFlow:

The #line directives are there to give clues to the debugger, because yes, you can put breakpoints into a scenario:

The way you usually write tests with SpecFlow is that you write the scenario first, let it fail, then write the translation of your Given, When and Then into code if they don’t already exist, which results in running but failing tests, and then you write the code to make your tests pass (you implement the scenario).

In the case of FluentPath, I built a simple Given method that builds a simple file hierarchy in a temporary directory that all scenarios are going to work with:

As you can see, the When attribute is specifying the regular expression that will enable the SpecFlow engine to recognize what When method to call and also how to map its parameters. For our scenario, “bar\notes.txt” will get mapped to the path parameter, and “foo” to the newExtension parameter.

And of course, the code that verifies the assumptions of the scenario:

These steps should be written with reusability in mind. They are building blocks for your scenarios, not implementation of a specific scenario. Think small and fine-grained. In the case of the above steps, I could reuse each of those steps in other scenarios.

Those tests are easy to write and easier to read, which means that they also constitute a form of documentation.

Oh, and SpecFlow is just one way to do this. Rob wrote a long time ago about this sort of thing (but using a different framework) and I highly recommend this post if I somehow managed to pique your interest:
http://blog.wekeroad.com/blog/make-bdd-your-bff-2/

10 Comments

Maybe I'm less "pure" in terms of TTD-ness but I don't see a problem with testing against System.IO. It's part of the framework, and I don't see people trying to abstract other framework things like collections or streams. Sure, you may want to abstract bigger framework APIs, but I guess it's a matter of personal taste whether you put System.IO in the "big stuff" or the "small stuff". And since I don't really think the side effects of the tests (manipulating the file system) are that bad, I'd consider it to be "small stuff" and I wouldn't lose sleep over it.
Now it's interesting that if you follow my opinion here, then unit tests and integration tests end up being exactly the same... which basically means that your library is unitary to begin with. And I think it's the case, so it's all good to me.

@Ludovic: yeah, I don't want to split hair. The FluentPath integration tests run fast enough that testing against the actual file system is not a problem, but if I had a bigger system with hundreds of such tests, or if I was testing against a slower, non-mockable system (let's say something remote for example), I'd be in more trouble. All this to say that my silly little library may not expose the badness of the methodology, but more serious stuff will for sure.

@Simon: Moles look interesting, thanks for the pointer, but from the manual it can't mock static methods, which is pretty much all that we use in System.IO. It's not clear why static methods are out of the picture though: the way they do things (by rewriting the code), I don't see why you couldn't re-route static calls.