Monday, June 11, 2012

Fuchu: a functional test library for .NET

I explained how many concepts in typical xUnit frameworks can be more simply expressed when tests are first-class values, which is not the case for most .NET and Java test frameworks.

More concretely, test setup/teardown is a function over a test, and parameterized tests are... just data manipulation.

Since first-class tests greatly simplify things, why not dispense with the typical class-based, attribute-driven approach and build a test library around first-class tests? Well, Haskellers have been doing this for at least 10 years now, with HUnit.

HUnit organizes tests using this tree:

-- | The basic structure used to create an annotated tree of test cases.
data Test
-- | A single, independent test case composed.
= TestCase Assertion
-- | A set of @Test@s sharing the same level in the hierarchy.
| TestList [Test]
-- | A name or description for a subtree of the @Test@s.
| TestLabel String Test

Where Assertion is simply an alias for IO (). This is all you need to organize tests in suites and give them names.

Actually, I first ported HUnit (including this DSL), then discovered that MbUnit has first-class tests and later wrote the DSL around MbUnit. Everything I described in those posts (setup/teardown as higher-order functions, parameterized tests as simple data manipulation, arbitrary nesting of test suites) applies here in the exact same way.

which turns out to be very similar to the tree we translated from HUnit, only the names are embedded instead of being a separate case.

I called this HUnit port Fuchu (it doesn't mean anything), it's on github.

Assertions

Fuchu doesn't include any assertion functions, or at least not yet. (EDIT: assertions were added in 0.2.0) It only gives you tools to organize and run tests, but you're free to use NUnit, MbUnit, xUnit, NHamcrest, etc, or more F#-ish solutions like Unquote or FsUnit or NaturalSpec for assertions.

Tighter integration with FsCheck is planned. (EDIT: it was added in the first release of Fuchu)

Runner/Tooling

As with HUnit, the test assembly is the runner itself. That is, as opposed to having an external test runner as with most test frameworks, your test assembly is an executable (a console application). This is because it's more of a library instead of a framework. As a consequence, there is no need of installing any external tool to run tests (just hit CTRL-F5 in Visual Studio) or debug tests (just set your breakpoints and hit F5 in Visual Studio). Here's a clear signal of why this matters:

This function defaultMainThisAssembly does exactly what it says on the tin. Notice that it also takes the command-line args, so if you call it with "/m" it will run the tests in parallel. (Curiously, you can't say let main = defaultMainThisAssembly, it won't be recognized as the entry point).

By the way, this is just an example, you wouldn't normally annotate every single test with the Tests attribute, only the top-level test group per module.

If you reference the assembly under test in the REPL, fsi.exe blocks the DLLs, so you have to reset the REPL session to recompile. But if you're testing F# code, you can work around this by loading source code instead of referencing the assembly.

Other tools

Integrating with other tools is not simple. Most tools out there seem to assume that tests are organized in classes, and each test corresponds to a method or function. This also happens with MbUnit's StaticTestFactory: for example in ReSharper or TestDriven.Net you can't single out tests. Still, they can be made to run let-bound tests (which may be a test suite), so it should be possible to have some support within this limitation.

Also, no immediate support for any continuous test runner. I checked with Greg Young, he tells me that MightyMoose/AutoTest.NET can be configured to use an arbitrary executable (with limitations). Remco Mulder, of NCrunch, suggested wrapping the test runner in a test from a known test framework as a workaround. Maybe executing the tests after compilation (with a simple AfterBuild msbuild target) is enough. I haven't looked into this yet.

Coverage tools should have no problem, it makes no difference where the executable comes from.

Build tools should have no issues either; obviously FAKE is the more direct option, but I see no problems integrating this with other build tools.

C# support

I threw in a few wrapper functions to make this library usable in C# / VB.NET. Of course, it will never be as concise as F#, but still usable. I'm not going to fully explain this (it's just boring sugar) but you can see an example here.

NUnit/MbUnit test loading

Even though it may seem very different, Fuchu is still built on xUnit concepts. And since tests are first-class values, it's very easy to map tests written with other xUnit framework to Fuchu tests. For example, building Fuchu tests from NUnit test classes takes less than 100 LoC (it's already built into Fuchu)

This lets you use Fuchu as a runner for existing tests, and to write new tests. I'm planning to use this soon in SolrNet to replace Gallio (for example, Gallio doesn't work on Mono).

There is a limitation here: Fuchu can't express TestFixtureTearDowns. It can do TestFixtureSetups (and obviously SetUp/TearDown, as explained in previous posts), but not TestFixtureTearDowns (or at least not unless you treat that test suite separately). Give it a try and see for yourself :) . Is it a real downside? I don't think so (for example, TestFixtureTearDowns make parallelization harder), but it's something to be aware of. Also I haven't looked into test inheritance yet, but it should be pretty easy to support it.

Conclusions

Does .NET really need yet another test framework? Absolutely not. The current test frameworks are "good enough" and hugely popular. But since they don't treat tests as first-class values, extending them results in more and more complexity. Consider the lifecycle of a test in a typical unit testing framework. Inheritance and multiple marker attributes make it so complex that it reminds me of the ASP.NET page lifecycle.

What I propose with Fuchu is a hopefully simpler, no-magic model. Remember KISS?

9 comments:

When in doubt, roll your own. :) This looks great. It should hook into a file system watcher I just wrote [1]. I had different plans for my little script, but this would be a good use to continuously run the tests.

@Ryan: I've asked myself too if this isn't just a case of NIH. But I'm not quite comfortable with the current state of things, and so at least I want to see where this approach leads to.The filewatcher you mention sounds great, perhaps we should strive to build something like sbt (e.g. http://devblog.point2.com/2009/07/27/scala-continuous-testing-with-sbt/)

I would like to look closer into this. I am quite interested in a first-class test framework with an F#-y surface syntax. Are these features available (or are you interested, in the longer term, in building these - I can perhaps pitch in):

1. Gallio integration - this gives a lot of runners / presenters for free, and, say, AppHarbor uses Gallio. Shouldn't be hard in theory? Converting between first-class tests is just a function right? If Gallio has problems on Mono - that's fine, just release a separate Fuchu.Gallio package that does this integration.

2. WebSharper support (with QUnit backend for reporting?) - this involves some constraints on operator overloading, but it would be nice to use the same syntax for tests in that run in JavaScript. Can replace this: http://bitbucket.org/IntelliFactory/websharper/src/863b99b87d95ab5644b6d8388d94bcc4e38b4295/IntelliFactory.WebSharper.Testing?at=default

1. Yes, mapping tests to MbUnit should be straightforward. I already have mappings in the opposite direction for NUnit and MbUnit. Mapping to MbUnit would require a StaticFactory as an entry point for Gallio.

2. WebSharper.Testing looks very interesting, are there any examples using it? No idea what it would take to enable WebSharper support. Also just in case note I dropped the operators in favor of simple named functions. See the project README https://github.com/mausch/Fuchu#readme

Thanks! I am excited. I hope I will find time to try it out this week.

If you are not using aggressive overloading, WebSharper integration will be simple - it will take annotating code with `[]` and running WebSharper.exe as a post-compile step. I can take care of that on a fork when I get a chance.

Of course we are using WebSharper.Testing to test WebSharper! :) https://bitbucket.org/IntelliFactory/websharper/src/863b99b87d95ab5644b6d8388d94bcc4e38b4295/IntelliFactory.WebSharper.Tests?at=default