Choosing a BDD framework for .Net

Specs are written by developers: my personal recommendation NSpec. Runner up StoryQ.

Introduction

BDD or behavior driven development is the practice of writing an executable testable specification that describes the application’s behavior. This specification is often written in a fluent interface, a DSL or in plain English (or rather close to plain English).

While in TDD the focus is on writing tests that single out units of the application, BDD is focused on writing tests on the behavior of the application. This can also be tought as testing features of the application.

Before presenting the frameworks I will make a distinction between xBehave and xSpec style frameworks.

xBehave

xBehave frameworks are about writing user level stories in a form comprehensible by anyone. These stories can be written by the users themselves or by a group consisting of developers, users and testers.

These framework typically use a story defined in a DSL close to English and then map this story to a test written in code by the developers. Cucumber is a well known example of such a framework.

xSpec

xSpec frameworks are usually about developers writing tests in code using an approach that favors testing behavior and functionality. These tests are usually closer to unit tests in both appearance and granularity but feature some differences.

Both approaches could be combined on a single project. The user stories could be converted to xBehave tests, while the developers could adopt an xSpec framework to replace or complement xUnit tests.

Available frameworks

Here is a compilation of xSpec and xBehave BDD frameworks for .Net. This list is current as of the writing of this post and while I tried my best to find most frameworks, I realized while compiling this list that there are just too many, I am sure I must have missed a few.

By the way, it’s not an error in the table, I have found two frameworks named NSpec which appear to be totally independent. The first one listed is the most active and most popular. The second one is an older project that hasn’t seen any recent activity.

*** This framework is harder to classify. Based on my interpretation of xSpec/xBehave and my comprehension of the framework, I personally place it in a gray zone. They suggest to start with a plain text story but it isn’t directly used by the code. The specifications are directly in C# test code contrary to other more “pure” xBehave frameworks. It is a sort of hybrid and does tend to lean much more on the xSpec side than on the xBehave side.

Narrowing the selection

First of all, I would eliminate the following frameworks from investigation: specunit-net, NSpec (older project), NSpecify and .NetSpec as they are abandoned inactive projects with most never having really made it to “production status”.

NaturalSpec is in F# which is great for those doing F#. There is a link to a post on it’s GitHub page about how to use it with C# objects, but I would personally prefer a library that targets C# for a C# project.

I wasn’t able to see much activity for BDDish (last commit to GitHub more than 2 years ago, only 3 questions on StackOverflow, not much documentation or examples). I would also skip this one.

As an unrelated side-note, out of the remaining more serious contenders xbehave.net and SubSpec are strikingly similar. At first glance, both projects seem to be closely related. If you are considering one of these two, you might as well look into the other as well.

Making a selection for an xBehave framework

The first question you need to ask yourself is if you want to do xBehave style tests in a language closer to plain text, with specs that can be written by non developers.

If that is so, your choice is pretty slim. Your choice is between SpecFlow and NBehave.

StoryQ could also be considered if SpecFlow or NBehave are not satisfactory.

Both SpecFlow and NBehave use Gherkin (the DSL used by Cucumber) to write their specs. Both store specs in *.feature files. Both also use attributes on methods when writing tests.

[Given("I am not logged in")]
public void LogOut()
{

Both have documentation available on GitHub.

SpecFlow seems to have more documentation and it is also recommended in this StackOverflow question in the top two rated answers.

On the other hand the SpecFlow doc is pretty dry on code examples. I personally preferred NBehave’s documentation over SpecFlow.

Making a selection for an xSpec framework

If you do not need for your specs to be written in a plain text like format and want to save time and work use an xSpec framework. You won’t need to write the binding code and writing tests will be easier for those already familiar with an xUnit framework.

On the xSpec side, you have a lot of options.

I narrowed my search to the following frameworks: MSpec, NSpec, SpecsFor, StoryQ and xBehave.net.

The following table compares those frameworks by considering their total number of questions on StackOverflow. I also rated the official documentation on a scale of 1 to 3, with 1 being the poorest and 3 the best score. This rating is largely subjective and represents my appreciation of the available documentation. It should be noted I prefer a style which includes code samples and which gets you rapidly on your feet.

Name

Number of questions on StackOverflow

Documentation

MSpec

519*

1 / 3

NSpec

62

3 / 3

SpecsFor

5

3 / 3

StoryQ

36

3 / 3

xBehave.net

5

2 / 3

*: I added the number of questions for both MSpec and Machine.Specifications. Even if just looking at the number of questions for either MSpec or Machine.Specifications, it is the clear winner here.

My personal recommendation for an xSpec framework is NSpec.

After trying each of these frameworks for a simple two tests scenario here is my personal rating:

Story is A new tic tac toe board is empty
In order to to start a new game
As a player
I want to have an empty board
With scenario new board
Given a new board => Passed
When checking if board is empty => Passed
Then it should be true => Passed
Story is After placing a first X
In order to to make my first move
As a player
I want to have a non empty board
With scenario new board
Given a new board => Passed
And placing a first x => Passed
When checking if board is empty => Passed
Then it should be false => Passed

I tried to emulate the style in the StoryQ examples but I’m not sure if I am using StoryQ as efficiently as possible.

NSpec

Pros:

concise

easy to get started with

makes me think of RSpec

it kicks ass

Cons:

can’t use MS test runner

must manually rebuild test library before each NSpecRunner run

tests will fail if some conventions aren’t followed (ie: underscores in method names)

10 thoughts on “Choosing a BDD framework for .Net”

BDD is nice in theory but in practice (in the development of LOB application), in my opinion, it suffers from the same problems as tools generating code from UML and the likes. In essence, “BDD tests” only validate your comprehension of the specifications, and tests that in the end your code matches your specifications and not the client’s. So having to write more code first, before showing the result to the client and having to refactor more code every time the comprehension between the client and coder changes.

I still believe that the most basic definition of agile means the shortest amount of time should be spent between receiving the specs from the client and showing them your resulting comprehension in the form of working software he can play with, while still producing quality code through practices like TDD. If the argument is turned around, saying that the client/analyst/business analyst should write the xSpec, I would only respond that these actors will start using BDD tools the same day they start using our Issues and Bugs Tracking system, which is never.

Having said this, every software framework in existence should have it’s suite of BDD tests. When the specs come from the coder, the best tools for any coder to understand how and when to use your code should be written in a language that coders understand.

Documentation, specifications, UML et al, xSpec etc… are all trying to unite and unify the intangible ideas of the client to the tangible world of the digital software and are all failing in the same manner. In the end they go contrary to the essence of agile which is to let go deterministic specifications, and adopt a more open ended world.