Introducing BDDfy: the simplest BDD framework for .Net

This article is an introduction to installing and using bddify - a simple to use and extend BDD framework for .NET developers.

Introduction

[Update: 2013-09-23]: At the time of writing this article the framework was called bddify. Since then it has gone through significant changes and has improved a lot. The framework is now called BDDfy, is part of TestStack organization and lives in GitHub. You can also read the latest documentation on the documentation website.

BDD is a very interesting concept. It is amazing how a rather small mindset shift can provide such a big difference. If you have not heard about BDD or would like to refresh your understanding, you may want to have a quick look at Dan North's article here. While you are at it, you may also read his article about what is in a story. I use one of the examples in his article for my code here.

There are quite a few open source .NET BDD Frameworks like StorEvil, SpecFlow and StoryQ. This article, however, is about a new BDD framework for .NET called BDDfy (pronounced B D Defy)! The name comes from the fact that this framework allows you to BDDfy your tests very easily, which means you could turn your tests into BDD behaviors simply. BDDfy is now a verb!!

BDDfy can run with any testing framework. It does not force you to use any particular framework. Actually BDDfy does not force you to use a testing framework at all. You can just apply it on your POCO (test) classes!

BDDfy does not need a separate test runner. You can use your runner of choice. For example, if you like nUnit, then you may write your BDDfy tests using NUnit and run them using NUnit console or GUI runner, Resharper or TD.Net and regardless of the runner, you will get the same result.

BDDfy can run standalone scenarios. In other words, although BDDfy supports stories, you do not necessarily have to have or make up a story to use BDDfy. This is useful for developers who work in non-Agile environments but would like to get some decent testing experience.

You can use underscored or pascal or camel cased method names for your steps.

You do not have to explain your scenarios or stories or steps in string, but you can if you need full control over what gets printed into console and HTML reports.

BDDfy is very extensible. In fact, BDDfy core barely has any logic in it. It delegates all its responsibilities to its extensions.

Using BDDfy, it is easier to switch to BDD. So if you are on a project with a couple of hundred tests already written and you think using BDD could make your tests more valuable, then BDDfy can help you with that. You are still going to need to make some changes; but hopefully they will be minimal.

Prerequisites

BDDfy runs on .NET 4 and .NET 3.5. The samples in this article are coded using Visual Studio 2012. I am also going to use MSTest as a testing framework and Resharper test runner to run the tests.

ATM Sample

Forget the bullet points: let's see BDDfy in action. I am going to use Dan North's ATM sample for this. I will copy his sample here for your convenience:

Story: Account Holder withdraws cash

As an Account Holder I want to withdraw cash from an ATM So that I can get money when the bank is closed

Scenario 1: Account has sufficient funds Given the account balance is $100 And the card is valid And the machine contains enough money When the Account Holder requests $20 Then the ATM should dispense $20 And the account balance should be $80 And the card should be returned

Scenario 2: Account has insufficient funds Given the account balance is $10 And the card is valid And the machine contains enough money When the Account Holder requests $20 Then the ATM should not dispense any money And the ATM should say there are insufficient funds And the account balance should be $20 $10 And the card should be returned

Scenario 3: Card has been disabled Given the card is disabled When the Account Holder requests $20 Then the ATM should retain the card And the ATM should say the card has been retained

ATM Solution

Let's do some coding. I start by creating a new project called 'BDDfy.Samples.Atm'.

BDDfy also copies a file called 'BDDfy.ReadMe.txt' in your project root folder. This file explains a bit about how BDDfy works as well as some of its conventions.

Scenario 3: Card Has Been Disabled

Without further ado, let's write our first scenario. I am using the last scenario for this sample because it is simpler than other scenarios and we can focus more on BDDfy than on the scenario's implementation:

This class represents our scenario and has one test method on it called Execute (it can be called anything). Inside this method, I have one line of code that calls BDDfy extension method on the class. Let's run this test to see what happens. I am using ReSharper test runner to run the test:

Figure 1. CardHasBeenDisabled console report before the scenario is implemented

That is the console report BDDfy generates. Note that BDDfy tells you that the 'Given' step has not been implemented yet and the other steps were not executed.

By default, BDDfy also generates an HTML report called 'BDDfy.Html' in your project's output folder:

Figure 2. CardHasBeenDisabled Html report before the scenario is implemented

HTML report shows the summary on the top and the details on the bottom. If you click on scenarios, it also shows you the steps of that scenario along with the step result (and in case of an exception, the stack trace).

Note: As indicated in HTML and console reports, 'Given' step was unsuccessful due to the exception. When there is an exception in 'Given' or 'When' steps BDDfy will not run the remaining steps. It is shown in the console report with '[Not Executed]' in front of steps and in the HTML report with 'Not Executed' icon. This is because if your 'Given' or 'When' steps fail, there is no reason to run other steps. This rule does not apply to asserting steps (i.e. 'Then' parts) which means that you could have three asserting steps with one of them failing and the other two passing. In this case, BDDfy runs all the steps and shows you which of your assertions failed.

How Does BDDfy Find the Steps?

BDDfy uses reflection to scan your bddified classes for steps. In this running mode, it has two ways of finding a step: using attributes and method name conventions. You find the complete list of conventions in the 'BDDfy.ReadMe.txt' file. Out of those conventions, the following are used in this article. Method names starting with the following words are considered as steps:

Given: setup step

AndGiven: setup step running after 'Given' steps

When: state transition step

AndWhen: state transition step running after 'When' steps

Then: asserting step

And: asserting step running after 'Then' steps

How Does It Generate the Scenario and Step Titles?

BDDfy uses method names to generate the step titles (You may write your methods using camel or pascal casing or you may use underscores) and uses the scenario class name to generate the scenario title.

As mentioned above, BDDfy does not stop the execution when there is an exception on your asserting steps. In this case, you can see that 'Then the atm should retain the card' step has failed; but BDDfy has run the next step and it shows you that it has passed. Of course, the scenario will be red until all its steps pass.

Both console and HTML reports show that my scenario has failed. It seems like I have a bug in my Atm class. So I fix the bug (i.e., uncomment the only commented line in the Atm class) and run the test again, and this time I get green result:

Figure 7. CardHasBeenDisabled green console report

Figure 6. CardHasBeenDisabled green HTML report

Scenario 2: Account Has Insufficient Funds

Let's implement another scenario. This time, I will not bore you with the red and green phases:

This scenario is a bit more involved. Let's run the test and see the reports:

Figure 7. AccountHasInsufficientFund console report

Figure 8. AccountHasInsufficientFund HTML report

ExecutableAttribute

When reflecting over your test class, BDDfy looks for a custom attribute called ExecutableAttribute on the methods and considers the method decorated with this attribute as a step. You can use attributes either when your method name does not comply with the conventions or when you want to provide a step text that reflection would not be able to create for you.

To make it easier to use, ExecutableAttribute has a few subtypes that you can use. In this scenario, I used GivenAttribute, WhenAttribute and AndThenAttribute attributes because I wanted to show '$' in the step text that would not be possible using method name reflection. Other available attributes are AndGivenAttribute, AndWhenAttribute and ThenAttribute. If you think some other ExecutableAttribute could really help you, then you can very easily implement one. Just to show you how easy it is to implement a new executable attribute below, I have included the implementation for a few of the built-in attributes:

While we are talking about attributes, there is also a non-ExecutableAttribute called IgnoreStepAttribute that you can apply on a method you want BDDfy to ignore as a step. This is useful when you have a method whose name complies with naming conventions BDDfy uses; but is not really a step.

Where Is My Story?

As you may have noticed, we have not still implemented any story. BDDfy is capable of executing standalone scenarios and generating report from them which I think is quite useful for teams that do not do Agile/BDD but are interested in a better testing experience and reporting.

Any class decorated with a StoryAttribute represents a story. Using StoryAttribute, you can also specify the story narrative. To associate the story with its scenarios, you should implement a test method per scenario. I called them the same as the scenario; but you can call them anything you like (when you are using reflecting scanners).

That is it. Just before we run these tests, we should get rid of the Execute test methods in our scenario classes as we no longer need them. We only had them there because we implemented those as standalone scenarios. Now that our scenarios are part of a story, they should not run standalone. Let's run the tests again:

Figure 9. Scenarios moved to story - console report

We now have only one test class which includes two test methods; one per scenario. Also note that the story narrative is now appearing on the top of the console report for each scenario.

Figure 10. Scenarios moved to story - HTML report

In the HTML report, the story narrative appears only once above the story's scenarios.

Note: In the summary section of the HTML report before we implemented the story, we had two namespaces. After adding the story, the namespace count turned into zero and now we instead have one story. BDDfy only counts namespaces for standalone scenarios.

If you compare the above reports with the ones generated when we had Execute methods in the scenarios, you see that these reports group your scenarios by story instead of namespace which makes the reports more readable.

Scenario 1: Account Has Sufficient Funds

Let's do our last (or I should say first) scenario. For this one, I am going to use a different technique. This technique is rather similar to how StoryQ works. StoryQ uses method groups to specify steps - BDDfy uses lambda expressions.

This looks very much like the other scenarios with one difference: the naming conventions are not quite right and you think that BDDfy would fail to match some of these methods - specifically those starting with 'And' instead of 'AndGiven'. If you were to use reflecting scanners, those methods would have been picked up as asserting steps which meant they would run and report in incorrect order! You could very easily customise BDDfy's naming conventions or rename your methods or use ExecutableAttribute to make these methods scannable by reflecting scanners; but I wrote the class like this to show how you can use lambda expression to let BDDfy find your methods/steps:

You may write this method in your scenario class if you want to run it as a standalone scenario. I added it to my AccountHolderWithdrawsCash story to make it part of my story.

Step Scanners

By default, BDDfy uses two scanners namely MethodNameStepScanner and ExecutableAttributeStepScanner - which I collectively refer to as reflecting scanners . The former scans your scenario class for steps using method name conventions and the latter looks for ExecutableAttribute on your methods. There is also a third scanner called FluentStepScanner which we used in the above example.

Note: Reflecting scanners run in a pipeline which means you can mix and match their usage in your scenario; however, when you use FluentStepScanner, BDDfy does not use other scanners which means method names and attributes are ignored for scanning methods. In other words, you are in full control of what steps you want run and in what order.

For reporter modules, it does not make any difference what scanner you use; so the HTML and console reports are going to look the same regardless of the scanners.

An Alternative Implementation

Using FluentStepScanner, you can implement your stories/scenarios in an alternative and rather interesting way. Instead of having one class per scenario and a class for your story, you could write one class that represents all your scenarios as well as your story:

This way, you will not need a separate story class or one class per scenario - everything is mixed into one class called AccountHolderWithdrawsCashFluentScanner. Running tests in this class generates the very same console and HTML reports.

This style of writing stories and scenarios help you be a bit DRYer; but you will violate SRP. It is important to note that you could achieve DRYness without using FluentStepScanner. In order to do that, you would need to use inheritance or composition to compose your scenario class from classes that would hold the common behaviors. For example, if you put your 'Given' and 'When' steps inside a base class and your 'Then' steps inside a subclass, BDDfy will scan all these steps into your scenario. That would not give you as much freedom as the FluentStepScanner though.

Where do the Titles Come From and How to Customize Them?

Story Title

By default, BDDfy uses the name of the story class for the story title as we saw in the first few samples. You can override this behavior by passing the title into the Story attribute as I have done in the above example. I named my class AccountHolderWithdrawsCashFluentScanner to differentiate it from the story class in the other implementation; but I do not want the story title to end with 'fluent scanner'. So I provided the story with a title I will be happy to see in the reports:

[Story(
Title = "Account holder withdraws cash",
AsA = "As an Account Holder",
IWant = "I want to withdraw cash from an ATM",
SoThat = "So that I can get money when the bank is closed")]
publicclass AccountHolderWithdrawsCashFluentScanner

Scenario Title

By default, BDDfy parses the class name into scenario title; for example in the first scenario, BDDfy extracted the scenario text 'Card has been disabled' from the class name 'CardHasBeenDisabled'. In the above example, because all your scenarios are fetched from the same class, one would expect BDDfy to give them all the same title! That is not the case though. In this case, BDDfy detects that you are using FluentStepScanner and uses the test method's name to generate the scenario title. For example, the CardHasBeenDisabled method results into 'Card has been disabled'. That said, if you want to have full control over scenario title, you may pass the title to BDDfy method; e.g.

Step Title

By default, BDDfy uses step method names for the method title and it is also capable of injecting the input arguments in the title. In the above example, Given(s => s.GivenTheCardIsDisabled()) results into 'Given the card is disabled' and When(s => s.WhenTheAccountHolderRequests(20)) results in 'When the account holder requests 20'; but sometimes that is not good enough (e.g., the account holder does not request 20 - he or she requests 20 dollars). In cases like this, if you are using the FluentStepScanner, you can pass in the desired title into the step indicator methods; e.g.

And(s => s.CardIsRetained(false), "And the card should be returned")

The string that you pass into these methods could also have placeholders for input arguments. This way, you can reuse a string template across several scenarios as I did above. I declared a const on the class level:

...which resulted in the step title: 'Given the account balance is $10'. It is worth mentioning that BDDfy uses the template in a string.Format() method to generate the title; so you may use as many placeholders and wherever in the title as you like as long as they match the method inputs.

As mentioned before, when using reflecting scanners you may use ExecutableAttribute or a subtype of it to provide custom step texts. The string provided to these attributes also accepts placeholders that are filled by method input arguments.

Reflecting and fluent scanners offer similar functionalities (but some through different means). Below, you may find a quick comparison:

Functionality

Reflecting Scanners

Fluent Scanner

Story title from story class name

Yes

Yes

Story title from Title in StoryAttribute

Yes

Yes

Scenario title from scenario class name

Yes

No

Scenario title from test method name

No

Yes

Custom scenario title passed in BDDfy method

Yes

Yes

Implementing story and scenarios in one class

No

Yes

Finding step methods using naming convention

Yes

No

Finding step methods using attributes

Yes

No

Finding step methods using lambda expression

No

Yes

Running step methods with input arguments

Yes - using RunStepWithAttribute

Yes - using lambda expression

Step title using step method name

Yes

Yes

Using input arguments in the step title

Yes

Yes

Custom step title

Yes - using attributes

Yes - passing into step indicator methods

Using the same method for several steps

Yes - using RunStepWithArgsAttribute

Yes

Ignoring a method as step

Yes - using IgnoreAttribute

N/A - Do not indicate the method

Dispose method

Yes - Implement a method starting with 'TearDown'

Yes - Use TearDownWith step indicator

Using inherited step methods

Yes

Yes

BDDfy Extensions

You may think that the first and second models are significantly different and that a huge amount of effort has been put to implement both models; but the ONLY difference between these two models is in their step scanners which are not even part of the core. BDDfy is very extensible and the core barely has any logic in it. It instead delegates all its responsibilities to its extensions, one of which is step scanner implementing IScanForSteps. The same applies to scenario scanner (IScanForScenarios), story scanner (IScanner), report generators, test runner and exception handler (all implementing IProcessor). All these interfaces contain only one method which makes it rather straightforward to implement a new extension. Step scanners are a very small part of this framework, and if you think you could benefit from a different scanner you could very simply implement it.

If I were to implement another scanner, I would implement something like:

this.WithSpec("AccountHolderWithdrawsCash.spec").BDDfy();

This scanner would scan the specification file and using reflection would match the story in the test class (specified by this) and its scenarios that match the textual specification and would run them. If it did not find a good match, it would just output its expected class into the console so the developer could copy and paste it into the code and fill in the blanks.

Where Can I Find the ATM Sample?

The sample we worked through in this article is one of the BDDfy samples. There are a few more samples that are implemented in different ways and use some other BDDfy features I did not explain in this article. So I think it is worth looking at all the BDDfy samples.

You can find all the samples in the code

ATM sample along with other BDDfy samples can be downloaded as part of the code from the project homepage on GitHub. All the samples are in the solution in a project called 'Samples'.

There are also over 200 unit tests and BDD behaviors in the code that should make it easier to learn about the framework.

And of course you can install them using NuGet

BDDfy samples are also available for download through NuGet. To get the samples, you may create an empty C# class library project and then type the following in the 'Package Manage Console':

Install-Package TestStack.Samples

Entering the above command will install BDDfy framework and its dependencies as well as the ATM and TicTacToe samples and NUnit framework. When the installation is complete, you can simply run the tests. This sample implements the story using both scanners discussed in this article.

Note: BDDfy samples you can find in the code and on NuGet are all implemented using NUnit; but, as shown in this article, you can do it using any other testing framework.

Have Any Questions or Issues with the Framework?

Please feel free to leave your comment and feedback here. You may alternatively use GitHub issues to ask questions or provide feedback.

Also, you may follow BDDfy on twitter to stay up to date with latest changes and features.

All the exceptions are handled when the step methods are run and depending on the exception type you get different outcomes in the reports; e.g. NotImplementedException and any inconclusive exception result in inconclusive result of the step.

If the exception is thrown in non-asserting steps the flow stops; but in asserting steps the flow continues till end so you know which of your asserting steps have failed.