We can write automated tests to cover various aspects of the code we write. We can write unit/integration tests that test that the code is producing the expected outcomes. We can use ConventionTests to ensure internal code quality, for example that classes following a specified naming convention and exists in the correct namespace. We may even add the ability to create a business readable tests using tools such as SpecFlow or BDDfy.

Another aspect that we might want to ensure doesn’t change unexpectedly is the public API that our code exposes to callers.

Using PublicApiGenerator to Generate a Report of our Public API

The first step of ensuring our public API hasn’t changed is to be able to capture the public API in a readable way. The PublicApiGenerator NuGet package (from Jake Ginnivan) gives us this ability.

These tests help us ensure the code is doing the right thing but do not offer any protection against the public API changing. We can now add a new test that uses PublicApiGenerator to generate a string “report” detailing the public members of our API. The following test code shows this in use:

[Fact]
public void ShouldHaveCorrectPublicApi()
{
var sut = new Calculator();
// Get the assembly that we want to generate the public API report for
Assembly calculatorAssembly = sut.GetType().Assembly;
// Use PublicApiGenerator to generate the API report
string apiString = PublicApiGenerator.PublicApiGenerator.GetPublicApi(calculatorAssembly);
// TODO: assert API has not changed
}

If we debug this test and look at the content of the apiString variable we’d see the following text:

The first time we run this it will fail with a message such as “Failed Approval: Approval File "c:\…\Demo.Tests\CalculatorApiTests.ShouldHaveCorrectPublicApi.approved.txt" Not Found”. It will also generate a file called CalculatorApiTests.ShouldHaveCorrectPublicApi.received.txt. We can rename this to CalculatorApiTests.ShouldHaveCorrectPublicApi.approved.txt, run the test again and it will pass.

If we now modify the public API by changing a method signature (e.g. to public void Clear(int someParam)) and run the test again it will fail with a message such as “Received file c:\...\Demo.Tests\CalculatorApiTests.ShouldHaveCorrectPublicApi.received.txt does not match approved file c:\...\Demo.Tests\CalculatorApiTests.ShouldHaveCorrectPublicApi.approved.txt”.

Modifying the test and adding an Approval Tests reporter attribute ([UseReporter(typeof(DiffReporter))]) and running the test will now gives us a visual diff identifying the changes to the public API as shown in the following screenshot.

Shouldly is an open source library that aims to improve the assert phase of tests; it does this in two ways. The first is offering a more “fluent like” syntax that for the most part leverages extension methods and obviates the need to keep remembering which parameter is the expected or actual as with regular Assert.Xxxx(1,2) methods. The second benefit manifests itself when tests fail; Shouldly outputs more readable, easily digestible test failure messages.

Failure Message Examples

The following are three failure messages from tests that don’t use Shouldly and instead use the assert methods bundled with the testing framework (NUnit, xUnit.net, etc):

In each of the preceding failure messages, there is not much helpful context in the failure message.

Compare the above to the following equivalent Shouldly failure messages:

“schedule.TotalHours should be 9 but was 5”

“schedule.Title should not be null or empty”

“schedule.Days should contain "Monday" but does not”

Notice the additional context in these failure messages. In each case here, Shouldly is telling us the name of the variable in the test code (“schedule”) and the name of the property/field being asserted (e.g. “Total Hours”).

Test Code Readability

For the preceding failure messages, the following test assert code is used (notice the use of the Shouldly extension methods):

schedule.TotalHours.ShouldBe(9);

schedule.Title.ShouldNotBeNullOrEmpty();

schedule.Days.ShouldContain("Monday");

In these examples there is no mistaking an actual value parameter for an expected value parameter and the test code reads more “fluently” as well.

In a test, we are often testing (asserting) individual items such as an (int) age is correct or a string matches an expected value.

If we are practicing test-first development we’ll write our asserts first.

Approval tests allow us to go beyond simple asserts.

What if the thing we’re checking is not a simple value, for example that a pie chart image matches the input data? Or what if we want to use our human judgement to decide when something looks correct, something that is hard to codify in one or more basic asserts?

ApprovalTests for .NET can be install via NuGet. Once installed, it gives us a whole new world when it comes to checking the output of code.

For example, say we are developing a class to represent a stickman. We want to be able to tell an instance to raise left arm or raise right leg for example.

We want our test code to be as high quality as possible, this means smaller amounts of code duplication, reasonably easy to find where things are in Visual Studio, etc.

One possible organization structure is to think in terms of the individual features in the application. The approach you take will probably depend on the complexity and size of the test suite, system under test, etc. Because BDDfy is "just code" we can use all the normal techniques of composition and inheritance that we'd use in our production code to get to the right level of code reuse and organization for the application we're testing.

Organisation by Feature

Organising by feature enables a reasonable amount of code reuse between test scenarios and it also helps to think user- or business-first rather than code/implementation first.

So for example, if we’re using BDDfy to test a banking application we might have the following features:

Login

Logout

Move Money

Pay Bills

View Transactions

etc

Each of these features contains a number of scenarios, for example Login would probably contain scenarios for successful login, bad password, locked out account, 2 factor login, etc.

In Visual Studio we could create folders in the test project to represent and organize these features as the following screenshot illustrates.

So a (cut down) Login BDDfy story class could look like the following code:

Now for arguments sake, say we have a Navigation story class that represents how the user should be able to move around the applications features.

We could reuse the individual given/when/then methods in CustomerLogin but we don’t want our navigations scenarios to be bloated, we want them to represent the essence of the scenario with the right level of detail.

So the first thing we could do is to create a “step aggregation” method in CustomerLogin as follows:

So here were are making use of this aggregate method from the CustomerLogin feature class.

If we don’t need to represent the fact that the customer is logged-in in the report and all the tests in the class assume a logged-in user, we could use some (e.g. NUnit) test setup code that logs the user in but doesn’t get reported.

The HTML output of this looks as follows:

While in these examples we have a single story class for the feature (e.g. CustomerLogin) once we start adding scenarios (and steps) this single story class might become too bloated. If this is deemed a problem then we can break it out into sub features/stories or if it’s applicable we could use inheritance to hold common given/when/then steps. The individual story classes relating to the customer login feature would all inherit this base class. We probably however, do not want multiple levels of nested inheritance in our stories as this may make maintenance and discoverability harder.

BDDfy enables the creation of tests that, in addition to verifying that the system works correctly, also results in business-readable, living documentation.

Business-readable means that the tests are described in natural language (e.g. English) that the business can read, understand, and ensure that the correct features and functionality is being built.

Living documentation means that the report results directly from the passing or failing of the tests. It’s not a word document somewhere on a shared drive or SharePoint site that may or may not actually be in sync with what the system actually does.

The Report

Below is an example of what a typical BDDfy HTML report looks like (there is a “metro” inspired report coming in V4). There’s also the ability to output the test report in markdown.

Notice that it’s not code-centric output, but rather business-centric.

Here the two test scenarios are grouped into a story, but they don’t have to be.

The underlying test framework that is used to execute the test doesn’t matter – you could use NUnit, xUnit.net, MSTest, etc.

The Test Code

The tests that produce this report could be configured in BDDfy in a couple of ways. There is a reflective style that uses methods that following a specific naming convention. There is also the fluent style.

The test code to produce this report looks like the below (we are using NUnit in this example).

Here there are a couple of NUnit tests defined – the methods with the [Test] attribute applied to them.

Within these test methods the BDDfy fluent style is being used to define the different steps in the test scenarios.

The “given” phase sets up the initial context or state of the thing being tested. The “when” phase acts upon the system to produce some change. The “then” phase is typically where we have assert code, it’s in this phase that the resulting state of the system is checked.

Notice how the individual method names appear in the test report in a nice business-readable way.

With over 15 years experience, Jason Roberts is a former 5-time Microsoft .NET MVP, freelance developer, writer, and Pluralsight course author. He has written multiple books and is an open source contributor. In addition to enterprise software development, he has also designed and developed both Windows Phone and Windows Store apps.