Acceptance Tests Layer in .NET Core

This article continues the testing layers series, covering in detail a proposal on how to test a modern cloud application. The previous post in the series focused on the first and largest of the testing layers: unit tests. The next layer up, acceptance tests, validates the cloud app in a way that is equally as important as unit tests.

These types of tests don’t get nearly the attention that unit tests do and often end up as afterthoughts. But they should be part of any project that wants to achieve Continuous Delivery or Deployment. Without further ado, here’s the what and how of acceptance tests in .NET Core 2.

What Are Acceptance Tests?

We’ll start by defining what an Acceptance Test is because there is inevitably more than one term and definition for everything in software development. You’ve probably seen these types of tests called integration tests. Although they do test the application in an integrated way, the requirements they satisfy go well beyond a normal integration test. They are also commonly known as Contract Tests.

The purpose of Acceptance Tests are to:

Verify the requirements of the system are satisfied, typically based on acceptance criteria in an Agile environment.

Ensure that the application fulfills those requirements once deployed, whether it be development, staging or production.

Ensure that no breaking changes are introduced to the customer-facing contract.

Layers of testing a cloud app

There should be as many acceptance tests as are needed to fully validate that the deployed cloud application is running as expected.

Most cloud applications today get deployed in a semi-automated way (typically triggered via a tool like Octopus Deploy) followed by a team member performing sanity checks to ensure that everything is working correctly. This approach is slowly changing as more organizations adopt Continuous Delivery and Deployment. Deployments in both these models happen on a highly frequent basis (sometimes even on every commit), so you need a quick, thorough, and efficient way to validate the deployments from a business and operational perspective. Acceptance tests provide you with all this.

In other words, automated acceptance tests enable us to build faster and safer with the confidence that our deployed application is running as expected.

How To Write Acceptance Tests

You can find the full source code for the examples below in the Supermarket application source code, which can be found here.

The first step to writing good acceptance tests is to have them live alongside the production code they are meant to test. To that end, the Supermarket sample splits the production code from the test code via the src and test folders. The test folder is then further subdivided into unit and deployment folders to make the deliniation as clear as possible.

Acceptance tests should test the behaviours that the cloud application exposes. That means testing happy paths and the most common edge cases. The tests should be exhaustive enough so that a succesful run makes you confident that the code is running as expected.

Let’s look at what an acceptance tests look like. There are a total of 5 tests in the Acceptance folder of the Supermarket.Deployment.Tests project. These tests validate that:

A request with missing parameters is not accepted

A valid postal code results in a successful response

The edge cases of tax calculation are handled correctly

Each test has the same basic structure that is similar to that of a unit test. The test is set up by retrieving an access token on the environment on which it is running. The request is then sent to server and the response is checked for the correct result. The needed details, such as URL and credentials to use, should be provided to the tests via environment variables.

And that’s all there is to it. Having these tests run succesfully on every deployment tells us that the application is in an expected state and can act as a “first alert” when something goes wrong with a deployment.

Using Visual Studio or Code To Write Acceptance Tests

A cloud application should be in a working state by the time you’re writing acceptance tests. It should already be thoroughly unit tested [LINK] and there’s been some manual testing of the application.

You can run acceptance tests locally, albeit with a small caveat: you can’t debug both the tests and the production code because Visual Studio doesn’t support launching the WebApi and running the tests simultaneously. Follow these steps to run the tests locally:

Build the WebAPI project in Visual Studio

Open a command prompt and dotnet run the WebAPI project

Execute the acceptance tests from Visual Studio

Another approach could be to have two separate SLN files at the root of the project. The first for the production code, and the other for the acceptance tests project. This would allow you to have two separate instances of Studio running that let you debug both the acceptance tests and WebApi project simultaneously.

Use test playlists in Visual Studio to configure which tests to run together. You can set up a “Unit Tests” and “Acceptance Tests” playlist which will allow you to easily run or the other set of tests. For those who are Visual Studio Code inclined, I’d suggest creating some tasks that run dotnet test with filter attributes to run the test sets.

The real power of acceptance tests is running them against deployed environments. The image below shows how a CI/CD server executes the tests as the application is deployed across environments, ensuring that everything is running expected from feature development to production.

The Big Picture

Acceptance tests need to run on every environment to which you deploy the application. The best way to ensure that happens is to integrate them as part of your CI/CD pipeline. When a developer commits code, it gets compiled, unit tested, deployed, and then acceptance tested. Rince and repeat for every environment that the code gets deployed to.

Above all else, the tests need to be reliable. They need to run consistently, over and over with little to no false positives. This may seem obvious but is really hard to achieve. Network issues, downstream service failures and configuration problems can result in all sorts of false-positives. Spend the time needed to fix any brittle tests or you’ll quickly lose the value that they provide.

Acceptance tests should not change once they are stable. It’s possible to add on to them but a change to an existing acceptance test implies introducing a breaking change. Without even trying, they protect you from breaking downstream clients.

Writing acceptance tests in this way ensures that your application is deployed succesfully, fulfills its requirements, and that you don’t accidently break a contract. Most importantly, it will enable you to build higher quality cloud applications.