Running tests with dotnet xunit using Cake

In this post I show how you can run tests using the xUnit .NET CLI Tooldotnet xunit when building projects using Cake. Cake includes first class support for running test using dotnet test via the DotNetCoreTest alias, but if you want access to the additional configuration provided by the dotnet-xunit tool, you'll currently need to run the tool using DotNetCoreTool instead.

dotnet test vs dotnet xunit

Typically, .NET Core unit tests are run using the dotnet test command. This runs unit tests for a project regardless of which unit test framework was used - MSTest, NUnit, or xUnit. As long as the test framework has an appropriate adapter, the dotnet test command will hook into it, and provide a standard set of features.

However, the suggested approach to run .NET Core tests with xUnit is to use the dotnet-xunit framework tool. This provides more advanced access to xUnit settings, so you can set a whole variety of properties like how test names are listed, whether diagnostics are enabled, or parallelisation options for the test runs.

You can install the dotnet-xunit tool into a project by adding a DotnNetCliToolReference element to your .csproj file. If you add the xunit.runner.visualstudio and Microsoft.NET.Test.Sdk packages too, then you'll still be able to run your tests using dotnet test and Visual Studio:

With these packages installed and restored, you'll be able to run your tests using either dotnet test or dotnet xunit, but if you use the latter option, you'll have a whole host of additional arguments available to you. To see all your options, run dotnet xunit --help. The following is just a small selection of the help screen:

This will run a restore in the solution directory, build the solution, and then run dotnet test for every project in the test sub-directory. Each of the steps uses a C# alias which calls the dotnet SDK commands:

DotNetCoreRestore - restores NuGet packages using dotnet restore

DotNetCoreBuild - builds the solution using dotnet build, using the settings provided in the DotNetCoreBuildSettings object

DotNetCoreTest - runs the tests in the project using dotnet test and the settings provided in the DotNetCoreTestSettings object.

Customizing the arguments passed to the dotnet tool

In the previous example, we ran tests with dotnet test and were able to set a number of additional options using the strongly typed DotNetCoreTestSettings object. If you want to pass additional options down to the dotnet test call, you can add customise the arguments using the ArgumentCustomization property.

For example, dotnet build implicitly calls dotnet restore, even though we are specifically restoring the solution. You can forgo this second call by passing --no-restore to the dotnet build call using the ArgumentCustomization property:

With this approach, you can customise the arguments that are passed to the dotnet test command. However you can't customise the command itself to call dotnet xunit. For that, you need a different Cake alias - DotNetCoreTool

Running dotnet xunit tests with Cake

Using the strongly typed *Settings objects makes invoking many of the dotnet tools easy with Cake, but it doesn't include first-class support for all of them. The dotnet-xunit tool is one such tool.

If you need to run a dotnet tool that's not directly supported by a Cake alias, you can use the general purpose DotNetCoreTool alias. You can use this to execute any dotnet tool, by providing the tool name, and the command arguments.

For example, imagine you want to run dotnet xunit with diagnostics enabled, and stop on the first failure. If you were running the tool directly from the command line you'd use:

dotnet xunit -diagnostics -stoponfail

In Cake, we can use the DotnetCoreTool, and pass in the command line arguments manually. If we update the previous Cake "Test" target to use DotNetCoreTool we have:

The DotNetCoreTool isn't as convenient as being able to set strongly typed properties on a dedicated settings object. But it does at give a great deal of flexibility, and effectively lets you drop down to the command line inside your Cake build script.

If you build your scripts using Cake, and need/want to use some of the extra features afforded by dotnet xunit then DotNetCoreTool is currently the best approach, but it shouldn't be hard to create a wrapper alias for dotnet xunit that makes these arguments strongly typed. Assuming noone else has already done it and made this post obsolete, then I'll look at sending a PR as soon as I find the time!