Customizing Code Coverage in Visual Studio 11

The Visual Studio 11 Code Coverage tool by default analyzes all solution binaries (exe / dll) that get loaded during unit tests. We suggest you retain this default; this works most of the times.

However, there may be cases where you want to customize this behavior. One common case is to include / exclude specific binaries from code coverage. When this is done, there is a possibility you might lose track of new assemblies introduced into the project, and not measure code coverage for them. Still, if it needs to be done, it can be achieved by customizing code coverage settings. The other common case is to specify symbol search path for binaries that don’t have symbols installed locally.

This blog talks about customizing code coverage for scenarios like these.

1. The .runsettings file

Advanced VS 11 code coverage settings are specified in a .runsettings file. This is the configuration file used by VS 11 unit testing tools.

Some of you, who have used VS 2010, may be familiar with .testsettings file. You may be wondering why the same file was not used for VS 11. The reason for this was that the .testsettings file applies only to the MSTest based unit tests of VS 2010. The VS 11 unit testing tools not only apply for MSTest, but also 3rd party unit testing adapters such as NUnit and xUnit.net. Using a .testsettings file will not work with these. Hence a different settings file, the .runsettings, was used.

To customize code coverage, you will need to add .runsettings file to your solution. There isn’t a built-in template for this right now. You would need to add it as an xml file and edit its contents:

2.Copy the sample runsettings content given later in this blog into your file.

3.Edit it to customize code coverage, as explained in next section.

4.Set it as the default setting file to be used via Test -> Test Settings -> Select Test Settings File

After this, the specified settings should be used whenever you analyze code coverage.

2. Common customization scenarios

2.1. Customizing the set of binaries analyzed for code coverage

Here, we want include / exclude specific binaries from code coverage. Using the runsettings file, there are two approaches to do this:

Customize by exclusion

Here we include everything, and exclude what we don’t need – an opt-out approach. This may be useful if the code of interest changes often and new binaries get introduced frequently: you would like these new binaries to be automatically included for code coverage when they are loaded. There are some known binaries that you don’t want coverage, like some test automation code. These are excluded.

Here, leave the includes as empty to include all assemblies. Specify the assemblies to be excluded thru by their name / path.

<ModulePaths>

<Include>

<!-- Do not specify any includes. This will attempt to include all binaries -->

</Include>

<Exclude>

<!-- Exclude modules that aren't to be processed, by their name / path -->

<ModulePath>.*MyCompany.MyTests1.dll</ModulePath>

<ModulePath>.*CPPUnitTestFramework.*</ModulePath>

</Exclude>

</ModulePaths>

Customize by inclusion

Here we include exactly what we need, and exclude everything else – an opt-in approach. This may be useful if the code of interest is present in well-known binaries; you just want these binaries to be covered.

Here explicitly include the binaries of interest. Leave the excludes as empty to exclude everything.

<ModulePaths>

<Include>

<!-- Include modules of interest, by their name / path -->

<ModulePath>.*MyCompany.MyComponent.dll</ModulePath>

</Include>

<Exclude>

<!—- Do not specify any excludes. Anything not included will get excluded -->

</Exclude>

</ModulePaths>

Note that while both inclusions and exclusions can be used together, it can make things very confusing. So it is preferable to use one mechanism to customize.

2.2. Specifying symbol search paths

Code coverage requires symbols (PDB) for binaries to be instrumented. For binaries built by your solution, symbols are generally present alongside the binary and code coverage automatically works. In some cases, you may be referencing binaries external to your solution, but still need code coverage for these external binaries. In such cases, you can specify the symbol search path to look for symbols as part of the runsettings file.

<SymbolSearchPaths>

<Path>>\\mybuildshare\builds\ProjectX</Path>

</SymbolSearchPaths>

A set of local and remote paths can be specified here. Note however that symbol resolution can take time, especially when using a remote build share with a lot of binaries. Best would be to avoid this by copying symbols alongside the binary, and not specify any search paths.

3. Sample .runsettings file

Below is a sample runsettings file. When creating a new runsettings, start with this, and customize it as needed.

4.1. In Visual Studio IDE

4.2. In command line

The tool vstest.console.exe is the way to run unit tests from command line. It is the successor of MSTest – and can run not only the MSTest based tests, but also tests of third party unit adapters (like NUnit, xUnit.net, etc).

4.3. In Build

Here, there is a TFS Build Definition, as part of which unit tests are run and overall code coverage is measured. If there is a .runsettings file customizing code coverage, the same runsettings file can be used in the Build as well.

Below are the steps to do it.

1.Edit the build definition. Go to Process tab, and edit test run details.

What additional plug-in should i install besides VS2012 ultimate to use these two dlls: Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=11.0.0.0?

Hello Sudhakar – My unit tests fail with Code Coverage but passes otherwise.. I linked it up to a runsettings file as advised in your blog.. It seems to run for th first few lines of code in the test but fails again and with a different error.. What's perplexing is – The test works just fine otherwise.. Wonder what could be going on here.. In case if it helps, here is the link that will have all the details of the error that I am seeing..