Visual Studio Team System Unit Testing

Of all the features in Visual Studio Team System (VSTS), none is more controversial than the unit testing features. The community campaign to convince Microsoft to move unit testing from the premium-level VSTS product into a more readily accessible version has been unsuccessful so far, and many are legitimately concerned that the requirement for VSTS will significantly decrease the uptake of the unit testing features. For all the controversy, the actual functionality of the unit testing has not received a lot of attention. This article examines the VSTS Unit Test offerings from a Visual C++ perspective.

C++ has had the CppUnit unit testing framework available for many years, but it seems like unit testing has not made the inroads in the C++ community that it has in .NET and particularly Java. The reasons for the slow uptake have been variednative C++ lacks the metadata that makes hooking up unit tests easy and intuitive, CppUnit has not been as polished as its managed contemporaries, and the agile mindset that has been the driving force behind unit testing has not been as globally applicable in the C++ world where large, established projects are more prevalent. VSTS unit testing cannot solve the two intrinsic problems with C++ unit testing (cultural issues related to agile programming and lack of metadata in native C++), but it does deliver a nicely polished unit testing framework that is ready to roll with C++.

Thankfully, the few caveats that do exist with VSTS Unit Testing and C++ are fairly obvious:

To conduct unit tests for native code, a standard native entry point into that code needs to be exposed.

The unit testing execution engine in VSTS is designed around managed code and managed code attributes. Hence, native code cannot be used to write unit tests.

Code generation of unit tests is supported only if the production code (i.e., the code that is to be tested) is compiled with the /clr:safe option.

The fact that the set of limitations is so small is quite impressivethe VSTS team has pushed the functionality right up to the limits of what the language and the .NET and Windows platforms support. The full breakdown of the entire unit testing and C++ combinations and their behaviors is available on the MSDN site.

Authoring and Executing Unit Tests

For those familiar with unit testing frameworks like NUnit, the VSTS unit testing functionality will seem quite familiar. A unit test is a managed method that is marked with the TestMethod attribute. Unit tests must be included in a type with the TestClass attribute. Unit tests execute some type of logic and, at the end of executing this logic, compare the result achieved with an expected value using conditional logic in the Microsoft.VisualStudio.TestTools.UnitTesting,Assert class. If the comparison executed through the Assert class is correct, the unit test passes. Otherwise, it fails.

There is obviously a lot of boilerplate code required before the real functionality of the unit tests can be added. That is where the code generation abilities of VSTS come in. The simplest way to add a unit test for a method is to bring up the context menu in the text editor by right-clicking near the method, and selecting Create Unit Tests. (see Figure 1).

Figure 1. Adding a Unit Test to a C++ Project

Note that this feature will be available only if the code that is being unit tested (referred to as production code in the MSDN documentation) is compiled with /clr:safe. For the simple method shown in Figure 1, the unit test generated will look as follows:

Notice that the generated test calls Assert::Inconclusive to let the unit-testing framework know that the test shouldn't be counted as a passed test. The default result for a test in the absence of a call to an Assert class method is a pass, which can be useful if a test is written purely to check that the tested code runs without throwing an exception (if an unhandled exception is thrown, the test will be defined as a failure). Modifying the unit test so that it actually tests the Add method correctly would result in the following code:

Unit tests can be run directly from within Visual Studio by selecting either Start Selected Test Project With Debugger or Start Selected Test Project Without Debugger.

Managing Tests

Adding a test project to a solution will also result in the creation of a configuration file that defines the various configuration properties for running tests. Thankfully, VSTS also ships with a Visual Studio Addin for managing the configuration file. Double-clicking on the file in Visual Studio will bring up the dialog shown in Figure 2.

Figure 2. Configuring Unit Tests

This dialog allows tests to be configured to run on remote machines; it configures code coverage settings; and it allows scripts for setting up and tearing down before and after a test run to be executed.

VSTS has a great deal of flexibility in selecting which tests to run. The simplest way to configure which tests to run is through the Test Manager. Selecting Tests | Windows | Test Manager will bring up the Test Manager, as shown in Figure 3.

Figure 3. Test Manager

It is possible to create a filter that runs tests only with a field that matches a particular keyword. You also can create a custom Test List and drag-and-drop tests into the List. Once a filter or Test List has narrowed the tests that need to be run, the selected tests can be run with or without the debugger. The test management settings built using the Test Manager are stored in a Visual Studio Metadata file, which is an XML file with a vsmdi extension. This file is stored with the solution, and settings made in the Test Manager are not lost when it is closed.

The test configuration file dialog shown in Figure 2 can be used to instrument binary files for code coverage analysis. Once the instrumentation has been added and the tests re-run, the code coverage results can be brought up with a context menu in the Test Results window, as shown in Figure 4.

Figure 4. Accessing Code Coverage Results

Unit Testing C++ Made Easy

For those that have found getting into unit testing with C++ too hard, the release of VSTS represents an excellent opportunity to have another shot at integrating unit testing into your development practices.

About the Author

Nick Wienholt is an independent Windows and .NET consultant based in Sydney, Australia. He is the author of Maximizing .NET Performance from Apress, and specializes in system-level software architecture and development with a particular focus on performance, security, interoperability, and debugging. Nick can be reached at NickW@dotnetperformance.com.

Comments

Unit test for C++ win32 application

Posted by sira
on 12/07/2012 12:37am

First time I learn about unmanaged C++ unit test from this tutorial. Defnitly this will help to beginers
http://codeketchup.blogspot.sg/2012/12/unit-test-for-unmanaged-c-in-visual.html

Using printf in VC++ unit tests

Posted by Roger Pearse
on 10/11/2012 12:48am

One thing that will make you hate Microsoft; things like printf produce no output. Google searches will bring up stuff like OutputDebugString() which will not work either.
What you have to use is Console::WriteLine(L"myoutput string");

.NET performance

Posted by Annie Calvert
on 08/08/2012 02:00am

It's not a profiler but an Application Performance Management Solution that gives you the ability to trace every single transaction in your application - not only on one CLR but across all CLRs you have in your system.
http://www.dapfor.com/en/net-suite/net-grid/features/performance

Stepping in to unit tests

Posted by metaforge
on 12/14/2009 07:52pm

I've got a native C++ DLL project and I've written an extern mC++ NUnit project that can call into the native code, get a result, and test its correctness. However, when I run the test in debug mode, it doesn't seem to let me step into the native code - breakpoints in the native code are greyed out saying no symbols loaded. I do have the native code project in the same solution, and if I write a native command line exe project, I can call the DLL and step into it. Any ideas?

Top White Papers and Webcasts

Data integrity and ultra-high performance dictate the success and growth of many companies.
One of these companies is BridgePay Network Solutions, a recently launched and rapidly growing financial services organization that allows merchants around the world to process millions of daily credit card transactions. Due to the nature of their business, their IT team needed to strike the perfect balance between meeting regulatory-mandated data security measures with the lowest possible levels of latency and …

Today, users, applications, and data exist in more places than ever before, creating an unprecedented challenge for IT. How can IT achieve the flexibility and agility it needs to offer multiple types of applications in multiple locations? To better serve business demands for information everywhere, enterprises must develop new strategies for optimizing multiple kinds of networks. Read this white paper to learn how hybrid networks provide an unprecedented level of network dynamism, enterprise agility, and the …