In this post you will find my PowerPoint and source code I used for my presentation at Charlotte Alt.Net May meeting. I had a good time presenting this to the group even though it was very broad and shallow. I covered the basis on why you want to leverage tools and practices in a Lean Agile environment. I got into topics like Source Control, Unit Testing, Mocking, Continuous Integration and Automated UI Testing. Each of these topics could have been an entire 1 hour presentation on its own.

Here are the links to the tools that I talked about in the presentation:

I ran into a strange issue today when I was trying to figure out why my TFS Build was reporting that the build was partially successful even though every test was passing. The normal build report really did not give any good reason why it was partially successful other than the fact that it was something related to the unit tests (I am using MS Test in this case). So I cracked open the build log and peeled through the entries. I noticed that when the code coverage was attempting to instrument the assemblies it reported that several of the assemblies could not be located. Then I remembered I did some refactoring and I renamed and consolidated some assemblies.

Well that should be an easy fix all I had to do was remove these assemblies from the test run config in the Code Coverage section. So I opened up the LocalTestRun.testrunconfig file in Visual Studio 2008 and selected the Code Coverage section to make my changes. As soon as I did this the config editor closed down (crashed). Wow that was weird I never saw that before. Hmmm I wonder what it could be. Well here is what I did to try and locate the issue.

Perhaps the Test Run Config file needs to be checked out of source control for write access. Nope that wasn't it.

Well if I cant edit it in VS 2008 then I might as well try notepad. I removed the offending assemblies using notepad in the LocalTestRun.testrunconfig. However once I opened up the Test Run Config editor and selected the Code Coverage the editor still crashed.

Perhaps I malformed the Test Run Config xml file. So I opened it up again in notepad and the XML looked fine. Besides if this XML was malformed I don't think the Test Run Config editor would not open at all.

So to be sure that I got the Test Run Config file right I removed my Database project from my solution made my edits for Code Coverage in the Test Run Config editor, then added the database project back into the solution.

In my day job at JDA Software I have been looking at code coverage options for determining the effectiveness of our testing. My team uses four types of tests to test the software we write.

Unit Tests - Tests focused on a single component of the application. These tests are MSTests that exercise a specific software component and typically mock out any dependant components.

Integration Tests - Tests focused on multiple components of the application. These tests are MSTests that exercise a component and it's dependencies to ensure the components work together.

Manual Tests - Tests that are executed by are testers manually from a GUI interface.

Automated functional test - Tests that are scripted in a fashion that can be repeated build after build to ensure the build is still functional. Sometimes referred to regression testing.

The goal of implementing a code coverage process was to determine the effectiveness of the types of tests listed above. Visual Studio 2005 Team Edition (for testers and developers) provides some nice code coverage features we wanted to tap into. Code coverage of unit tests and some integration tests worked fine from the visual studio IDE. But for the integration, manual and automation tests that used web services we began to run into problems. The web services that where hosted under IIS where not getting covered. After some research I determined that code coverage under ISS was not going to work. So I started looking into alternatives. One thing I noticed is that web service projects that where not hosted under IIS had no problem getting covered. These types of projects used the development web server that comes with ASP.NET 2.0. So I decided to look into using the same development web server in place of IIS for code coverage purposes.

The generic steps for establishing code coverage for web services is as follows:

Turn on instrumentation for the binaries that you want to cover

Start the coverage monitor

Start the development web server for a specified unused port pointing to the folder that is supposed to represent the web service.

Execute your tests

Stop the development web server

Stop the coverage monitor

Review the results in Visual Studio 2005

For the following commands use the Visual Studio Command prompt.

Turning on instrumentation of assemblies from the command line is done by the following command:

vsinstr -coverage myassembly.dll

To start the coverage monitor you use the follwoing command:

start vsperfmon -coverage -output:mytestrun.coverage

To start the development web server use the following command:

start WebDev.WebServer /port:8080 /path:c:\mypath

To stop the development web server simply right click on the task bar icon for the web server and select stop

To shutdown the coverage monitor use the following command:

vsperfcmd -shutdown

We were now able to do code coverage of all types of tests that we planned on implementing. Some of our test runs are going to executed by the build and some will be executed by the testers. Since VS.Net 2005 supports the ability to merge code coverage results this should not be a problem to combine all runs into a single report that now shows how effective our tests really are.