Executing MSBuild Targets in Parallel – Part 1

Coming in the next release of the MSBuild Extension Pack is a new task which allows you to easily run Targets in parallel. This will be limited to the .NET 4.0 releases, so next up that’s 4.0.5.0. (you can try the task now by getting changeset 74609).

Typically in MSBuild we execute a sequence of targets and MSBuild will do just that; execute sequentially. If you have simple scripts that run quickly then this may not be a problem. If however you have complex scripts that take considerable time to execute, the new parallel functionality may be useful to reduce execution times.

In the following samples I’ll work through how the tasks can be used and various things to be aware of.

Below I have a sample which executes three targets. In each target there is a Sleep task so we can provide some metrics as we extend the sample.

Executing the above sample we get the expected 7 second or so build time.

Assuming there are no dependencies between these targets, the fastest time we we can execute this is around 4 seconds, the time of our slowest target, plus the initialization cost.

Rather than having to run these targets sequentially, what I really want to do is run non-dependent targets in parallel and reduce the build time. In the next sample I’ve removed the DependsOnTargets and added the parallel task to execute BuildTargetsInParallel

Executing the above sample we see our build time drop to 4.60 seconds.

This is a great reduction in build time, but not as close to the four second mark as we may have expected. The reason that we have a slightly higher build time is that the parallel task needs to start MSBuild and initialize a new instance of your build file before it executes each target you have requested to execute in parallel. This is an important point to remember if you have a large build file and a high initialization cost. The aim is not to simply run every target in parallel, but rather to find an efficient way to run your build in parallel.

In this first parallel sample I ran all targets in parallel, initiating the build file four times. It may be more efficient to run sets of targets in parallel. The parallel task allows us to specify BuildTargetSetsInParallel.

In this case we have initiated 3 rather than 4 instances of our build file and chosen to execute two parallel sets of targets. Note that I have the longest running target running by itself and have the remaining targets running sequentially on another instance. Given that the execution time of the remaining targets was less than Target2, there is no need to run more than two sets at once.

In this simple example we’ve managed to reduce our build time by over 42%.

In Part 2 I’ll cover how we maintain state when running in parallel as well as some limitations and tips.

IS there a way of running the same target in parallel more than once. What I have is an ItemGroup, and the items in the group are passed to an MSBuild task. I’d like the execution of the MSBuild task to be run in parallel with each item in the ItemGroup…
e.g.

Inside a.proj;b.proj;c.proj I call sln files among other actions.
How do I deal with with the dependency issues in parallel target execution?

Using the suggested target sets would solve the issue but it would not be the most efficient way of running a complex build, since you would have to hard code dependencies between sets. That would mean set 2 would have to wait until set 1 completes, even if there’s only one project in set 1 we are waiting for and some projects in set 2 could already run.

So part of using this task is understanding that you are going to hit unexpected sequence and race conditions. There isn’t any dependency resolution built into the task. You may be able to benefit from a combination of this task and using BuildInParallel http://msdn.microsoft.com/en-us/library/bb651793.aspx

I am using fxcop activity given in MSBuildExtensionPack, but I have a issue setting up fxcop path for the below task. By default fxcop is looking at C:\program files\Microsoft fxcop 1.36\.
On my server it is located at C:\program files (x86)\Microsoft fxcop 1.36\ and I am getting error FxCopCmd.exe was not found in the default location. Use FxCopPath to specify it. Searched at: C:\Program Files\Microsoft FxCop 1.36 and \Microsoft FxCop 10.0