Incorrect solution build ordering when using MSBuild.exe

UPDATE: This issue has been fixed in Visual Studio 11/NETFX45. As always, feedback is welcome! Please leave your comments in this blog post and report any bugs on Microsoft Connect.

We’ve had a few reports of cases where Visual Studio, and previous versions of MSBuild, will build the projects in the solution in the correct order, but the 4.0 version of MSBuild.exe gets the order wrong.

Here’s a full description of what’s going on, why it began in 4.0, and the fix we recommend to your projects to solve the problem. If you’re not interested in the “why”, skip ahead to the workaround.

Archetypical case exhibiting the problem

This diagram shows a solution file containing three projects, A, B, and C. Let’s say they are C# projects.

A has a regular project reference to B, so it will invoke B, then when B comes back done, it will build itself. At the same time in the solution file, there is a manually specified dependency: B depends on C.

I’ve shown regular project references with solid lines, and the information in the solution with dotted lines.

This manually specified dependency was set up in the solution by right clicking on the solution and choosing Project Dependencies…, then checking a box. Below I’ve shown the context menu and what you see in the dialog when you have this setup.

The build ordering you expect here is C, then B, then A, and Visual Studio shows that correctly in the Build Order tab, as you see below.

Of course a real case would have more projects in it, but it would boil down to this case.

Why does this happen (skip ahead if you just want the “fix”)

Essentially the problem is that MSBuild doesn’t know anything about the project files until it starts to build them.

Solution files, as you know, are not in MSBuild format (yet). On the command line, MSBuild.exe is on its own, so it parses them and generates one or more in-memory MSBuild format files that are essentially a translation. If you want to see these ugly files, set an environment variable MSBUILDEMITSOLUTION=1 then build the solution. You’ll see a .sln.metaproj file emitted next to your solution file, and possibly one or more files with an extension like .csproj.metaproj next to some of your projects.

The .metaproj generated for B.csproj is how MSBuild makes sure that the solution dependency is respected — at least, it’s created so that the solution itself does not invoke B until C is built. It does this by invoking the B metaproj instead of B directly, and in the B metaproj, it builds C before B. This is exactly equivalent to someone going into B and adding a project reference to C, instead of a solution dependency, but it means that we don’t have to edit the B project directly.

Here’s what it looks like after this translation:

Why doesn’t that work in this case? In short, the problem is the project reference from A to B. Here’s what happens: the solution invokes A.csproj, B.metaproj, and C.csproj concurrently (or at least, in undefined order), which would normally be fine. B.metaproj invokes C.csproj, and waits, then invokes B.csproj. However in the meantime, A.csproj was invoked, and because it has a project reference to B.csproj, it invokes B.csproj — it “goes around the back”. C.csproj hasn’t necessarily built yet, so the build breaks.

Why did this work in previous versions of MSBuild?

In previous versions, we loaded and scanned every project file listed in the solution, and any they referenced, in order to draw a complete graph. Then we used the graph to create the MSBuild format equivalent of the solution file. The reason we did all this scanning was not actually to address this problem, it was to make interop with old-style non-MSBuild VC projects (“.vcproj”) work correctly. It was also slow, especially for large solutions.

In VS2010, VC projects converted to MSBuild, so in 4.0 we took out this complex interop code. After making .metaproj’s to express any dependencies stored in the solution file, we could now simply invoke all the projects in the solution and the build would order itself. That was potentially much faster, because we didn’t need to load any projects (potentially hundreds) to scan them before building anything. (Of course, when MSBuild 4.0 is fed a VS2005 or VS2008 solution file, it still calls into the old code in the old assembly to do it the old way, since they may contain .vcproj’s. So those guys don’t have this problem.) The oversight was this case — where a project reference “goes behind the back” of a solution-expressed dependency.

To fix this we would have to revert to loading and scanning, which slows things down — the correct approach is to use project references instead of solution dependencies, as I explain below.

How to fix this

Follow this principle: do not use dependencies expressed in the solution file at all! Better to express dependencies in the file that has the dependency: put a project reference in the project, instead. In our example, that would be a project reference from B to C.

You may not have done that before because you didn’t want to reference the target of the project reference, but merely order the build. However, in 4.0 you can create a project reference that only orders the build without adding a reference. It would look like this – note the metadata element, and all this is inside an <ItemGroup> tag of course:

If you’re using C++ projects, you are less likely to have this problem, because in the upgrade process that converts .vcproj’s to .vcxproj’s, it moves any solution dependencies relating to them to project references for you. However, if you do, there’s a similar fix. For C++/CLI project references to other managed projects, use project references like the one above. For the equivalent situation with a project reference to a static lib, where you want a project reference without linking in the referenced lib, the metadata you want is

Summary

Although it’s tiresome to have to edit your projects in this way to make the bug go away, it’s a best practice to use project references instead and consider the solution file merely a “view”, and you’ll end up with projects that if you want can be built without a solution file.

Post Script

I know of one other, more obscure and completely different case, where MSBuild 4.0 does not order correctly but Visual Studio does. This can happen if you have web application projects, AND you build with 64 bit MSBuild (which is the default, in Team Build 2010). I won’t go into the tedious details but the fix is to do one of these things: (1) set a property or environment variable named MSBuildExtensionsPath to C:\program files (x86)\msbuild or (2) Build with 32 bit MSBuild, which I recommend in general for other reasons or (3) copy the web application targets files under the 64 bit program files MSBuild folder to the equivalent location in the 32 bit program files MSBuild folder.

If you find any other case where the solution is not ordering correctly yet this workaround does not work, that’s interesting. Please make a minimal repro like the one in this bug, and send it to Dan at msbuild@microsoft.com.

Dan

Update (12/25)

Luc C points out below that sometimes removing a project reference in favor of a new solution dependency reference is an alternative solution. That assumes you’re only using the project reference for ordering , or you replace it with a file reference. Still, I do recommend project references in general, on the principle of “express the dependency in the place it applies” – you can look at the project and see that the dependency is correct, and you can include the projects in more than one solution easily.

Luc also points out the case of a managed project depending on a non-CLR native project (presumably for PInvoke). In my experiments, VS will let you add a project reference, albeit with an ugly bang, and it will do the ordering correctly, as will msbuild.exe.

Tags

What about the issues with batch building which does not respect Debug and Release configuration for builds? E.g. when batch building in some cases Debug dlls may end up in the release output directory!

7 years ago

Vitor Canova

I think that "what" is always important.

7 years ago

Sergey D

Dear Dan,

We tried this approach (with <ReferenceOutputAssembly>false</ReferenceOutputAssembly>) few months ago.

It works well with C++ projects, but in case of C# projects VS shows exclamation mark on the project reference. And, AFAIR, we had some strange compilation behavior periodically.

There is a problem with your proposed solution of adding a project reference directly in the project as far as I can see.

The one place I have been using the dependecies set in the solution file a lot is when C# code is P/Invoking into my own native DLLs – both the the C# project and the native code (C++ project) are projects in the solution, but the C++ DLL is not an assembly, just a plain old DLL, so cannot be referenced in C#. The only way to add this dependency (as far as I can see) is to add the dependency in the solution.

In my scenario the native DLL (a C++ project) is the bubble 'C' in your diagram, and the C# library using it is bubble 'B'. I bumped into the problem described into this post when building 'A': an application using the libray 'B' and therefor indirectly also 'C'. The way I fixed this was by adding a dependency of 'A' on 'C' in the solution.

7 years ago

Dan [MSFT]

@Harry — batch build has a bug (I believe this has been true for a while). It's a VS bug, unrelated to this. The workaround I recommend is to build on the command line with either MSBuild or devenv.exe /build using a batch file. That has the advantage of not tying up VS while it runs, as well.

@Sergey D — the bang is a visual issue only, as far as I know. I'm not aware of the 'strange compilation issue' but if you can narrow it down please open a Connect issue.

@Luc C — (1) I just tried this in VS2010 and it does let me add a reference to a non-CLR VC project – albeit with the bang – and does respect it both in VS and with msbuild.exe (as an ordering-only reference). (2) Good point — I'll update the post to point out that sometimes adding more solution-based dependencies is an alternative.

7 years ago

Owen Wengerd

Dan, you mention a "batch build" bug, but there is also a "clean" bug that might be related:

This makes it still work, but hides it from VS. I hope we can fix this in the next version.

Dan

6 years ago

Jesse Houwing

The problem I have with this setup is that when you use a partitioned solution approach, you need to add all reference assemblies/projects to the satellite solutions.

We prefer to use assembly references for the satellite solutions and one encompassing solution that contains all the project files from all the solutions.

Currently this solution breaks on the buildserver and works great from within Visual Studio.

6 years ago

Pavel Gatilov

@Dan, I could only add such references to 2 class library projects, but couldn’t add it to the main pair of projects: Silverlight application and hosting ASP.NET Web Application. The reference broke the build with the message

error MSB4006: There is a circular dependency in the target dependency graph involving target "CopySilverlightApplications".

Although my solution now builds fine on the build server, the workaround doesn’t fit in all cases.

6 years ago

Rob Reynolds

You might want to ensure you keep the reference output assembly set to false or it will try to bring it in and fail on other references missing:

I believe I've fixed this for the Beta release of 4.5. Essentially during the build of each project we will synthesize any missing necessary non-referencing project references.

When you get the Beta, please consider giving this a try with your original project files, in order to confirm the fix is good.

Thanks, Dan

6 years ago

James Manning

Somewhat related to verifying the fix, but is there any way with devenv on the cmdline (or msbuild if not) to show the dependency graph but not actually execute them? I'm using VS11 Beta if that makes a difference. 🙂

Especially when trying to resolve things like circular dependencies, that would be very useful! 🙂

5 years ago

David

This appear to be an issue in Visual Studio 2010 with SP1 and Visual Studio 2012 RC. I have a solution with several projects and multiple project references for each project (a solution of 150+ projects).

I have the same problem as above and have been able to "temporarily" solve it by unloading the project with "circular references" and reloading them, This appears to cause a rescan of the solution and what Visual Studio 2011/12 thinks is a circular reference is no longer the case. The build order is then rectified.

I initially thought that this problem was fixed when I first used Visual Studio 2012 RC, as I have spent removing lots of unessential project references. I saved it, and it didn't fix… I came in the morning after and it started working again. I could close Visual Studio 2012 RC and reopen and the references were fixed.

A week later after further developing for our build process I decided to make a copy of the source code for a sanity check, copied it to a separate folder… opened the solution and there it was, the circular references. I can open the original location and the circular references are resolved, however when I copy the source or get the latest from TFS it then tells me that the circular references still exist.

Not sure what to think about this, or how inconsistent the experience has been so far.

5 years ago

Ragu

Good one

5 years ago

Sean Hanna

I had an issue with msbuild order being incorrect. Tried the solution in this blog but it didn't help. I am compiling 64 bit, but the solution does not have any web apps. In addition, the project reference was a stranger one, a class library depends on an EXE project. (The exe project was used during a pre-build event of the class library to generate some source code). It was also not using any of the more complicated dependency hierarchies but since search brought me here thought it would be worth pointing out:

I solved the problem by re-ordering the project declarations in the SLN file, and it seems that despite the build order stuff shown above, msbuild was just building the projects in the exact same order they were listed in the SLN file.

Here is a dependencies situation I don't know how to solve in VS2012. The issue can be synthesized as follows: 5 projects: A, B, C, D, E. In solution 1, the dependencies are A, B, D, E. In solution 2, the dependencies are A, B, C, D, E. The issue comes from the project D which depends on B in solution 1 but depends on C in solution 2.

The only way I figured to solve this so far is to have two vcxproj for the same project: One used in solution 1 and the other used in solution 2. But that is not ideal because when doing changes in one project, like adding a new source file, I now have to go and do the same change in the other project file too.

Is there a way to have a ProjectReference section in the same project file that would be conditional in some way?

Yves

4 years ago

Samer Adra

Yves: Yes, you can edit the project file and add <ProjectReference Condition=""…. > node, in which case the reference will only be added if the condition is met. Condition can be added to pretty much anything in MSBuild.