October, 2006

Here is an interesting problem I worked on a couple of weeks ago for a customer who raised a call with Technical Support: they had an ASP.NET 2.0 application written with Visual Studio 2005, and while working on it they found that when changing a code file contained in App_Code folder and then building the site from the Visual Studio menu command, this took up to 7 minutes to complete... and if they then run a page pressing SHIFT+F5 it took about 25 seconds to compile and display the page. If they just modified the HTML layout of the pages everything worked fine; moreover, it's interesting to note that if they simply modified and saves their files without building the solution in Visual Studio, but just browsed the page (thus relying on ASP.NET to compile on the fly their code) again everything worked as expected. This seems quite clearly a problem with Visual Studio.

It turned out that this project was originally build with Visual Studio 2003 and ASP.NET 1.1, and they decided to upgrade it to ASP.NET 2.0; they used the upgrade wizard provided by Visual Studio 2005, which will turn on batch compilation (<compilation batch="true" /> in web.config) when it will detect circular references in the project.

When batch compilation is disabled, ASP.NET will create a separate assembly for each code behind file; in this case the solution had about 3000 "code" files, which would have generated 3000 separate assemblies when compiling and executing the application, each of them individually compiled (with csc.exe or vbc.exe), and as you can imagine this will affect performance. However, we do recommend to set this flag to false immediately after running the wizard in order to catch and fix possible broken assembly references; after these conditions are fixed, you should set batch="true" (or remove the batch tag) again.

By the way, if you set batch="true" the compilation model does the building based on each directory, which means it builds one assembly for each directory (if pages' languages are different we build separate ones). If you request the page through IIS outside Visual Studio, the runtime only builds the directory where the requested page is located. If you build the web inside Visual Studio we build all the directories, and again this is why it takes longer.

Changing the content in “App_Code” folder will cause clean build throughout all the web site. You can think of the “App_Code” folder is a project, and each web page is a separate project which all reference to the “App_Code” project; if the “App_Code” is updated, we will have to rebuild all the web page projects. Incremental build might make sense within each project, but not cross projects scenarios; this is a change from the Visual Studio 2003 project system and I have to say it is hard for users to realize...

In this situation I recommend you turn off “Build web”: you can do this by opening the “Property Pages” dialog, switching to “Build” tab and setting the “Before running startup page” to “No Build”. Now you will have the delay only caused by runtime (~45 seconds). You can manually invoke “Build web” anytime later if you want to validate those pages.

But now compilation of App_Code alone happens only when I hit F5 (to debug)... If you choose "Build", it still compiles App_Code *and* web pages. How could you then build only App_Code, without hitting F5?

The feature of building "App_Code" only was cut from Visual Studio 2005; the closest approach is to build one aspx page ("Build"->"Build page"), which will first build "App_Code", and then that specific page, while no other pages will be built.

I just realized that my last post is 20 days old... I spent this time working on support calls I own (or course, that's my job!), reading documentation and playing to get ready for the upcoming releases I'm quite sure you already heard about (Vista, Ajax, IIS 7, Internet Explorer 7...), reading some good books in my spare time (currently I'm almost done with The Twelfth Card by Jeffery Deaver) and among other things I've been searching (and found) a new home. (well, actually my girlfriend decided where we'll go to live for the next years, I guess you know how this king of things works...).

While I had to deal with the contracts and terminology I'm not familiar with, I realized that sometimes people tend to give some things and facts for granted, while other people may not know what we are talking about...

This also apply to our business: I had some conversations with our customer getting "Application Server Unavailable" errors in their ASP.NET sites, and then we discovered that they tried to execute two different CLR versions (usually ASP.NET 1.1 and 2.0) in the same worker process... This is not allowed, and I always thought this was clear, but a couple of customers asked me for some written documentation about this limitation, and despite my researches I've not been able to find something specific enough for my needs. I'm considering to write a post about it, so a couple of days ago I sent an email to my team alias asking if someone could point me to such docs. I had a short but interesting email thread with Doug, and basically this is an abstract of that conversation:

Why does the customer need such documentation? This is just "common knowledge" to most people that have worked with the CLR for any length of time. The core runtime is implemented in a native DLL - mscorwks.dll (and sometimes mscorsvr.dll) and as most Win32 developer know you can only have one version a DLL loaded in a process (unless you get into Win32 SxS), It's a bit like some saying to Fiat "where is the documentation that you can only have one engine per car?".

If you install the 1.1 SDK, and look at the doc for the CLR hosting interfaces (\Program Files\Microsoft.NET\SDK\v1.1\Tool Developers Guide\docs\Hosting Interfaces.doc) you'll find statements like"CorBindToRuntimeEx is the primary API that hosts use to load the CLR into a process""GetCORVersion returns the version number of the CLR that is running in the current process."

It is implicit in the use of language here that there is only one per process. If you could have multiple CLR per process we would have written something like"GetCORVersion returns the list of version numbers of the CLRs that [is]are running in the current process."

Unfortunately I already tried that path in my talks with customers, and a couple of them who simply said "If it was written somewhere I would have not run into this error”; questionable statement (not because I don't trust our customers, but because it's hard to think that one can read and remember all the documentation Microsoft and other IT companies release about their products, at least I have to admit I'm not that kind of person), but they still wanted it…

To enforce this, a couple of weeks ago I took a case raised by a customer who had some random crashes in his web application, and in this case the easiest thing we can to to troubleshoot the problem is capture a crash dump with adplus.vbs to analyze. As always, after a phone call with the customer to discuss the problem to gather a better picture about the application the problem, I sent him an email (actually a template I use for this kind of problems to save time and not continue writing the same things over and over for every customer) with the action plan and detailed steps to setup the debugger. Easy... then we just had to wait for the crash to happen again.After a few days I had a phone call from this customer telling that they had the crash, but no dump was captured... we tried to understand what went wrong and after some discussion he told me that he connected to problematic web server through a Terminal Server session. Nothing strange on this, but he then told me that they configured their server to automatically end a user session after 30 minutes of inactivity...

That's the problem! If the terminal session is terminated, it's easy to imagine that also the processes (like adplus) you left running in that session will be terminated... so when the next crash occurred the debugger was already dead, and we missed the dump (and also wasted almost one week because we had to configure adplus again on the console session and wait another 4 days to get the new crash and the dumps).

When I spoke to that customer I had the clear impression that he was a well prepared technician and probably with lot of experience, but he claimed that the KB article I sent him (http://support.microsoft.com/?id=828222) doesn't say if the terminal session has to be left open, and he logged off when done... Again, sometimes it obvious but sometimes it’s not, at least not for everyone…