Formerly the core server PM for IIS 7.0 and ASP.NET, now I run LeanSentry.

In a previous post titled Where did my IIS7 server go? Troubleshooting "service unavailable" errors, I covered the basic steps for dealing with cases where the IIS7 web server fails to process the request. This error mostly occurs whenever HTTP.SYS, the kernel HTTP driver that manages http connections for IIS, fails to create an IIS worker process to process the request.

This failure is typically caused by a critical error during worker process initialization, or more likely an unhandled exception / access violation occurring during worker process startup. The most common instance of this is a module AVing inside its RegisterModule function during worker process initialization. It can also be caused by the process going away due to an unhandled exception / AV during the processing of the request that causes the process to be torn down.

After a certain number of failures, the application pool will trigger Rapid Fail Protection, a WAS feature designed to stop application pools with a persistent failure condition to avoid an endless loop of failing to start worker processes. At this point, all requests to applications within the stopped application pool will result in the 503 error, and the application pool will need to be re-started manually.

My previous post covers the steps necessary to diagnose most of the error conditions that may cause 503 errors. In most of these cases, there should be an event log event in the "Application" event log that indicates why the worker process has terminated.

However, in the case of an unhandled exception or AV, you may need to run the worker process under a debugger to get more information about the error. This can give you critical insight into the problem, including the DLL causing the exception, and the exception stack trace, which you can then use to remove the offending module, or provide details to the developer to fix the problem.

2) Restart the application pool if it entered RFP (rapid fail protection) and is currently stopped:

> %windir%system32inetsrvappcmd start apppool <APPPOOLNAME>

Where <APPPOOLNAME> is the name of your application pool.

3) If the problem is happening during request processing of a particular request, you can try to make a different request to the site that does not cause the exception, and then attach the debugger by running the following from the command-line:

> windbg -g -p <PID>

Where <PID> is the PID of the IIS worker process you are debugging. You can get it from Task Manager or by running tasklist /fi "imagename eq w3wp.exe".

If you have the debugger attached at this point, skip steps 4 and 5

4) If the problem is happening on every request, then you won't have a worker process running that you can attach to since it will go away as soon as you make a request. In this case, you will need to attach a debugger immediately after the worker process is started, so that you can catch a startup exception before it tears down the process.

This is where startup debugging comes into play. It is done by configuring the debugger to attach to the process as soon as its started, by setting the Image File Execution Options registry key from the command line:

This creates a sub-key named "W3WP.EXE" in HKEY_LOCAL_MACHINESOFTWAREMicrosoftWindows NTCurrentVersionImage File Execution Options, and in that sub-key creates a String value named "Debugger" and set its value to "c:debuggersntsd.exe -server npipe:pipe=w3wp -g".

For this to work, you have to make sure that the path to the debugger is correct. If the debuggers are already in the path, you can omit the full path.

5) Now, lets attach.

- Make the request that causes the crash

This will trigger the IIS worker process to be started, and will automatically attach the debuger, breaking into the process before any IIS initialization takes place.

- Now, connect to the debugger from the command-line:

> windbg -remote npipe:server=localhost,pipe=w3wp

5) Inspect the exception stack trace

The debugger window should now be broken in at the location of the exception. You can inspect it further by typing "k" and pressing Enter.

This should give you the full stack trace of the exception.

If you are missing symbols, you can get them from the Microsoft's public symbol server. To do that, type the following in the debugger:

Hi! Mike,
I am using vista business and VS2005 TS w sp1. I have installed IIS (click everything). I also install latest debugger from the link you provided.
I got this problem 503. I hv tried both way easy 1 and this but unsuccessful. I cant any w3wp.exe and the debugger from my vista. Any idea?

Mike, how are you ?
We are experiencing the 1011 error (A process serving application pool 'AppPool' suffered a fatal communication
error with the World Wide Web Publishing Service. ) and having our IIS crashed several times a week.
We are running on W2003 and IIS 6.0.
I've seen that yu have a procedure for IIS7.0, do you have one for IIS 6.0 ?
Can you help me, please ?
I started running the debug diagnostics but I really don't understand it.
Thanks a lot.

You can apply this technique to IIS6 as well to attach a debugger at the start of the worker process. However, in your case it sounds like your crash does not happen at startup, so you can attach a debugger normally at any time and wait for a crash to occur.

Tools like DebugDiag can do this for you (monitor for a crash, and then attach a debugger).

Strange, it sounds like that detection wasn't working for me then. I was using cdb just as you describe here (not VS or another managed-only debugger). After about 90 sec I'd get an error from WinDbg saying that the connection to the server had been terminated. Disabling the ping stopped that. I saw this on two different Win2k8 machines.

Hi Mike, have a situation for you Our IIS7 is failing with weeks/months in between occurrences. I tracked down errors in the event logs and found the "Application pool 'DefaultAppPool' is being automatically disabled due to a series of failures in the process(es) serving that application pool." and a few ""A process serving application pool 'DefaultAppPool' terminated unexpectedly" before that. I tested and the DefaultAppPool was indeed causing the 503 error that wouldn't go away. Now I know that. I came upon your page here and also your "Where did my IIS7 server go? Troubleshooting 503 "service unavailable" errors" post. Thanks, very helpful! So how to fix the problem that keeps causing it mysteriously with long periods in between failures? I see other issues in the event logs around the same time and see several "Faulting application name: w3wp.exe" and "Fault bucket , type 0, Event Name: APPCRASH, Response: Not available, Cab Id: 0, Problem signature:P1: w3wp.exe" in the Application Log. So I think it's the Application crash scenario you mentioned. I look in the actual web server logs and just see a ton of requests for mysql admin tools which we don't have around that time. So how do I prevent this in the future? Any suggestions? Thanks for any help!

AS THE SERVICE UNAVAILABLE IS GONNA FALL DOWN AND 503 ERROR MESSAGES WILL BE BURNED FOREVER AS THE WORLD IS GOING TO END ON DECEMBER 21, 2012 AS WE KNOW IT DUE TO THE VERY VERY VERY LONG LONG MAYAN CALENDAR AND THE END OF OUR PLANET EARTH AND THERE WILL BE HUGE TSUNAMIS, EARTHQUAKES, HUGE SNOWSTORMS OR EITHER TRIPLE SUNS TO RISES ON OUR PLANET EARTH AND NO FUEL, NO WEATHER NO WATER AND NO OTHERS AND WE WILL BE UNABLE TO BE SURVIVE AS WE KNOW IT AS THE RESULT OF THE BOT PROJECTS TO BE BURNING ALL ERROR MESSAGES PRIOR TO THE END OF THE WORLD AS IT IS NOW NEAR TILL DECEMBER 21, 2012 AS WE KNOW IT.

Hi
I try to make an app pool to accept Enable 32-bit=True ...bu all the website that I try to browse thru that apppool return 503.
Windows Server 2008 R2 SP1 x64,
I tried to browse by that apppool simple app with no db drivers or else.
I tried the trick with rpc ... witch is unusefull ...
I do not know what to do more ...