Basic Configuration

Copy sos.dll from the framework directory to the folder where you installed windbg. Make sure you copy it from the same Framework version as the one you wish to investigate. If you'll be working with both 1.1 and 2.0 you can rename the SOS.dlls to SOS11.dll and SOS20.dll or put them in separate folders.

Create a folder where you want to cache all the symbol files. For example: "C:\Symbols".

Start windbg and open the dialogue to configure the symbol path by clicking File -> Symbol File Path.

Enter the path, as well as the address from which you'll want to download missing symbols using the following syntax:srv*[cache path]*[symbols path]I'd recommend the following path:srv*c:\symbols\public*http://msdl.microsoft.com/download/symbols

You should now be set to go. You're now ready to open up a saved dump, or attach to a process

Useful commands

I'll be using a dump from an IIS6-server to demonstrate some useful commands.

The first thing you'll want to do is load SOS. You'll do this using the .load command. The syntax is simple. .load [extension filename]. So if you want to load sos and haven't renamed the .dll you'd simply write:

.load sos

You'll now have all the cool commands from the SOS-extension at your disposal as well as the default windbg ones. Commands from extensions are always preceded by a "!", so if you want to run the help-command for sos you'd write

!help

If you should happen to have two extensions with an identically named command you can always separate them by typing ![extension name].[command] Example:

!sos.help

Okay, now that we know how to run the commands from the extension, try running !help. It should give you the following result.

0:000> !help-------------------------------------------------------------------------------SOS is a debugger extension DLL designed to aid in the debugging of managedprograms. Functions are listed by category, then roughly in order ofimportance. Shortcut names for popular functions are listed in parenthesis.Type "!help " for detailed info on that function.

For more documentation on a specific command, type !help [name of command]

.time

This is not an SOS-command, which is evident by the command not beginning with a "!". Running the .Time command will show you relevant info about the time, as well as system uptime, process uptime and the amount of time spent in kernel & user mode.

As you can see the system has been up for over 4 days. The process has been running for 24½ minutes and has an accumulated CPU-time of about 8 minutes total. This would give us an average CPU-usage for the process of around 32,5%

!threadpool

We can then use the !Threadpool-command to find out exactly what the CPU-usage was at the time the dump was taken. We'll also get some useful information like the number of work requests in the queue, completion port threads and timers.

As you can see the total amount of time does not match the total CPU utilization time that we got from the .time command. That's simply because threads get reused and recycled. This means that the total amount of CPU time used by a thread may have been split up over several page requests.

!threads

To get more information about the running threads we can run the !Threads-command. This will list all managed threads in the application, what application domain the thread is currently executing under, etc. The output will look like this:

The threads with an ID of XXXX have ended and are waiting to be recycled. We can also see that the finalizer thread has an ID of 22. So if we'd seen an unhealthy amount of activity on thread 22 when we ran the !runaway-command we would now have known that we had a finalizer-issue on our hands.

Switching to a specific thread

To go to a specific thread we use the ~-command. The syntax is as follows: ~[number of thread]s. So to switch to thread 50 we would type the following:

0:000> ~50s

We have then switched to thread 50, and can use a lot of other useful commands.

!clrstack

This great command will list the callstack for the current thread. If you want additional information you can add the "-p" switch which will show you parameters and local variables as well.

We can now look at the parameters, like the DirectoryRequest that was sent to the SendRequest and SendRequestHelper methods. To do this we simply copy the address of the request, (0x27246e38) and use it as an argument for our next command.

!dumpobject (!do)

This is another crucial command. Dumpobject will dump the object at the specified address, so if we send the address of the request as a parameter we will get the request dumped to screen.:

Okay, so what's this? Well, it's a System.DirectoryServices.Protocols.SearchRequest object. This means that it has various properties defined by the System.DirectoryServices.Protocols.SearchRequest class. If you want to know more about these properties I suggest you look up the SearchRequest class in msdn. We have the RequestId, the Scope, the DistinguishedName, etc.

So, let's say we want to know what the distinguished name is for this particular request. We boldly assume that the dn-property we see in the listing above is what is called DistinguishedName by MSDN. Simply copy the address of the dn-property (27246d00) and use !dumpobject again. We can see that the property is a System.String, so the output should be pretty clear.

Apparently, the distinguished name used was "CN=Dummy,CN=Accounts,CN=useradm,DC=dummy,DC=net". If we want to find out more we simply continue using !dumpobject to examine the objects and their respective values.

In my next post I thought I'd continue using the !dumpobject-command to probe through the w3wp-process, and introduce a couple of other great commands.

Great article. I’m just getting into debugging for the first time and finding it quite frustrating 🙂 I suppose that comes with the territory.

Perhaps someone can explain the confusion i’m having over something. I’m troubleshooting a problem with a .net app that will grow very large in mem usage and eat up high cpu until it eventually becomes unresponsive. This happens randomly after the app will be up and working fine for hours. I suspect that garbage collection might be an issue (would GC really be suspect of the hang if the problem was that it grew too large and then started too much GC? I think I’m barking up the wrong tree and should be looking for a mem leak)

Never the less, based on the output I get from doing a !threads, I want to take a look at thread 25, which appears like this:

After all this rambling, one of my main questions is this – when I do a !clrstack -p on this same thread, I get output identical as doing it without the -p option. What would the reason for this be? I want to delv into this thread further by using the !do but I cannot.

In your article you said "The threads with an ID of XXXX have ended and are waiting to be recycled.", but you didn’t say what is considered "ended". I thought that IIS will continue using the same threads over & over again from its thread pool and thus would not end any threads in the threadpool. Is having "DeadThreads" normal or does it require some investigation? Thanks.

The !threads-command only lists the managed threads. If you want the native threads you should use ~ instead.

Managed threads are occasionally ended when there’s nothing left to do. One thread may handle a couple of requests and then end. Having ended threads on the heap, like in the sample above, is completely normal.

Gr8 article. I came looking for help on debugging memory fragmentation for unmanaged applications. I have a simple CC++ application running on WS2K3 SP1. I’m not sure, loading SOS in my WinDbg will do me much good. Could you give me some information on WinDbg commands for unmanaged applications.

Also, I’ve been using !heap -l to find out leaked heaps, but can I be really sure all the heaps displayed are really leaking memory.

What can be done to investigate threads that have ended and are waiting to be recycled?

I’ve got an intermittent ASP.NET worker process crash. Running the !threads command on the dump shows a managed thread with a StackOverflowException. Is there any way to get the stack trace on that thread?

If not, do you have any suggestions for setting up adplus to do a memory dump that does have the stack information?

about those threads with an ID of XXXX, should they go away after certain amount of idle time like 2 minutes? I am trouble shooting an application while the application should sit idle (because of no stimulation), however it is still using quite some CPU time and and threadpool threads, where it should not. So I created an adplus dump when it is sitting idle, and I found that there are many threads with ID XXXX, and I created a dump file again after 15 minutes, again, it still have the exact same XXXX threads. All those threads are completion port threads, and I wonder why they did not get recycled?

Also, when doing a ‘~*e !clrstack’, most of the worker threads and completion port threads are showing "Failed to start stack walk: 80004005". Is there a way to show the stack for those threads, because those are the threads I am intereted in.

The GC will be triggered when you’re allocating memory. If your application is inactive, then so is the Garbage Collector. Even if you’re making a few, random allocations they still may not be enough to trigger a GC, so this is not at all unexpected.

The “Failed to start stack walk: 80004005”-error is displayed when the thread did contain a managed stack, but no longer does.