In my professional career, I have seen most of us use Visual Studio for debugging but not many of the other debuggers that come for free. You may want such a debugger for many reasons, for example, on your home PC which you do not use for development but on which a certain program crashes from time to time. From the stack dump, you can figure out if IE crashed because of a third party plug-in.

I did not find any good quick starters for WinDbg. This article discusses WinDbg with examples. I assume you know the basic concepts of debugging – stepping in, stepping out, breakpoints and what it means to do remote debugging.

Note that this is meant to be a Getting Started document, which you can read and start using WinDbg. To know more about specific commands, consult the WinDbg documentation. You can use the commands presented in this document with any debugger provided by Microsoft, e.g. from the Command window of Visual Studio .NET.

This article is based on WinDbg 6.3.

This is the first of a series of articles on debugging. In my next article, I shall explain how to write debugger extension DLLs.

WinDbg is a debugger that wraps NTSD and KD with a better UI. It provides command-line options like starting minimized (-m), attach to a process by pid (-p) and auto-open crash files (-z). It supports three types of commands:

PDB files are program database files generated by the linker. Private PDB files contain information about private and public symbols, source lines, types, locals and globals. Public PDB files do not contain types, local and source line information.

Doing remote debugging using WinDbg is easy and can be done in one of a number of ways. In the following, ‘debugging server’ is the debugger running on the machine where you’d like to debug; ‘debugging client’ is the debugger controlling the session.

Using the debugger: You need CDB, NTSD or WinDbg on the server. A WinDbg client can connect to any of CDB, NTSD and WinDbg, and vice versa. The server and client have choices of TCP and named pipes for communication protocol.

Using remote.exe: remote.exe uses named pipes for communicating. If you use a console-based application like KD, CDB or NTSD, you could use remote.exe to do remote debugging. Note: use @q (not q) to quit the client without quitting the server.

To start a server:

Remote.exe /s “cdb –p <pid>” test1

To connect from a client:

Remote.exe /c <machinename> test1

test1 above is the arbitrary named pipe name we chose.

Server will display who all are connected from which servers and commands executed. You can quit the server by issuing ‘qq’; or quit the client using File->Exit. You’d need to belong to the “Debugger Users” user group and the server has to allow remote connectivity if you want to remote-debug.

The section “Enabling Postmortem Debugging” in the WinDbg documentation discusses this well. In short, you can set WinDbg as the default JIT debugger by running Windbg –I. This sets the registry key HKLM\Software\Microsoft\Windows NT\CurrentVersion\AeDebug to WinDbg. To set WinDbg as the default managed debugger, you’d need to set these registry keys explicitly:

HKLM\Software\Microsoft\.NETFramework\DbgJITDebugLaunchSetting to 2

HKLM\Software\Microsoft\.NETFramework\DbgManagedDebugger to Windbg.

With the JIT setting, WinDbg will be launched if an application throws an exception while not being debugged and does not handle the exception itself.

WinDbg 6.3+ supports managed debugging, with the Whidbey .NET CLR. There is a good discussion on managed debugging in the documentation. Remember that there are no PDBs with managed code since managed code is compiled to ILASM; the debugger talks to the CLR to query extra information.

Points to note:

You can set a breakpoint at a managed code function only after it has been invoked at least once; because that is when it is JIT-compiled to ASM code. Keep in mind:

Complications with function addresses and hence breakpoints:

The CLR can discard compiled code, so function addresses may change.

The same code may be multiply compiled if multiple app domains do not share the code. If you set a breakpoint, it gets set for the app domain of the current thread.

Specialization of generics can cause multiple addresses for the same function.

Complications with data layout and hence data inspection:

The CLR may change data layout arbitrarily at runtime, so field offsets in a structure may change over time.

Type information is loaded only on first use, so you may not be able to inspect a data field if it has not been used yet.

Complications with debugger commands:

When tracing through managed code, you would pass through chunks of runtime code like the JIT compiler code because you stepped into a function for the first time, or, when transitioning from managed to unmanaged code.

You can debug a service just as any other application using WinDbg, both after starting the service by attaching to the service process, and, by using WinDbg as a JIT debugger and programmatically calling DbgBreakPoint or DebugBreak, or an ASM int 3 on x86.

A debugger gets notified of each exception twice – it is notified the first time before the application gets a chance to handle the exception (‘first chance exception’); if the application does not handle the exception, the debugger is given a chance to handle the exception ( ‘second-chance exception’). If the debugger does not handle a second-chance exception, the application quits.

.lastevent, or, !analyze –v will show you the exception record and stack trace of the function where the exception occurred.

You can also use the .exr, .cxr and .ecxr commands to display the exception and context records. Note also that you can change the first-chance handling option for an exception using the sxe, sxd, sxn and sxi commands.

Debugger extensions are DLLs that you can hook up with a debugger to execute custom commands from within the debugger. There are certain functions that a DLL needs to implement and some requirements that a DLL needs to meet in order to qualify as an extension DLL. In the next article, we shall learn how to write an extension DLL yourself. The bang (!) commands are commands executed from your extension DLLs. Note that extension DLLs are loaded in the process space of the debugger.

You can take snapshot information of a process using the dump facility. A mini-dump is usually small, unless you take a full-memory minidump (.dump /mf). It is useful to dump handle information also, as .dump/mfh. A mini-dump contains information about all threads including their stacks and list of loaded modules. A full dump contains more information, like that of the process heap.

If your Windows OS crashes, it dumps the physical memory contents and all process information to a dump file, configured through System->Control Panel->Advanced->’Startup and Recovery’. It is also possible to take dumps of any live process by breaking into it. You can also take a dump of any process (.dump) that terminates abnormally by configuring WinDbg as a JIT debugger. Note that figuring out bugs in the code from a crash dump could be an involved process.

To analyze a dump, follow these steps:

Step 1: In WinDbg, File->’Open Crash Dump’, and point to the dump file

Step 2: WinDbg will show you the instruction your app was executing when it crashed.

Step 3: Set your symbol path and source path properly. If you cannot match symbols, you could have a hard time figuring out control flow. If you can match the symbols to source code of the appropriate version, it should be easy to figure out the bug at this point. Note that private symbol files have line number information and will blindly show the line in your source code without further checks; if your source is not version-matched properly, you’d not see the correct source code matching the assembly code. If you have public PDB files, you’ll see the last public function (on the call stack) that was invoked.

Note that debugging drivers or managed code is much different. Refer to [2] for debugging techniques for device drivers.

You need symbols in order to be able to do effective debugging. Symbol files could be in an older COFF format or the PDB format. PDBs are program database files and contain public symbols. These debuggers allow you to mention a list of URIs where they would look for symbols for loaded binaries.

OS symbols are usually installed in the %SYSTEMDIR%Symbols directory. Driver symbols (.DBG or .PDB files) are usually in the same folder as the driver (.sys file). Private symbol files contain information about functions, local and global variables, and line information to correlate assembly code to source code; symbol files that are usually made available to customers are public symbol files – these files contain information about public members only.

You can set symbol directories through File->Symbol File Path, or using .sympath from the WinDbg command window. To add reference to a symbol server on the web, add:

SRV*downstream_store*http://msdl.microsoft.com/download/symbols

to your .sympath, thus:

.sympath+ SRV*c:\tmp*http://msdl.microsoft.com/download/symbols

Where c:\tmp is the download_store where necessary symbols will be downloaded and stored. Note that this particular symbol server exposes public symbols only.

The debugger matches information like filename, timestamp and checksum when matching a PDB with a binary (DLL or exe). If you have symbol information, you’d be able to see function names and their arguments in your call stack. If the binaries and PDBs are from your application, you’d additionally have information about private functions, local variables and type information.

The sympath can consist of multiple URIs. Sympath is initialized from the _NT_SYMBOL_PATH system environment variable.

You can set source code directories through File->Source File Path, or using .srcpath from the WinDbg command window. If you set source code directories, the debugger will pull up matching source code based on line number information from the PDB files during debugging.

There is no “step out” (Shift+F11). You have to find the return address on the stack manually and use “g adr”. You can find this address by using “k”. If you know the function uses ebp frames you can use “g poi(ebp+4)” to step out.

To inspect local variables:

Use the “dv” command.

Then use the “dt <variablename>” command.

Note: you may not see correct values if values are stored in registers or due to FPO.

when you run WinDbg, attach to a process and issue kb, you’d be seeing the stack trace of the thread injected by the debugger. All debugging commands are executed in the context of the injected thread.

Frame Pointer Omission (FPO):

Means that when your code is compiled, frame pointers (EBP) will not be put on the stack. This makes function calls faster and makes the EBP register available as a scratch register. The optimization option /Oy in the MSC++ compiler => FPO; /O2 or /Ox (full optimization) => /Oy.

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

Comments and Discussions

Thank you for this article - I'm still reading it, but it's very informative so far. I came across a minor inconsistency that wanted to bring to your attention; in the text below (prefixed by ARTICLE-SNIPPET), if "downstream_store" were to be replaced by "download_store" (or vice-versa), the terms would be consistent.

For instance, if you were to replace "downstream_store" with "download_store", we would have:

To add reference to a symbol server on the web, add:
SRV*download_store*http://msdl.microsoft.com/download/symbols

ARTICLE-SNIPPET:
Symbol Files and Directories

To add reference to a symbol server on the web, add:
SRV*downstream_store*http://msdl.microsoft.com/download/symbols
to your .sympath, thus:
.sympath+ SRV*c:\tmp*http://msdl.microsoft.com/download/symbols

Where c:\tmp is the download_store where necessary symbols will be downloaded and stored. Note that this particular symbol server exposes public symbols only.

*** ERROR: Module load completed but symbols could not be loaded for Test1.sys
*** ERROR: Module load completed but symbols could not be loaded for UMATest.exe
Probably caused by : Test1.sys ( Test1+55b )

As we .NET Developers dive deeper into the MSFT OS, we find that there really is no substitute for WINDBG for the worst of the worst. I used this article today after having read it 5 years ago to solve an issue on our production servers today. Thanks for this post!!!!!

This article will be valid for a long time...it is timeless.

TIP: If you use WINDBG for .NET, make sure to first load SOS... ".loadby sos mscorwks" You can avoid parsing method tables looking for values by running !dso which will show values of parameters.

You said to run WinDbg on the server. But how do you get that loaded during boot so you can debug a driver that loads during boot? It seems to me that I remember from long ago that you add something to boot.ini. Does anyone know?

First, you need to install your debug driver along with its symbols. Then run msconfig on the target machine and go to boot -> advanced options. Click on "Debug" to enable the lower panel and select how you wish to connect: COM, Firewire, etc. Note down the baud rate/firewire port you set there and then make sure to have the host system ready and running WinDbg with the same parameters in "Kernel Debug Mode". Reboot the target machine. If all goes well, upon reboot you should connect with the target machine. You'll also need a crossover COM cable and the corresponding COM port (you can also set up virtual COM ports with an USB-to-COM port driver). Alternatively you can also use Firewire or maybe a regular network cable, not sure about that. That's more or less how to set that up.

hi, now in my office, my computer can't connect the internet, ,the way you mentioned
"SRV*downstream_store*http://msdl.microsoft.com/download/symbols" may be a big problem
to me.So, I want to know that, how can I set the symbol without the internet? Any suggestion?
Thank you!

If you have the windows installation CD it will have the symbols in the support folder. Once you install it from the CD the symbol path is automatically set. If not, just remember the path it extracted the symbols to and set the path using .sympath command.

I have a Ajays USB debug cable, and I want to link two EeePCs(no 1394, no COM Port) to do some debugging works. I have use bcdedit change the boot entry, but I still can't link WinDbg(KD) sucessfully! Do you have any experience or sollution about this issue? Thanks for your help!

I am trying to run the example source in my VS 2005 to generate symbols and images for Windbg.
But I get a lot of compiling errors. .vsproj file was not included in the download.
All the errors are from wdbgexts.h/
My machine is Vista 64 and using VS 2005.
I created a new Windows console application.

I'm no expert but I think it depends on the type of minidump file (there are several levels of detail). See the /m command for ntsd. By default the heap is not dumped so you will not get stuff from there. Try using the /ma switch instead of the default /m

hi
I'm new to debugging softwares... I'm using windbg and couldn't understand anything wht exactly is happening inside it..... can anyone post some useful links or material regarding how to get started?

I found windbg very helpful. Indeed this is the tool I use for everyday debugging.
It can be used for both user mode and kernel mode debugging.
The only problem I feel is the inconvenient way in which it allows me to access code. It is not as user friendly as visual studio debugger.
I would be happy to know the reason why we should use windbg and not Visual Studio for user mode debugging.

I imagine this is personal preference. In a perverted way, I know of a few who prefer command line debuggers. For example those who came from the early Unix and Linux who are masters at gdb.

A little known factiod: WinDbg is mainted by the Operating System team at Microsoft, while Visual Studio is a product of the Development team. So WinDbg is much more intimate with the OS and its structures. For example, how does one view the PEB in Visual Studio?

Is it possible write a small script that can attach to running process
and run few commands like list of threads and get stack trace of all
threads and then detach without terminating the target process?
And then I would like to save the output in a file.

As a starter, using windbg is very difficult.
e.g. I am building a .dll with .pdb file and
and .exe with .pdb file. I am using a Makefile
to build these. The probelms I face are as
follows:
a. How can I debug this exe and the dll in Visual Studio?
b. In WinDbg I set the path to source, symbols and image
correctly. (In my case these are in <homedir>/Debug)
c. When I open the .exe it gives errors about symbols
- Checksum could not be found for DLL and EXE
- Symbols could not be loaded.
d. For any variable it says
- typeinfo not found

I am finding it painful to debug using windbg and dont know how
to debug in Visual Studio

To debug an app (exe) in Visual Studio (VS) not built from within Visual Studio, you can do one of two things:
1- run the process (if this is a service, start it from the command line or from within Service Control Panel) and attach to the proces using VS. If you're using VS 2005, Debug->'Attach to Process' will do it.
2- start the process from within VS. To do this, create a solution and add the exe to the solution (you don't need to add source files to the project) and Run.

The above will not take care of situations like problems that arise on app start-up. If this is your situation, please reply to this thread with more info and I can answer better.

To debug a dll in VS, attach to the process that has loaded the dll that you've compiled.

If you're doing this exercise on the machine on which you've built the binary (dll or exe), VS should be able to find the pdb. If this is a different machine, place the pdbs on some folder and point VS to the folder for symbols.

The above applies to windbg too. Attach to an existing process (windbg -p <pid> or windbg, then select File->Attach to a process). You can start a process from within windbg (File->Open Executable) or when launching too.

Hi,
I'm a WinDbg newbie.
The 'vertarget' command doesn't display the name of the computer on which the crash occurred, the way you get it in a Dr. Watson log. Is there any command available in WinDbg to do this?
Thanks in advance.
Regards,
Vidya.

I have an application that creates some processes (executables that could be seen in the Task Manager as a processes). when I run the application from command line, both Dr. Watson and WinDbg can create dump files when one of the processes was crashed. But when I run the application as a service dump files are not created when any of the processes was crashed. I tried it on Win 2000 and Win 2003.
Does anybody now what should be done to intercept the crashes by these postmortem debuggers?