Windows Drivers Debugging

In this tutorial we are going to see how to setup a Debugging Environment for our Drivers. This is not a complete guide, it’s just a quick tour intended to give a fast overview of Windbg and problems involved into Driver Debugging.

The Problem

Setting up a full working Kernel Debugging Environment is often seen, by novices, as a pretty challenging task. Difficulties comes essentially by the fact that working with a Debugger at Kernel Level implies that entire machine is completely locked for a certain amount of time.

Here pop up the necessity to work with Two Machines ( a physical and a virtual target machine ) in order to enstablish the following relationship:

Host System used to debug and Remote Machine.

Guest System that acts as ther Remote Machine to be debugged.

Became clear that we need a debugger that supports:

Kernel Level Debugging

Remote Debugging

WinDbg debugger fits all our needs, so we must setup correct communication layer between Host and Guest and successively use Remote Debugging functionalities of our debugger.

Setting Up Communication Between Host and Guest

The approach that we will follow is identical to the one described by Lords of Ring0. Where the idea is to setup a Virtual Serial Port Connection that will be used by the Host that have WinDbg to access the remote machine.

You can follow the proposed step by step guide also in case you want to use VirtualBox VM.

When you have completed these steps, let’s build a Batch File that contains the necessary command line arguments to start a kernel mode debugging session:

windbg -b -k com:pipe,port=\\.\pipe\com_1,resets=0

The batch file need to be placed in the same directory where is located windbg.exe

Our First Session

If your installation is correct after executing windbg you should see something like this:

Last line tells you that WinDbg is now holding the control of your Guest System, indeed by switching to the Virtual Machine, if things went well OS should be completely frozen.

But as you can see, the debugger alert us that there are some errors about Symbols, don’t worry it’s normal, we need to setup correct Debugging Symbols now.

Setting Up Symbols

Debugging Symbols are extremely important components that Correlates the Assembly with Source Code Level Informations, a great amount of informations necessary for a Correct and Painless debugging session. It’s important to distinguish between:

Public Symbols

Private Symbols

First thing is to verify that Windbg can find symbols, this is done by setting the Symbol Path, from command line:

Successively would be good practice to increase verbosity level of Symbol Loading Operations, we can use the following command:

kd> !sym noisy

And finally reload symbols:

kd> .reload /f

At this point Symbols if symbol reload and resolution goes well, our Windbg is working with the correct symbols.

Debugging Our Driver

System is now ready to debug our driver, we just need to deal with our Driver Project which is composed by:

Source Codes

Binary Driver File

The first step is to place the entire driver project (Src+Bin) into a shared folder between host and guest.
Otherwise, the .sys need to be loaded into Guest Machine.

Just a side note, our debugger still holds the control of the target machine, to release control just press F5 until you see ‘Debuggae Running’.

Time now to instruct Windbg to hold control of our driver, let’s assume you want to Debug from early stages of Driver Execution, we must put a BreakPoint on DriverEntry() function.

So before starting starting the driver we must set a breakpoint in this way:

kd> bu module!name

Assuming that driver is called Test.sys

kd> bu Test!DriverEntry

Windbg still holds the control of the target machine, to release control just press F5 until you see ‘Debuggae Running’. We can now Load the driver with OSROnline DriverLoader:

After loading it, guest machine is again frozen, because Windbg stopped execution at DriverEntry point.

Organizing Work Space

Windbg is rich of views that can greatly improve debugging experience, here a screenshot of most handy ones:

Windbg Commands

In the following subparagraphs we will talk about Commands and Functionalities of Windbg that could come handy during the analysis.

The essential aim of these paragraphs is not to give a complete syntax description, but more easly to inform the reader about the existence of certain features.

It’s strongly suggested to people that wants further informations about Windbg Commands to read debugger.chm help file, delivered with Debbugging Tools.

Some of the commands samples reported are taken from kerned_debugging_tutorial.doc a file that can be found in the working directory of Windbg.

Managing BreakPoints

The simples for of breakpoint is the bp one, that can be used in the following forms:

kd> bp Module!Name

or

kd> bp address

As you have previously seen, to break into our driver we used bu, this is a Deferred Breakpoint.

If a breakpoint is set for a routine name that has not been loaded, the breakpoint is called a deferred, virtual, or unresolved breakpoint. Unresolved breakpoints are not associated with any specific load of a module. Every time that a new application is loaded, it is checked for this routine name. If this routine appears, the debugger computes the actual coded address of the virtual breakpoint and enables the breakpoint.

If you set a breakpoint by using the bu command, the breakpoint is automatically considered unresolved. If this breakpoint is in a loaded module, the breakpoint is still enabled and functions normally. However, if the module is later unloaded and reloaded, this breakpoint does not vanish. On the other hand, a breakpoint that you set with bp is immediately resolved to an address.

Should be now clear why we used in our sample an unresolved breakpoint instead of bp.

We can also see all breakpoints we defined, via bl (Breakpoint List Command)

kd> bl

e stands for enabled and d for disabled.

Breakpoints can be disabled via bd (Breakpoint Disable)

kd> bd bp_index

Otherwise can be permanently removed with bc (Breakpoint Clear)

kd> bc bp_index

It’s also truly important to mention Conditional Breakpoints. They cause a break to occur only if a specific condition is satisfied.

A conditional breakpoint is created by combining a breakpoint command with either the j (Execute If – Else) command or the .if token, followed by the gc (Go from Conditional Breakpoint) command.

Generically:

kd> bp Address “j (Condition) ‘OptionalCommands’; ‘gc’ “

A pratical sample:

or

kd> bp mydriver!myFunction “.if @eax = 0xa3 {} .else {gc}”

Break on myFunction only if eax register is 0xa3.

It’s also pretty interesting to put in evidence the fact that Windbg supports per Process and per Thread break conditions.

bp /p 0x81234000 SIoctl!SioctlDeviceControl+0x103

or

bp /t 0xff234000 SIoctl!SioctlDeviceControl+0x103

Respectively, Stop at the indicated point only if the EPROCESS is at 0x81234000, and Stop at the indicated point only if the ETHREAD is at 0xFF234000.

Data Displaying

We can Display the content of a range of memory in various ways, commands designed for that task have the following form:

d* [Options] [Range]

da -> ASCII characters.

du -> Unicode characters.

db -> Byte values and ASCII characters.

dw -> Word values (2 bytes).

dd -> Double-word values (4 bytes).

Thanks to Symbols support we can directly display also the values of local variables:

dv [Flags] [Pattern]

From a pratical sample:

kd> dv Irp
Irp = 0xff70fbc0

Pretty useful is also dt that displays information about a local variable, global variable or data type. This can display information about simple data types, as well as structures and unions.

Data Editing

In the previous paragraph we have seen how to read data, but undoubtedly we have to know also how to modify data.

For this scope, windbg offer us:

kd> e* Address [Values]

pattern is similar to display command:

eb -> Byte values.

ew -> Word values (2 bytes).

ed -> Double-word values (4 bytes).

If our editings needs require to fill entire blocks of memory with some static pattern, we can use f (fill) command:

f Range Pattern

Examining a BugCheck

Investigating causes of a crash is a core aspect of driver debugging. We have at disposition a pretty handy debugging extension called !analyze. This extension displays information about the current exception or bug check, most common way to use it, is:

As you can see we have and handy summary of the Faulting Module plus Details about the Fault. We have enough points to start an in depth analysis, here the starting points that we should keep always in mind:

Bucket ID

Stack Trace

Exception Parameters

Let’s see now the commands that can help us to analyze these points.Stack Backtrace can be displayed with:

Now suppose that we are working on a MultiThreaded target, we can introduce at this point ~ which is a Per-Thread-Identifier:

~. -> The current thread.

~# -> The thread that caused the current exception or debug event.

~* -> All threads in the process.

In our case:

kd> ~*k

Will show the stack backtrace of each thread, awesome isn’t it? 🙂

Exception analysis is another important aspect of fault analysis. The core command is .ecxr that displays the context record that is associated with the current exception. Please note that .ecxr is available on Crash Dumps only (minidumps)

A dump can be created with .dump command:

kd> .dump Options FileName

Finally here a list of commands that could come really handy during debuggingUseful Commands

Debugger Extensions

Together with commands we have also a large set of Extensions, they are of great help in case of Well Defined Specific Debugging Scenarios. Where we can detect a well defined pattern, we can use or write extensions.

Extension commands are exposed by DLLs distinct from the debugger.

An extension is Loaded with:

kd> .load DLLName

and Unloaded with:

kd> .unload DLLName

To List Loaded Extensions:

kd> .chain

It’s also pretty easy to distinguish between a classical command and an extension:

Commands starts with .

Extensions starts with !

Debugger extensions are invoked by the following syntax:

![module.]extension [arguments]

Handy Extensions

In this subparagraph we will see some available extension specific for Driver Debugging.

The !process extension displays information about the specified process, or about all processes, including the EPROCESS block.

Link References

The End

Here ends this little tutorial, again would like to put in evidence that this is only a reference tutorial, intended to offer a path to guide an hypothetical novice and give him, searching starting points.