Setting Up Kernel Mode Debugger in Windows

This article will guide you through the process of setting up kernel mode debugger in Windows

Introduction

Whenever there is a bug in your program, you usually open a debugger (GDB, Visual Studio debugger, etc.) to fix it, but how do you debug a bug in the operating system? Do you load the running OS into the debugger? Is it possible? The simple answer is no, Because even the debugger works with the help of OS. It's a Catch 22 situation.

History

Earlier developers used two machines; one is the defective OS(slave) and other machine is the one containing debugger software(master). Now the defective slave is connected to master machine using a high speed cable, and then both slave and master machines are started, with the help of debugger the execution of slave machine is paused. But this solution has some drawbacks.

The connection speed between the machines is too slow, because the data and commands should be passed to and fro between master and slave.

Requires extra hardware like cable and two separate machines

Current Process

Fortunately, we now have much better options for beginners who want to study the internals of the OS by debugging. With the help of virtual machines, now we do not require two separate machines. The slave machine can be thought of as a guest VM and master machine can be thought of as a host computer(your real physical machine). The connection between these host and guest have been made even simpler with the help of a software called VirtualKD(Virtual Kernel Debugger)[without this tool, we have to manually set up a named pipe in the guest and modify boot.ini to enable some special options. It's a little time consuming]. So in this tutorial, I will help you set up kernel mode debugger.

I will be using the following tools:

WinDbg (Windows Kernel Debugger)

Virtual Box (Virtual Machine Manager)

VirtualKD (Tool to enable very high speed kernel debugging between host and just machines)

Hereafter, whenever I refer to OS, it will be one version of Windows.

First thing we have to do is install virtual box and then install a guest OS of your choice. Here I would like to demo using Windows XP as my guest [installing a guest VM].

Second, extract the virtualKD to some folder.

Rename the VBoxDD.dll file in your VirtualBox program files folder to VBoxDD0.dll.Copy modified VBoxDD.dll from VirtualKD archive to VirtualBox directory (Ensure that you have selected correct version (x86 or x64) of VBoxDD.dll).

Third, install WinDbg (simple next next install).

For any debugger to properly work, we should have symbol files of the program being debugged, think of these files as extra information about your program which helps debugger in displaying meaningful information to the user. If we have correct symbol files, we will have the extract function names and the line numbers, etc. getting shown in the debugger, without that we will see some hard to understand Hex address. So the following few lines will help you configure symbol files for your operating system.