Hooking a DirectX/COM Interface

After all the helpful articles I read here, I am glad that I can contribute to a subject that has not yet been covered.

This article features a description on how to hook a DirectX/COM Interface. I used the DirectInput interface as an example of how to hook an interface function. For the basic Windows hook, I referred to an article by Wade Brainerd that describes the API hooking process.

Task

Intercepting a method of a COM Interface requires an extended approach compared to hooking an API call. If the desired DLL is examined, only the create Interface function is actually exported by the DLL. So, how can you hook your desired function?

A COM interface is basically a list of virtual function pointers that are linked together. You merely have to follow the links and modify every node until you finally reach the pointer of the function that you want to replace.

Step 1

As you can see, only the create interface COM functions are visible, so you have to start your hooking chain at the DirectInputCreate function, which returns a COM interface. Here, you have to inject your DLL into the import address table (IAT) of the calling program.

Step 2

If the calling program invokes a DirectInputCreate, your function is called. Also, you receive a pointer to a pointer of a pointer to a virtual function table that is the interface of the direct input.

Step 3

Now, you can create your device with CreateDevice. You again will receive an address to a different virtual function pointer table; it represents the Device.

Pick the method you want to replace and change the virtual function pointer table in the appropriate place to inject your function.

Step 4

Do the actual data manipulation.

Implementation

Step 1

To hook yourself into an API function, you simply can use the Windows API call SetWindowsHookEx. Here, you create a system hook to monitor the starting processes and match them to your desired program. After you have identified your program, you have to compare the import module names with the DLL you want to replace. Because this hook is written for direct input, the entry you want is:

DINPUT8.DLL

To find this entry, you have to loop through the descriptors until you find your DLL.

// Replace the IAT function pointer if we have a hook.
if ( HookFn )
{
// Cheez-o hack to see if what we're importing is code or data.
// If it's code, we shouldn't be able to write to it
if ( IsBadWritePtr( (PVOID)pIteratingIAT->u1.Function, 1 ) )
{
pIteratingIAT->u1.Function = (DWORD)HookFn;
}
else if ( osvi.dwPlatformId == VER_PLATFORM_WIN32_WINDOWS )
{
// Special hack for Win9X, which builds stubs for imported
// functions in system DLLs (Loaded above 2GB). These stubs are
// writeable, so we have to explicitly check for this case
if ( pIteratingIAT->u1.Function > (DWORD)0x80000000 )
pIteratingIAT->u1.Function = (DWORD)HookFn;
}
}

The only thing remaining is to restore the memory attributes, as though nothing ever happened.

Hooking a DirectX/COM Interface

Step 2

Inside of the CreateInterface method, you start hooking into the COM interface by injecting your own CreateDevice function pointer into the Virtual function table (Vtbl), which is returned in the ppvOut pointer of the original call.

Resolve the pointer until you get the pointer to the Vtbl of the interface.

With this address, you again have to remove the memory protection before you can inject your function into the table and save the old function pointer for later use. Inject your function pointer into the offset of the CreateDevice function pointer inside of the interface Vtbl and restore the memory protection.

As you can see, the CreateDevice is the fourth method of the DirectInput interface; this means the offset inside of the Vtbl is 0x0C (pointer(DWORD)*index 3).

After you know where to inject it, you can start thinking about the implementation. When you look at the declaration of CreateDevice in dinput.h, it does not match up with the declaration you see in the DirectX Help.

This is important. You have to make sure to use the __stdcall calling convention in your declaration; refer to the MSDN.

The __stdcall calling convention is used to call Win32 API functions. The callee cleans the stack. __cdecl is the default calling convention for C and C++ programs. The stack has to be cleaned up by the caller, which is not the case with your function.

When you look at the disassembly of this call, you can see the stack pointer verification function _RTC_CheckEsp is called after the call to the interface function.

If you forget to declare your function with __stdcall, your function will process fine, but you will fail the esp pointer test of this function; it sets the eax and tests it after the function call.

Step 3

When you create the device now, the call is re-routed to your CreateDevice function. After you call the original function, you receive a new pointer in lplpDirectInputDevice, which will direct you to the Vtbl of the device.

In my sample, I replaced the GetDeviceState because I can add additional input in the return data of the calling function. To get to the offset, you have to look at the definition inside the DInput.dll. You see that the GetDeviceState function is the tenth method inside of the Device, which leads to an offset of 0x24.

After you know the offset, you can proceed with the instructions in Step 2 to remove the protection, store the old pointer, inject your own function, and restore the memory protection.

Step 4

When the GetDeviceState function is called by the target program, the injected function is called and you can manipulate the data as you wish.

About the Author

Martin Mueller

My first PC program was written on MS DOS in Assembler, after I got Peter Norton's Assembler book where the Edlin and DOS Debugger supported me in exploring my and code of others. I went from there over IDE/Debugger like Borland Turbo Debugger over several microcontroller H8, ST16, 8051, SLE66, SLE88 to my favorite MS Visual Studio 6. Now I mainly program in C++ and Java, where I prefer C++ since your abilities to interfere with the system are an easier task, which is still my favorite, than implementing some natives in Java to enhance the functionality. I am working since 18 years as a programmer and still love every day of it. My experience include ever level of programming from Hardware programming, where you have to take care of nano seconds
in your assembler program over real time operating system programming, to VM programming, system services, applications, game programming, web server to high level web script programming.

Top White Papers and Webcasts

Live Event Date: March 19, 2015 @ 1:00 p.m. ET / 10:00 a.m. PT
The 2015 Enterprise Mobile Application Survey asked 250 mobility professionals what their biggest mobile challenges are, how many employees they are equipping with mobile apps, and their methods for driving value with mobility.
Join Dan Woods, Editor and CTO of CITO Research, and Alan Murray, SVP of Products at Apperian, as they break down the results of this survey and discuss how enterprises are using mobile application management and private …

The mobile market is white hot. Building a well-crafted product development plan that addresses market research, strategy, design, and development will provide the greatest chance for success. Each phase of an app's lifecycle is critical to the its overall success and feeds into the next step of the process of product development for the app. This white paper examines the five key phases of mobile app creation, one by one, to understand how they work together to help create a successful mobile app.