Secrets of the Application Compatilibity Database (SDB) – Part 3

Continuing over from where we left last time, today’s entry will look at how the loader interacts with the AppCompat/Shim Engine Interfaces to determine that a module requires shimming or not. Unfortunately, it seems like this process underwent several revisions inside Microsoft’s codebase, so it may be difficult to experiment on your own based on this information. I will however, present all the known implementations to me in a generic fashion, without going too much under the hood in terms of actual assembly code.

Like many Win32-specific features, the Shim Engine actually gets initialized by the parent process through kernel32.dll, and not by the actual PE Loader/Startup routines inside the NT System DLL (ntdll.dll), although it also plays an important role in the process. As CreateProcessInternalW executes, it eventually calls BasepCheckBadApp (which is actually an exported API). The first thing that immediately happens is a check on whether or not the Shim Engine is disabled, followed by a lookup inside the Application Compatibility Shim Cache (done through BaseCheckAppcompatCache).

This cache is implemented in 2 different ways depending on the OS. On pre-Windows 2003, kernel32 maintains a shared section which other instances can use for caching the information, and a lock/unlock is done each time the cache is accessed. On post 5.2 kernels, there is a new Native API, NtApphelpCacheControl which supports the following classes:

In both cases, if the cache lookup doesn’t find anything, a “long” lookup is performed. This is where the architectural differences are the largest. In Windows XP SP0, this is done by using CSRSS, and calling BaseSrvCheckApplicationCompatibility in basesrv.dll. In SP2, apphelp.dll is imported and ApphelpCheckExe is called directly. In Windows Server 2003, a connection to the LPC Port AELPort is made, and a lookup LPC message is sent. Finally, in Vista, we’re back to SP2’s method, albeit with a newer API, ApphelpCheckExeEx.

The end result, however, is that the Peb’s pShimData member is now filled out with Apphelp Information (we’ll see what happens with this later) if this is a “bad” application indeed, meaning that it needs to be shimmed. How are these checks actually made, however? Recall that one of the “constructs” or entries that an SDB can have is the Matching File entry. The checks first discover the “Executable” entry for the filename given, and if one is found, all Matching File entries are parsed. This can include the name of the application and its helper files, the publisher, vendor, company, version, file size, timestamp and even linker version and other obscure data. Several boolean operations are available which can be built on top of inclusion and exclusion rules. If the Matching File entries check out, then the pShimData is filled with an opaque Apphelp structure.

The next important part of the Shim Engine’s mingling with our application happens in LdrpInitializeProcess, which is part of the PE Loader inside ntdll.dll. Here, a check is made if pShimData is non-NULL. If this is the case, then this pointer is saved then cleared, and the Shim Engine DLL is loaded with a call to LdrpLoadShimEngine. A variety of callbacks are then setup through LdrpGetShimEngineInterface, which mostly consist of pre and post initialization, and DLL load/unload notifications.

A bit later during initialization, the pre-initialization hook is called, if the Shim Engine was previously loaded, and the old pShimData pointer is passed along to the Shim Engine, so that it may begin initialization. The routine is SE_InstallBeforeInit inside shimeng.dll, and most of the work is done by SeiGetShimData and SeiInit. The former unpacks the information from the PEB Shim Data pointer that it received. It also has a check to disable shimming for ntsd and windbg, as well as slsvc, on Vista (since this is a semi-protected process related to licensing). As for the latter, it will process all the compatibility layers, shims, flags and finally patches which are defined in the Apphelp entry for the executable.

Shims will usually consist of either internal flags that are saved inside shimeng.dll or inside the PEB (seeÂ +0x1d8 AppCompatFlags and +0x1e0 AppCompatFlagsUser), or by the IAT of the shimmed process to be hooked and redirected into one of the Ac***.dll files which contain an alternate, or hacked implementation. These contain two main exports, GetHookAPIs and NotifyShims which allow the Shim Engine to know which APIs should be hooked and to send notifications during loader events. The Shim Engine is smart and will also hook GetProcAddress to make sure that APIs are properly caught. Patches are done through a method that will be looked into more detail later.

During the next entry, we’ll take a look at an actual shimmed application in action, and future parts will cover patches/flags in more detail. It is my hope that this part was useful into giving some insight on how the hooking is performed. Many vendors of application/DLL hooking software risk of running into the Shim Engine during their testing and development process, so having a good handle on how and when everything happens is certainly helpful.