Stats

Restricting DLL loading

How to restrict a shell extension from being loaded during the development process.

Background

During the development of a shell extension, you have little control over when your extension can and cannot be loaded. This becomes rather annoying as you try to rebuild and are prevented from doing so because the file is in use. There does exist a set of registry keys to help with this problem, but they don't exactly make the process developer friendly.

The solution I'll present is simple and effective. I've been using it for some time and thought I'd share it with the community.

The Solution

The shell loads all extensions by calling the KERNEL32.DLL function, LoadLibrary, with the path of the DLL as registered in the registry. LoadLibrary goes through the process of loading the DLL into the address space of the calling process, typically explorer.exe with shell extensions, resolving other library imports, and finally calling the DLL's entry point function, DllMain. DllMain performs any initialization and returns a BOOL value of TRUE or FALSE to indicate success or failure. If FALSE is returned, the system immediately detaches the DLL from the process, unloads and closes it, and returns NULL to the caller of LoadLibrary to indicate that the DLL was not loaded successfully. So with this knowledge, we have a sure fire way of preventing a DLL from being loaded by a process. The remaining trick is to specify exactly which processes, or more specifically, which modules can or cannot load the DLL. Enter IsDebuggerPresent and GetModuleHandle.

Being that you are the one developing the DLL, you have a good idea of exactly which modules you do or do not want loading your DLL. For one, it is a fairly safe assumption that if the calling process is being debugged, then you want your DLL to be loaded. But there are also some processes that need to load the DLL, such as REGSVR32.EXE, which typically do not run in the context of a debugger. To support both cases, we have two functions available from KERNEL32.DLL. IsDebuggerPresent, which returns TRUE when the calling process is being debugged and FALSE otherwise. And GetModuleHandle which returns the HMODULE of the specified module if it is loaded in the process, and NULL otherwise. So with these two functions, we create a rule that states:

If the process is being debugged, or if the process contains a known module, allow DllMain to succeed and return TRUE, otherwise fail DllMain and return FALSE.

The Code

Part 1 - Known Modules

We first need to create a list of modules that are permitted to load the DLL in all contexts. This will include applications such as REGSVR32.EXE so that the DLL can use its registration features, other applications which might need to load the DLL for things such as resource acquisition, or other modules which utilize the DLL as part of a package or feature, such as DLLHOST.EXE or SVCHOST.EXE.

Next, we want a compiler flag to disable this feature for release builds or for when we do not want to restrict the DLL from loading.

/*********************************************************
* When the _LOADRESTRICTIONS_ON flag is defined, DllMain
* succeeds for the following conditions:
* 1) The process is being debugged.
* 2) One of the modules listed in _szUnrestrictedModules
* is loaded in the process.
*
* This prevents apps like the shell from loading and
* locking the module, preventing new builds. The
* _szUnrestrictedModules is needed so that non-debug apps
* like regsvr32 can load the module.
*
* Defining _LOADRESTRICTIONS_OFF will disable this
* functionality.
*/#ifndef _LOADRESTRICTIONS_OFF
#define _LOADRESTRICTIONS_ON
#endif

And there we have it. Adding this code to a shell extension DLL prevents it from being loaded except for those conditions that we specify. This allows rebuilds without waiting for the system to unload the DLL.

That's why when I debug my shell extensions I set "Launch folder windows in a separate process", which is also recommended practice.

This is a good article, though. Even following the recommended practice when developing shell extensions, certain events may cause the shell to load your newly registered shell extension. This is also good for other situations as well.

You really didn't have to log off, did you?
If you just kill your explorer.exe from task manager, and then re-start it, you'll be okay. Also, if you've set up your explorer options such that each explorer window starts its own process, all you need to do then is to kill all your explorer windows.

Not unless you write your own shim, no. By default when you register a .NET assembly to expose it to COM as a CCW, regasm.exe registers mscoree.dll as the managed host (a shim) which loads your assembly, the path or name (good when the assembly is in the GAC) of which (depending on whether you supplied the /codebase switch) is stored under the same registry key but is not the path to the COM server (since it's not a COM server).

If you write your own shim to take the place of mscoree.dll (and you'll need to change this default behavior either manually or use the ComRegisterFunctionAttribute and ComUnregisterFunctionAttribute to add and remove (in this case, you're changing) registry keys when regasm.exe is run against your assembly.