I'm using a local hook (WH_KEYBOARD) with ms word (OpusApp). Well, as far as I know a 32bit app with a 32bit DLL must work only with 32bit target applications. The weird thing is that the program only works with 64bits apps!!! That is it, only with 64bits APPS! For example, it works with IE 64 but not with IE 32!
The app and dll are 32bit compiled with radstudio XE2, I confirmed the version into PE header.
In 32bit OSs, the app and dll doesn´t work.

I found no solutions on net and see no starting point to solve this weird problem.

Clearly the operating system is offended that you wrote a MouseProc() for a keyboard hook and decided to invert everything.
–
Hans PassantSep 3 '12 at 19:14

Well, I'm a bizarre person too! I will try apologize OS, but I think it will not solve the problem.
–
sgmSep 3 '12 at 20:42

You need to show more code. Where is your call to SetWindowsHookEx()? Is it inside the app or inside the DLL? What parameters are you passing to it?
–
Remy LebeauSep 20 '12 at 22:51

1

Your code has some bugs in it. You need to store the HHOOK handle in a block of shared memory. You can use CreateFileMapping() and MapViewOfFile() for that. But more importantly, your callIt() callback in the app is not using the __stdcall calling convention but the DLL is expecting it to. You would have caught that error if your code were type-safe. The function parameter of InstallMouseHook() needs to declared as CALLIT instead of void*.
–
Remy LebeauSep 23 '12 at 19:52

1

Another problem is that you are installing a callback function pointer that belongs to one process but might be executed by the process being hooked. A better solution is to use an HWND instead of a function pointer and have the hook send a message to the window instead of calling the function pointer directly. That would be much safer.
–
Remy LebeauSep 23 '12 at 19:57

1 Answer
1

It is physically impossible for a 32-bit app to install a 32-bit hook DLL and have it executed in 64-bit processes. A 32-bit DLL simply cannot be injected into a 64-bit process. Period. MSDN says this in multiple places, including in the SetWindowsHookEx() documentation:

SetWindowsHookEx can be used to inject a DLL into another process. A
32-bit DLL cannot be injected into a 64-bit process, and a 64-bit DLL
cannot be injected into a 32-bit process. If an application requires
the use of hooks in other processes, it is required that a 32-bit
application call SetWindowsHookEx to inject a 32-bit DLL into 32-bit
processes, and a 64-bit application call SetWindowsHookEx to inject a
64-bit DLL into 64-bit processes. The 32-bit and 64-bit DLLs must have
different names.

Because hooks run in the context of an application, they must match
the "bitness" of the application. If a 32-bit application installs a
global hook on 64-bit Windows, the 32-bit hook is injected into each
32-bit process (the usual security boundaries apply). In a 64-bit
process, the threads are still marked as "hooked." However, because a
32-bit application must run the hook code, the system executes the
hook in the hooking app's context; specifically, on the thread that
called SetWindowsHookEx. This means that the hooking application must
continue to pump messages or it might block the normal functioning of
the 64-bit processes.

If a 64-bit application installs a global hook on 64-bit Windows, the
64-bit hook is injected into each 64-bit process, while all 32-bit
processes use a callback to the hooking application.

The fact that you say your app and DLL do not work on 32-bit OS versions suggests your hooking code is flawed to begin with. But you have not shown enough code to diagnose that one way or the other.

What could I say, the above code (compiled for 32 bits with rad studio) are working only with IE64, not with IE32 (same with others apps). The function SetWindowsHookEx is inside the DLL.
–
sgmSep 22 '12 at 23:43

There is absolutely no possible way that IE64 can be executing your 32-bit hook DLL. It is physically impossible. Only 32-bit processes can invoke 32-bit hooks. Period.
–
Remy LebeauSep 23 '12 at 19:58

Thanks Remy for the answer. callIt uses __stdcall in typedef void (__stdcall *CALLIT)(int,WPARAM,LPARAM); I understand that is not possible to a 32bit process, with a 32bit dll hook a 64bit application. But, unless radstudio are building with 64bit (in my version there are only option to build for 32bit windows) or Internet Explorer (64-bit) shortcut are point to the 32bit version, this bizarre thing are hapenning. See desktop images link In fact, call a function pointer in another process is bizarre ... but this works as a first try...
–
sgmSep 24 '12 at 21:16

In the EXE code, you declared the callIt() callback as CALLIT callIt(code,wParam,lParam), which is completely wrong. It needs to be declared as void __stdcall callIt(int code, WPARAM wParam, LPARAM lParam) instead. Again, your compiler would have caught that had you written type-safe code to begin with.
–
Remy LebeauSep 24 '12 at 21:38