Sunday, June 2, 2013

Working with Unmanaged Callback Functions in PowerShell

Update (12/12/2013)
Get-ChildWindows was updated to work in PowerShell v2tl;dr version

With a little bit of work, you can bind a scriptblock to an unmanaged callback function in PowerShell. The key to accomplishing this is by casting a scriptblock as a non-generic delegate that has the function signature of the desired callback function. Fortunately, creating non-generic delegates is made easy with my Get-DelegateType function.

I’ve previously spoken in depth about the various ways to interact with the Win32/Native API in PowerShell. The ability to do this unlocks all kinds of potential for PowerShell. There was one subset of functions however, that to this day I was unable to interact with – those that have parameters with unmanaged callback functions. C# developers may cry foul of my assertion and state that you can easily do this in C# code and then just call the Add-Type cmdlet. Those developers would be right. I however, wanted to accomplish this in pure PowerShell without compiling a single line of code.

For those who may not be familiar with callback functions, they are basically a block of executable code that gets passed in as a parameter to a function, typically as a function pointer. When the function executes the callback function, it generally passes several arguments. It is up to the implementation of the callback function as to what gets done with those arguments.

For example, the EnumChildWindows user32 function accepts a WNDENUMPROC (EnumChildProc) parameter – a callback function that gets passed to it a window handle and an optional user-defined parameter. Here are the function definitions for both EnumChildWindows and EnumChildProc:

Now, function pointers are simple in C. That concept doesn’t easily translate to the world of PowerShell, however. The closest analog to a function pointer in .NET is a delegate which we can work with in PowerShell. In fact, it is not well known but you can actually cast a scriptblock as a delegate! The PowerShell engine implements a lot of plumbing under the hood in order to make this happen. Here’s a silly example of a scriptblock being cast and invoked as a generic Func delegate:

Now back to callback functions. After a little bit of experimentation, I discovered that you cannot bind a scriptblock to a generic delegate like we did in the last example when dealing with P/Invoke methods. You can, however bind them to non-generic types. In fact, that’s exactly what Get-DelegateType was designed to do! As an example, this is the code I used to replicate the EnumChildProc callback function:

The only thing left to do is to define the method signature for the EnumChildWindows function. The following table lists the equivalent types that we’ll use in PowerShell.

Now that we have everything we need to get started, I wrote a function that enumerates all child windows and returns their handle and title – Get-ChildWindows

You may have noticed my use of ugly global variables in the $Action scriptblock. They were necessary because the scriptblock executes in a bizarre state where it does not have the full functionality of a PowerShell session. Those familiar with registering .NET events with the Register-ObjectEvent cmdlet will know what this it like. Basically, a good portion of PowerShell’s functionality breaks when running in this environment – the pipeline doesn’t work and objects cannot be output. The only real form of user output available to you is via the Write-Host cmdlet. A way around this is to save all output to a global variable. Then, once the scriptblock is done executing you can interact with your output like normal. Hopefully, this issue will be fixed in a future version of PowerShell.

That’s it. Hopefully, this will be helpful to those working with the Win32 API in PowerShell and to people like me who refuse to compile C# in their scripts. ;D

I gotta admit, a lot of this is over my head. However, it seems like this could be used to have a PowerShell callback when Ctrl-C is pressed. I've long wanted to be able to call kernel32's SetConsoleCtrlHandler() with a PowerShell function as it's callback vs writing a handler in C#. Thanks for the article and inspiration!