Anticipation

Unmanaged callbacks across AppDomains

In one of the projects I’m working on I hit a fairly nasty problem involving (as might be obvious from the title) both AppDomains and unmanaged code calling back into managed code. But first, a little background.

It all started with an unmanaged C++ class library. We’ve been using it for a while from other unmanaged C++ applications, but since we’ve been writing most new applications in C#.NET, the time had come to bring the library (kicking and screaming) into the .NET world. Naturally, the way forward was to write a compatibility layer in C++.NET, thereby taking advantage of its IJW framework (which makes C++ the best .NET language to use to interface with native code). (.NET 2.0, of course. Don’t even get me started about some of the problems interfacing native & managed code in 1.x.)

One of the key ingredients was an unmanaged callback. Since some of the work of the unmanaged base library was asynchronous, we gave it an object to call back through in order to tell us when the results were ready — meaning that the unmanaged code needs to call into the managed code. This is normally simple enough — while unmanaged classes can’t hold on to managed objects directly, they can through a GCHandle (or the equivalent helper class, gcroot). The code went something like this (paraphrased to protect the guilty):

So, all well and good so far, right? No such luck. At first this was all going smoothly; the code was coming together and manual testing of the application showed that it was communicating properly with the library and with native applications that were also using the library. Once the proof of concept was in place it was time to ensure nothing went wrong later on through the use of unit testing. This is where the problems began, however.

The unit tests in question were written using the NUnit framework, which runs the tests in a separate AppDomain. This is useful because it permits the test runner to treat the application/library under test as a plugin — keeping it loaded only while actually running the tests, thereby allowing it to be recompiled and have the tests run again without having to exit the test runner.

The problem is that AppDomains are purely a managed construct. They’re intended to keep sections of managed code mostly isolated from each other (as in the above plugin-like case), and as such objects usually only exist in one AppDomain at a time. Unmanaged code of course knows nothing about any of this. Consequently, when calling managed code from unmanaged code, the compiler has to pick one AppDomain to use, and it appears to pick the first one. This is fine for most applications, since normally apps only use one AppDomain — which is why this code was working at first.

But when running the unit tests, it was executing in a second AppDomain, and the callback failed. Specifically, when trying to access the m_Managed object (from the gcroot, which you’ll recall is a GCHandle) an exception was thrown saying “Cannot pass a GCHandle across AppDomains”.

The solution is to use delegates. They’re not just function pointers — they also contain an object reference, a few other odds and ends, and (most importantly for us) a reference to the AppDomain that created it. However there’s a catch. Delegates themselves are managed objects, and so would have to be stored in a gcroot if held in unmanaged code — and we already know that we can’t access anything in a gcroot outside its original AppDomain.

Fortunately there’s a loophole: delegates can be marshalled into unmanaged function pointers (via a thunking layer). There is a downside to this though. Since the method signatures must match as closely as possible, and since a purely unmanaged function pointer can’t have anything to do with managed objects, the parameters must be restricted to native types. This required a bit of a redesign, but fortunately since you’re coming in from unmanaged code all the data you’re dealing with is going to be native anyway. So here’s the redesigned code. It’s possible there’s still something that could be improved in it; but this one does the trick, and maybe it’ll help someone else who has been struggling with this issue

30 comments to Unmanaged callbacks across AppDomains

Seriously, I’m having the exact same problem as you did and luckily I found your post. Unfortunately, our code is still in the old MCPP syntax, although I’ve recompiled it on VS 2005 with the oldSyntax switch. Do you think I could do something similar to the above with the old syntax?

I haven’t tried it, but as far as I know anything you can do with the new syntax you can also do in the old syntax — it’s just a lot more verbose (and has lots of double-underscores). As long as you’re using VS2005, anyway (and targetting 2.0). (The above method won’t work in 1.x, since it uses functions added in 2.0 — but hybrid 1.x DLLs don’t work all that well anyway.)

If you take my code above and backport the syntax (eg. replace ^s with *s, refs with __gcs, etc) then you should be close.

INativeCallback and NativeFramework are placeholders for the native classes you’re trying to interface to. In other words, that’s your external code, not part of .NET — so you’ve already got it

And no, there won’t be any issues with passing C++ classes through the callback, it’ll just work (bearing in mind what I said later). Though if you’re going to be interfacing with C# or VB code at some point then you should convert the data to managed classes along the way (after calling through the delegate; you can’t do it earlier).

You saved my day (again even when we are no longer working together), Gavin!
I was trying to use AppDomain to load a native dll which receives phone calls, as I want to have control of unloading and reloading the dll. I was stuck at the callbacks part where your article just helped me fixed it.

The gcroot isn’t the problem then — the problem is the native framework call and storing a reference to the unmanaged handler class (MyCallback).

It’s not totally impossible for ManagedClass to be implemented in C#, but it’s far more hassle than it’s worth, at least for the scenario I’ve specified (implementing a C++ abstract callback class), since MyCallback has to be implemented in C++ anyway.

Ok, but you can just pass the std::string (or your Rfa::UnManaged::RdmLogMessage) through the function pointer and delegate (so it winds up in LogDelegateAdapter::Log). You don’t need to get a char* and use IntPtr.

(The downside of doing that is that you then won’t be able to call LogDelegateAdapter::Log from fully managed code, but then usually you don’t need to anyway. And if you do, then you can just define another overload.)

I’m currently working on adding easy to use Real Time Data server support to my Managed XLL Excel Addin system. This lets you use the =RTD() functionality of Excel to push real time data into your spreadsheets without needing to……

[...] This link is about creating a callback from unmanaged code into an AppDomain using a thunking trick. I’m not sure this can help you but maybe you’ll find this useful to create some kind of a workaround. [...]

To Pass C++/CLI delegate as function pointer into native C++ class, you can make use of this .net method: System::Runtime::InteropServices::Marshal::GetFunctionPointerForDelegate, the key point is: define the delagate (MyDelegate^ m_myDelegate in my ex…

I want to thank you for publishing this article. I had exactly this problem and would not have solved without your explanations and your sample code. I have seen other articles, but this is the most illustrative. Many, thank you very much!

I’ve managed to work round the same problem by using the COM Callable Wrapper for the managed object works: instead of storing a gcroot<ManagedObject>, store the pointer as an IUnknown * using GetIUnknownForObject, and do the reverse translation with GetObjectForIUnknown before making the callback.

I presume that the COM Callable Wrapper must wrap up the AppDomain information, because COM knows nothing about AppDomains itself. Certainly it seems to work in practice.

The downside is losing a little bit of type safety because IUnknown * loses the actual object type and you later have to downcast, but that seems like a small price to pay for the greater simplicity.

Stephen: no, it doesn’t pollute the global namespace, it pollutes the MarketData namespace. But yes, in my own code I usually try to avoid doing that sort of thing even though it makes headers more wordy.

Hi,
I have same problem. But my base c++ api expect object of specific type (object implement a interface which has one method “void OnReceive(vector<shared_ptr> obj)” which is called on callback ) not function pointer and any change in base libraries is difficult now. Any other way to solve this issue.

Nishant: there’s no problem in transferring any native type through the interface (at least as long as you’re doing it within the same module). See the followup post linked at the end of this post for more information and some examples.

I have “Receiver” class in C# and this is passed as input parameter to one of the api. This receiver object is used for callback. In C++/CLI I have created a Native /unmanaged class “ObjectBinder” which is same replica (has same methods) of managed Receiver class. It holds reference of managed receiver object in gcroot. When we call that api from C# it comes to CLI layer and app domain is “client exe”. we store the parameter “managed receiver object” in ObjectBinder in gcroot and pass reference of native ObjectBinder object to C++. Now the backend code (c++ and c) send an asyn callback (new thread) to c++ layer which use ObjectBinder object to send back call to CLI. Now we are in CLI layer in ObjectBinder object. BUT App domain has been changed (in case of WCF or NUNIT or any other service that creates it’s own App domain which is not known at compile time) . Now i want to access managed Receiver object which is stored in gcroot to send back callback to C# but it gave “Cannot pass a GCHandle across AppDomains”.

I have also tried IntPtr and IUnknown * instead of gcroot with Marshal::GetIUnknownForObject and Marshal::GetObjectForIUnknown but getting same error.