A Whirlwind Tour through the Managed Extensibility Framework

From a hotel lobby in the sunny city of Durban, South-Africa, waiting for my plane transfer after a great TechEd Africa event. Why not write a blog post on one of my talks: the Managed Extensibility Framework, or MEF. As we steam ahead to release .NET 4.0, it’s great to see the amount of new APIs that will make it into this release. I had the opportunity to talk on three of those the past few days:

The Dynamic Language Runtime (System.Dynamic) with dynamic support in C# and VB, and dynamic languages like IronPython and IronRuby.

Why does extensibility matter?

The world’s most successful APIs succeed in taking away developer frustration. Think of web services (“Add Service Reference”, anyone), LINQ (thanks to its unified query experience across domains) and many more. With extensibility, that’s no different:

That extensibility is a big deal for customers is no secret thing. Look at the success of add-ins within popular software packets: browsers, office productivity suites, developer tools, and even the tool I’m writing this post in: Windows Live Writer. Having a brand new installation of Windows 7, I haven’t installed something I’ll need further on in this post: the ability to paste code from Visual Studio in my post. Why didn’t the Windows Live Writer team think about that capability?

Honestly, that’s not a silly question to ask at all. But tell me, if 99% of users (okay, I’m magnifying things out a little bit) with a blog tend to write about how great the spaghetti was they consumed last night in one of their favorite bars, making pasting code from a tool as specific as Visual Studio doesn’t seem like it meets the bar of top feature requests right?

What if the Live Writer team would say: let’s not freaking care about such a feature ourselves but provide the ability for others to extend our product?

So what about ALT-PrntScrn, CTRL-V to illustrate my point?

Okay, I visited Windows 7’s new Paint utility for a short while to draw a red rectangle, but the above shows how the ability to paste code from Visual Studio is painfully missing but how the “Add a plug-in…” capability is the magic bottle of snake-oil waiting to be consumed. Clicking the link, I arrive on a web page with a bunch of extensions waiting for me to be added to my favorite blogging tool:

I find one plug-in that sounds promising, click Download, accept a few prompts and the installation is on its way. Seconds later, even without restarting Windows Live writer, I take another screenshot:

If you’re a true developer, you’ll have to admit being jealous on the Live Writer team. Don’t you want your application to be that pluggable as well? Sure you do! That’s why extensibility matters: customers love it, but it’s way too hard for developers to do and get right today. Guess what, that’s where MEF comes in…

Our running sample

Boring as I am, my demos always go the minimalistic route: no UI candy, no contrived samples, plain basics using the purest black gold in the Windows and .NET universe: console applications. I bet my statistics of application development in Visual Studio by project type exceeds the 100:1 ratio for console applications versus anything else. So live with it or navigate away (please don’t).

Because I release “Hello World” would be too boring of a sample, let’s go with an extensible console-driven calculator. Be prepared to be shocked as I show the stunning result:

Little do you know that Add is a built-in operation, while the Pythagorean option was loaded from an extension provided by someone else that the calculator “core” developer. And what about MEF? It’s the secrete sauce that put the pieces of the puzzle together.

Introducing FunCalc’s project structure

But before going into MEF terrain, let’s have a look at how we can make extensibility work without using MEF. Our goal? To show how much plumbing is involved in order to get such a seemingly simple thing off the ground. In order to be a bit architecturally hygienic, we’ll divide our application into a few components:

Our superb million dollar business UI, a console application project.

Extensibility interface(s), as a class library project.

Third party extensions, again as a class library project.

So we’ll go ahead and create a solution with three projects and the necessary cross-references between them: both the UI and the extension projects need to refer to the extensibility interface project to access the contract that’s established between the “host” and the “extenders”. Here’s what the interface will look like:

How did I insert the code snippet above? Right, using the plug-in I installed just a few minutes before. Life is great and I’ll be delighted to give this level of flexibility to the users of my superior quality calculator as well.

What does the interface look like? Honestly, it’s already overkill as I seem to have cared a tiny little bit about presentation. Shame on me: the Symbol and Name get-only properties are all about visualization. The former one represents the infix operator symbol while the latter one is used to add a menu option (‘+’ for an “Add” operation, ‘#’ for a “Pythagorean” operation – don’t ask me why I chose that symbol but I reserve the “symbolic mathematical right” to choose whatever symbol I see fit :-)).

Besides the UI candy properties (which more UI-savvy developers would bloat with whole control-exposing properties, which would act as the labels on a button in the calculator UI) there’s some real hardcore stuff in the interface as well: Calc is the binary operator that takes two integer numbers and produces another one back. That’s where the operator’s implementation will live.

But first, the solution’s project structure. Don’t worry about the code files just yet but rather focus on the references between projects:

I’ve also cleaned up the references to assemblies I don’t need (like System.Data) to make the structure as clean as possible for sake of the demonstration. The startup project, FunCalc, is the Console Application project. It references the Interfaces class library assembly where the aforementioned interface ICalculatorOperation is defined. You can already go ahead and do that. Similarly, the Extensions assembly (which you should really think of as being created by a developer on the other side of the world in a separate solution altogether) refers to the Interfaces assembly. I’m keeping everything in one solution again for the sake of demonstration but you should envision extensions coming from tons of places just as raw DLL assembly files.

One final thing left to do in this setting is to provide a way for the extensions to become discoverable to the host, FunCalc.exe, application. In other words, the Extensions.dll file needs to be placed in a well-known location where our application will look for extensions. Again, in a real-world setting this would be done by providing a well-known folder location (maybe registered somewhere in the registry) that plug-in installers would copy their extension DLL file(s) to. All of that would distract us from our core mission here, so let’s use the tools at hand. Using a post-build event command on the Extensions project, I’ll copy the produced assembly to an AddIns subfolder underneath the build location for the FunCalc executable:

Don’t worry too much about this little bit of magic but a few give-aways: TargetPath contains the full path to the compiled file (Extensions.dll), while SolutionDir and SolutionName contain the full path to the solution root and the name of the solution (FunCalc, which happens to be the same as the Console Application’s project name, the only bit of magic we’re relying on here) respectively. I’m also hardcoding the bin\Debug folder, which could be pimped up a little bit if you were to do this for real. But the crux of it is that we’re copying the extensions DLL file to the AddIns folder next to the FunCalc.exe binary. Give it a try and have a look in Explorer:

Sweet. We’re ready to go.

Designing the “UI”

Time to design and build the host application’s UI. In Program.cs, add the following code to the Main method for starters:

We don’t have the Calculator class just yet, but be patient for a little while. What’s the main loop do? It calls AskUserInput to show the menu with options and return the calculator operation the user chose. If the result is null, the user entered the exit option (0). To retrieve the operands to the selected (always to be binary/dyadic in our sample) operator, a call a helper function called ReadInt twice. The first argument will be the prompt, the second one a validator function (Func<int, bool>) to have ReadInt repeat its question till the validator passes (e.g. for range checks, see further). Finally we invoke the calculator’s selected operation by means of calling Calc and print the result with some fancy formatting resulting in infix notation using the operator’s Symbol property getter.

As trivial as it can get. Some printing of a one-based menu based on a yet-to-be-defined Operations property on the Calculator object (see further), followed by another use of ReadInt to ask the user for a menu option, this time passing in a validator function using a concise lambda expression making only the range of menu options (including 0 for exit) valid. If the choice was 0, we return null to indicate we should exit, otherwise we select the operation of the user’s choice using an indexer operation.

Simple once more. Keep on asking for user input given the specified prompt till the input is a valid integer. That’s all it takes to create the UI and you didn’t have to touch the mouse once. How good does life get?

The Calculator type

Finally our host needs its Calculator type which will hold on to all the supported operations. You can envision it to be much more complex, potentially providing UI candy and additional functionality (like support for different operator arities, a graphical calculator unit or unit conversion modes – have a look at the Windows 7 calculator for inspiration). But here is our minimalistic Calculator type:

At this point, we can try to run the application. It will fail in grandeur:

Why? Because we didn’t compose operators with the Operations list. It’s still null and has never been assigned. A capital mistake but it turns out this is the place where MEF will come in: gluing things together.

But before we embrace MEF, let’s try to wire up things manually “the naive way”. And even before we do so, let’s make our calculator sellable by having built-in operations for Add, Subtract, Multiply and Divide.

Built-in operations

Creating built-in operations is no rocket science: simply implement the interface. The built-in nature reveals where we’re going to define those operations: right inside the FunCalc console application project. Here’s a sample implementation:

Consult your books on higher-order mathematics to implement the extremely difficult Subtract, Multiply and Divide operations yourself. While you do so, I’ll go ahead and wire up those in the Main method as follows:

Extensibility the naive way

While we already have a LoadBuiltIns method, we also want to be able to load extensions dynamically at runtime by looking in our AddIns folder and discovering types in assemblies that are suitable for use by our Calculator type. The way we do so is by scanning the whole directory for .dll files (a first mistake, since a .dll is not necessarily a .NET assembly), loading them (a mistake if we do it naively as we shall see), looking for types that implement the interface (could be a mistake too, because we may pick up types that accidentally implemented the interface but weren’t meant – explicitly that is – for composition), and create instances of those types by invoking the default constructor (it gets tedious, but this isn’t ideal too). Here’s what I came up with as the most naive very error-prone way of doing extensibility the .NET 1.0 way. In fact, let’s make it the .NET 3.5 way and be super-naive writing everything as a LINQ query:

What’s naive about this? Quite a few things. This thing is largely side-effecting with lots of opportunities to cause exceptions: loading can fail, instantiating can fail too. Another big problem lies in the use of LoadFrom instead of reflection-only load: we’ll load assemblies even if they don’t contain extensions. Don’t worry about this, as all of this ugly code will go away further on when we loop in MEF.

Obviously we also need to patch up the assignment to the Calculator object’s Operations property to include the extension objects:

Writing our first add-in

But does the magic above work? Let’s try. Add the following class definition to your Extensions project which has been set up to copy the resulting assembly to the AddIns folder before. That should be enough to pick it up right?

Obviously you’ll have to import the Interfaces namespace where the interface lives, as referenced through the project reference to the Interfaces project. Build and run and lo and behold:

If it didn’t work, check a few things:

Did you include the LoadExtensions result in the assignment to Operations?

Was the rebuilt Extensions.dll copied correctly to the AddIns folder?

Is your Pythagorean operation class declared as public?

Reflecting on our solution

How do we like our current solution? Positive thing: it works. Negative thing: the implementation sucks. Why?

Discovering extensions was a pain.

Instantiation extensions was tedious.

Wiring up the list of extensions to the Calculator is tedious too.

So, if we boil down things to the bare essence we end up with a few things we want to make easier to do:

Contracts – The interface between various parties is something we can’t eliminate. Notice how general this concept is: things like WCF are contract-based too (address, binding, contract).

Catalogs – What about making discoverability generic by means of a catalog concept? Such a catalog could be based on a directory search, amongst other sources.

Composition – Why can’t wiring up parts of the application be as easy as throwing everything together and let the runtime figure out what fits where (e.g. operations should be added to the Operations collection).

That’s the essence of MEF: its three C’s:

MEF = playing with blocks

To make those concepts a bit easier to grasp, imagine yourself playing with Lego blocks again (or admit you spend all your free time doing so nevertheless):

Blocks fit onto one another in a very particular way: there’s a contract between different blocks that determines whether they fit. Next, there’s a catalog where we can find all the blocks we can play with. It’s about discovering what parts exist. Finally, in an ideal world, we’d be able to buy a magic box, throw the parts in there, and out comes a giant structure where everything is fit together automagically:

That magic box is MEF. It’s a composition engine. No wonder the MEF logo looks as follows:

MEF essentials

MEF is about give and take. The equation is plain easy:

Import is what our Calculator will do: it needs operations to play with. The operations on the other hand export themselves by saying: I have an operation available for you. MEF wires up the parts in order to reach a composed application. In a practical scenario the picture looks as follows:

The way to express the import/export relationship is done in a purely declarative way using custom attributes: ImportAttribute (with an ImportManyAttribute variant) and ExportAttribute.

Putting MEF into action

Time to simplify our application using MEF. First of all, let’s get rid of all the code in LoadBuiltIns and LoadExtensions. More specifically, we want to get rid of this line of code:

calc.Operations = LoadBuiltIns().Concat(LoadExtensions()).ToList();

This is where we’ll put some MEF code in order to request composition to happen. But first, we’re going to use the MEF attributes to denote what’s importing functionality and what’s exporting functionality. Let’s start with export. In order to make this happen we need to import System.ComponentModel.Composition, the assembly where MEF lives. We do so in both the host application (FunCalc) and extensions (Extensions) projects:

Next, go to all of your operation types (Add, Sub, Mul, Div and Pyth) to put the Export attribute on their definitions as follows:

Notice you’ll have to import the System.ComponentModel.Composition namespace. The core essence of the ExportAttribute is to explicitly state to MEF that the type exports a certain contract (which can be based on an interface type):

We do this for all of our ICalculatorOperation types, including the built-in ones. This allows us to have MEF take care of the composition for those built-in ones as well, taking the burden of doing the wiring-up ourselves away. On the other side of the equation we find the Import attribute where the exported parts will be accepted for composition:

We’re actually using ImportMany instead of Import in order to tell MEF that we’re doing a Many-to-One mapping here: our Operations property will contain a List containing all of the exported ICalculatorOperation parts it can find. This will ultimately include built-ins as well as extensions.

Now that we have attributed all of our implementers and the consumer, MEF is capable of wiring up everything for us as long as we instruct it to do so. Before getting into the domain of discovering parts, let’s do a manual composition first. The code to do this looks like this:

var container = new CompositionContainer();
container.ComposeParts(
calc,
new Add(), new Sub(), new Mul(), new Div()
);

Make sure to import the System.ComponentModel.Composition and System.ComponentModel.Composition.Hosting namespaces in Program.cs for the code above to compile:

using System.ComponentModel.Composition;
using System.ComponentModel.Composition.Hosting;

When we run the application, sure enough our calculator again exposes the included operations through its UI. MEF went ahead and wired up the objects for us, something you could see in action when setting a breakpoint on the Operations property setter on Calculator:

Notice I’ve also changed the property to have a private setter. This looks far cleaner from an encapsulation point of view and MEF is perfectly capable of initializing through private setters (and even directly on fields) because of its reflection-based nature (and its intentional support for this type of scenario).

Though we haven’t seen how to discover parts yet, the power of MEF’s composition engine has become apparent in the sample above. Time to move on to the discoverability aspect next: catalogs.

Catalogs in MEF

The concept of a catalog is what drives discoverability of parts in MEF. Various types of catalogs are built-in to MEF but one can create his own catalog easily by deriving from the ComposablePartCatalog class:

To get started with catalogs, we’ll use the TypeCatalog to abstract away from our manual hook up of the built-in operation instances:

How’s that any better I hear the reader wonder. If we still have to specify all the types, what are we saving here? In fact, using the TypeCatalog we’re leaving the burden of instantiating part objects to the MEF composition engine. More advanced features allow controlling the constructor invocation as well, so in essence we’re yielding more control to MEF here. But we can do better: what if we add a built-in operation to the calculator? Now we need to update the code above to include the implementing type to the catalog manually. Can’t we discover all types in the current assembly? It turns out we can using the AssemblyCatalog:

Here we’re instructing MEF to search the whole current assembly, i.e. the one where the calculator’s Program type is defined in, for attributed types. For the types it found, it will take over the plumbing associated with instantiating the objects and obviously the rest of the composition as it did before.

Finally, we want to add the extensions to the mix. To do so, we need two more things. First of all, we need a DirectoryCatalog for MEF to search an entire directory for assemblies containing types that can be used for composition. This is the superior equivalent of our LoadExtensions iterator we played with before. Since we now are faced with two catalogs: one AssemblyCatalog for the built-ins and one DirectoryCatalog for the extensions, we need a way to compose those catalogs. That’s what the AggregateCatalog is for:

And, as you can probably imagine, taking this one step further and mixing in other directories or other types of catalogs is trivial to do using the AggregateCatalog: it simply acts as the union of catalogs.

The result

We replaced a significant amount of plumbing code with five significant lines of MEF code and a few attributes here and there. For people extending our application, things are as easy as implementing the contract interface and sticking an attribute on the resulting type to opt-in explicitly to MEF composition. For us, the application creators, code got significantly easier to create an extensible application. Obviously, the resulting application still runs flawlessly:

In its true essence, MEF is a way to play with blocks or to brew soup (in the latter interpretation, one could argue it’s the Maizena plus in the picture below). We haven’t explore the concept of a CompositionBatch as mentioned below, so the figure below acts as a little stimulus for further exploration on your own :-).