Introduction

There are already quite a few articles about dynamic expression evaluation using .NET. What I want to show here is a new twist on the idea by using the DynamicMethod class in the System.Reflection.Emit namespace. In my research I've found two main ways to tackle runtime expression evaluation: compile .NET code using the System.CodeDom namespace or write a parser. There are pros and cons with each method but I knew I didn't want to have to write a parser since .NET can already parse C# and will do a much better job at it than I can.

The good thing about compiled code is it's fast. However, there are several problems with it. It seems to take quite a bit of time to compile compared to parsing. So if the expression is constantly changing then it won't be a good performer. But if the expression is something to be compiled once and called many times then its performance can't be beat. So this is the problem I'm trying to solve; given a set of expressions I need to execute them quickly for a given set of values or row of data. Then be able to do the same for a data set by repeating for each row in the set.

It would be a good idea to look at some of these to get an idea of how this article differs or on what concepts it's based.

So a problem with CodeDom is that compiling causes the resultant assembly to be loaded into the current AppDomain. Since in .NET you can't unload an assembly you would end up with a lot of assemblies since a new one is created for each new expression. To keep the generated assembly from being loaded right away another AppDomain can be created where code the is generated. But when we want to execute the expression the assembly still has to be loaded. So what can we do to get around this problem? Enter System.Reflection.Emit.DynamicMethod class to save the day.

By now I'm sure you just looked in MSDN to find out more about it. DynamicMethod allows a method to be created at runtime and have the speed of compiled code. This is great but just one problem: you need to write the IL code for it.

I knew I didn't want to write a parser and I sure didn't want to spend a lot of time with writing IL code. The bright idea comes from the Turn MethodInfo to DynamicMethod article linked above. Basically reflection can be used to get the IL code of a compiled method and generate a DynamicMethod from it. Perfect! Now we can write .NET code, compile it, get the IL code and create the DynamicMethod we need.

Using the code

To turn a MethodInfo into a DynamicMethod I needed to know what method was being used. I know I wanted access to functions in the same class as the method, so I didn't need to prefix with a class name. I wanted to provide a set of documented functions to the user without them having to know a lot of .NET. So I created an object with some functions on it and called it FunctionClass.

CodeDom Compiler

So to compile a C# expression that will have access to the methods on this class I'm going to make the compiled method on a class that inherits from FunctionClass. This way it will have access to all protected and public methods from the base class and call them without any type prefix. The assumption I've made is that an expression is one line of code so it looks like return [user code line];. To keep it type safe I'm also going to require that the user pick a return type for the expression; double, string, DateTime, etc. So below is the code for the compiler class.

The BaseExpressionCompiler provides the GetMethodState method that converts a MethodInfo into a DynamicMethodState. DynamicMethodState is the class that provides a serializable form of a method's IL and metadata token offsets. Once I have that I can throw away the assembly that was created.

The compiler class inherits from MarshalByRefObject because it needs to be used across an AppDomain as explained earlier. Now that there is a compiler to get a DynamicMethodState we then need to create the DynamicMethod object from it.

DynamicMethod Delegate

The ExpressionDelegateFactory calls to the compiler across an AppDomain it creates to compile an expression. It then creates a delegate from the DynamicMethodState returned by the compiler in the form of R ExecuteExpression<R,C>(C functionClass) where R is the return type and C is the function class type.

The 'magic' for DynamicMethod is that it behaves like a static method except it can also be used like an instance method. In IL code the first argument in an instance method is always the instance of the type to which the method belongs. So to trick this static method to behave like an instance, you make sure the first argument of the method is an instance of the type to which the method belongs. Read about this in MSDN because its the important concept that makes this code work.

I also wanted to hide the implementation of creating a DynamicMethod from the code using the expression so I created an ExpressionEvaluator that wraps the calls to the ExpressionDelegateFactory. The code is separated into 5 different projects: MsilConversion (IL reading and token resolving), MethodState (shared dll between app domains, DynamicMethodState), ExpressionCompiler (langauge specific compilers), DynamicMethodGenerator (DynamicMethod delegate factory), ExpressionEvaluation (wrapper around delegate factory).

I created a Windows Forms test application aptly named TestApp to call the ExpressionEvaluation engine. It has a simple one-time Evaluate and then a Loop function that shows the performance that a 'compile once, run many times' operation can achieve.

Possibilities

There appear to be quite a few possibilities from here for implementation enhancements or changes. Different .NET languages, a language translator from custom language to .NET language, supporting multiple code lines instead of a one line expression. Maybe being even more ambitious and creating a parser/IL generator to skip the CodeDom step and write IL directly.

Feedback

I would appreciate any feedback you can give me on the code, concept, or the article itself. Also, I'm curious about your ideas for enhancements and if you implement this concept what was the result.

History

3/13/2007 - First Published

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

Thank you for the source code - was really helpful. Just few problems found.

Big problem is in my understanding why input

newdouble[] {1,2,3}[1]

does generate TargetInvocationException?! (Here I use double to be the return value type)
When I use input

newdouble[] {X,2,3}[1]

the return value is correct without any exception!! This is a must to be able to extend your code further on. So any comments here are appreciated.

Another question: When I try to create VisualBasicExpressionCompiler and/or JscriptExpressionCompiler it works fine with standard expressions.

2+3

I cannot, however use FunctionClass members.

2+3*X

It seems to be because FunctionClass is part of CSharp library - so the language is not compatible. I expect that when I would create the FunctionClass in VisualBasic language library it would work. But that is not what I want to have. My idea was to keep working with CSharp in FunctionClass and be able to choose language for the input string. Then compile the string using corresponding LanguageExpressionCompiler. Any ideas how to merge these 2 things together - if possible?

Well, I figured out the problem. When you create an array the way you were doing it, the compiler does something tricky - it creates a global array struct that your method then starts to reference. This in turn the function code rely on elements outside the function scope, which is where your errors are coming from. Basically dynamic methods are supposed to behave as static methods - they shouldn't rely on anything that wasn't passed in as a parameter. Essentially in your expression compiler you can check methodState variable before it's return to see if any TokenOffset.Fields exist. If they do you might as well throw a meaningful exception at this point because the function is no longer truly "static" as it has dependencies outside it's scope.

I had the same problem too using the "Loop" version for calling the dynamic method. I think the issue is the parameter list for the dynamic method. In my project, I modified the source string for the code file in the CSharpExpressionCompiler class to declare a non-parameterless function. No matter what, AccessViolationExceptions kept occurring. To fix it, I created a simple class that contained a protected parameterless constructor and a constructor that accepted parameters. It looks like this:

Next, I made sure that the dynamic method declared no parameters. Then, once it was time to invoke the method through the delegate, I just passed it a new instance of the ScriptProxy class with the necessary variable predefined as class properties. It worked.

I'm guessing that it's possible to use properties with public set accessors instead of these private ones.

I tried to use your code here. I have an assembly in which I have a "framework" with all the common code I use. I referenced your project from my "framework dll" project. Then I am trying to use it from a test application.

Could you then try one more thing? Create another DLL project and reference your code there. Then try to use it. This is been annoying me. I've spent a full day trying to fix the problem. The DLL is there, the application has access to it, but somehow when that line of code runs the problem appears.

Nice peace of work!, I'm planning to use a few parts for my own work....

I've found one thing that looks like a bug......

In the CSharpExpressionCompiler you're formatting the return type to a string.
In case the returntype of the expression is a generic type containing a generic type or a nullable type, the formatting is incorrect (e.g. List or List>).

You're adding the fullname for all generic arguments, this can cause the compilation to fail.

The 'weird' timing is because of where I stuck the start/stop for the stopwatch class. The single evaluate covers the entire time it takes to compile using the separate AppDomain and then run it once. The loop only times the evaluation so you can see the compile time is the most time consuming operation. I guess I was thinking of the scenario of how this could be used. The expression would be compiled once (expensive operation), then executed many times (very fast).

I hope that explains the difference in timing.

Also, if you do get a chance to look at it further more comments will be appreciated.

Of course it works on my machine, if it didn't I'd work on it until it does.