Introduction

It has been a while since I wrote an article here at CodeProject, but that does not mean I have not been busy. Far from it, I have been very busy
digesting new things (at least new for me), and reading a lot. In fact the book I just finished reading (Martin
Fowler's 'Domain Specific Languages') kind of inspired this article.

I have basically always had an interest in the more obscure elements of
software engineering, sure I like doing pretty stuff with WPF, however I find
myself going back to my computer science roots a bit more these days, and
wanting to explore some of the weirder areas, and let me tell you Domain
Specific Languages (DSLs) are a pretty weird (yet powerfu) place.

If one were to follow Martin Fowlers thoughts, the DSL world would be broken
down into two parts:

Internal DSLs: These are DSLs that are meant to be used by software
engineers/developers and should not be shown to end users. As such they may
comprise some fairly technical syntax.

External DSLs: These are DSLs that are intended to be used by end
users, and as such can be expected to be described in a common language that
the end user and the software engineers who must parse the DSL text both
understand.

Now let me just tell you that
Martin Fowler's
'Domain Specific Languages' book is about 600 pages long, and goes into way
too much detail for me to even try and condense into a single article. So we
will not be going into External DSLs at all in this article, rather we shall be
concentrating on a sub set of Internal DSL tools/trickery.

I will start this article by going through a few common ideas/techniques used
for developing internal DSLs, and then I will go through a simple example, and
then we will proceed to go through the uber example.

Semantic Model

Put simply this is the model that's populated by a DSL. All the DSL does
is provide a readable way of populating that model.

Martin Fowler has this pretty good thing to say about semantic models in regards to a
StateMachine DSL that he has as part of his book.

Looking at it
from this point of view, the DSL merely acts as a mechanism for expressing
how the model is configured. Much of the benefits of using this approach
comes from the model rather than the DSLs. The fact that I can easily
configure a new state machine for a customer is a property of the model,
not the DSL. The fact that I can make a change to a controller at runtime,
without compiling, is a feature of the model, not the DSL. The fact I’m
reusing code across multiple installations of controllers is a property
of the model, not the DSL. Hence the DSL is merely a thin facade over the
model.

Builders

By using separate classes that construct our semantic model we are able to
keep our semantic model clean of any DSL parsing code, which illustrates a good
separation between parsing code and the eventual semantic model which a given
DSL represents.

I won't say too much about builders here, as they are easier to understand
with an example, all that they are really, is little helper classes that have the
ability to create a correctly populated semantic model. Like I say you will see
a demo of this is the examples associated with this article.

Parser Tree

Parser trees are an interesting thing that can be used in the creation of
internal DSLs. So just what are parser trees exactly? Well the way to easily
grasp what these are, is by thinking about the ability to pass around a tree of
code expressed as a data structure.

Parser trees are a fairly novel thing, and not all languages support this
concept. Luckily .NET has the
Expression namespace which may be used for this exact purpose. Just for
completeness the
Expression classes were introduced to allow LINQ to SQL/EF and IQueryProviders
to translate code passed around as data structures into SQL statements, but that is neither here nor there, all that
is important is that .NET supports the ability to pass around code as data
structures, which allows us to pass around strongly typed code as if it were a
regular data type.

The most widely used/seen/popular example of this might be to extract a property name from
a LambdaExpression which would be done as follows:

Which we might use like this, where this is commonly used with
INotifyPropertyChanged implementing objects:

PropertyInfo prop = <GetProperty<Person>(x => x.Age)<

Method Chaining

Method chaining is a common technique that can be used where each method
would return an object (which is most cases would be the current object itself
OR a builder object that builds certain parts of the semantic model),
which allows the calls to be chained together into a single statement.

Fluent Interfaces

I found a nice post on Ayende blogs (and he has his own
DSL book too, so he can
be trusted don't worry) that describes what I would consider to be
the essence of what fluent interfaces are and how they compare to method
chaining. Here is what Ayende said:

Method chaining is something that you would certainly use in the a fluent
interface, but that is like saying that you need to use interfaces when you
build a plug-in framework. The fact that you are using something doesn't mean
that what you do is only that something.

Fluent interfaces are different
than merely method chaining because they allow you to express your intent in the
domain terms and allows you to get more readable code. Method chaining, operator
overloading, nasty generics tricks are all part of that, certainly, but the end
result is much more than just a simple method chain.

Ayende also shows a nice comparison of method chaining and fluent interfaces,
here is what his blog said

Knowing When to End

One of the major problems in working with method chaining is how to preserve
the correct sequence of calls and how to know when the chain is actually
complete. So how does one know when to complete the chain, and thus return a
populated semantic model?

Probably the easiest way to deal with this dilemma is to include some sort of
"Ending" method that signifies the end of the chain.

Progressive Interfaces

Another issue when working with method chains is that you may always be
returning the current object which may allow the user to call to many
methods/the wrong methods, in the wrong places. One common technique in dealing
with this is to return interfaces which dictate what methods/properties etc etc
are valid at that point in the chain. This technique it called progressive
interfaces (at least that is what Martin Fowler likes to call it)

By using progressive interfaces, the chain of methods that may be called by
the calling code, will be limited to those methods exposed by the interface at
that point in the chain.

Simple Example

This first simple example is really easy, and should demonstrate several of
the techniques we just discussed, for example you will see examples of Method
Chaining / Fluent Interfaces / Builders / Semantic Model and we will know when
to end the chain.

Scenario

So before we look at the demo code, let's just set the scene of what we are
trying to do. I used to be into creating electronic music, where I had a studio
filled with various bits of electronic equipment, so I thought that a DSL that
creates a simple music studio might be fun. The studio is very simple and
contains only 2 things

A single mixing desk

Any number of samplers

Simple Example: Semantic Model

In terms of a Semantic model this is what we will be trying to create. I
could explain this further, but I think most of us can/should understand a class
diagram

Simple Example: Builders

As previously stated by using helper builder classes we are able to separate
out the DSL building / parsing code from our semantic model, which is a good
thing.

StudioBuilder

For this scenario the first place we need to start is the top level
builder, which is the builder that will build new Studio objects for
us. This is shown below.

The use of context variables that are used to store the current context
of the DSL as it's being constructed. This can be seen by the
currentMixingDesk and currentSampler variables. These
ensure we are manipulating the correct object whilst working with the DSL
creation

The use of method chaining, see the MixingDesk() method for
an example

It does exhibit a fluent interface, in that the method names have been
chosen to express their intent to the user to assist in buildng the DSL

Their is a way to signal the end of the chain, see the End()
method for an example

MixingDeskBuilder

So now that we have a top level builder in place, lets shift our focus
slightly to the MixingDeskBuilder. Remember there is only one of these allowed
in this demo DSL, as such the buider for this is very simple. Here it is:

SamplerBuilder

So next we move on to look at the SamplerBuilder. Remember there
can be any number one of these allowed in this demo DSL, as such the builder for
this needs a way of allowing more than one of these to be created. The way this
is achieved is to take a reference to the parent StudioBuilder and
to have a Sampler() method that simply calls the parents
Sampler() method that will add a new SamplerBuilder to its
collection.

Uber Example

Before we get started with this section, I need to also give you a bit more context about how
this article came about. So around the time that I was 1/2 way through
Martin Fowlers
'Domain Specific Languages' book, I was also debugging some of my code at
work, where I was in the middle of a test case where I was using the fabulous
Moq library, which is by far
my favorite mocking framework, and 2 things happened

I had an issue with a callback that I had setup with my mock, which
was not working, and I started debugging my code, and I noticed this thing
which I immediately recognized as a
Castle
dynamic proxy. I kind of had a eureka moment with
Moq where I was
like, Aha that's how they do it. Truth is they mention this on their home
page but I have never read that page, as
Moq has never really
gone wrong for me, so I never looked.

Massive kudos to the Moq
team I say, and how very timely I also say. True timing what you say

So without further ado, I wondering if I could write something like
Moq from scratch
using some of my new found DSL knowledge. It took me a while but I have managed
to make a working Moq
clone. I should point out that I did this without looking at their code at all, which I
am sure you will be able to tell if you compare the two. Now that I have
finished I obviously checked out there code and theirs is like well um just
plain better than mine actually.

Now before anyone mentions the dreaded word "Plagiarism" I actually contacted the
main author of Moq,
that's Kzu,
and told him what I was doing well before I wrote this article and he gave me
the ok. In case anyone is interested here is our email chain. (click the image
below to see larger version)

So there you go, I have Kzus blessing.

One other thing I should point out is that my simple

Moq
clone presented here is just that, it is simple, and is no where near as polished
as Moq, so anyone
thinking of trying out my simple demo here for their mocking needs, forget it, it was just done for
laughs really, use Moq.

That is not to say I am not happy with what I have done here,
which I kind of am actually, as it does most of the main things that
Moq
does in terms of DSL capabilities, which was after all, was the main reason I tried
to write this simple Moq
clone. So yeah I am happy with how this worked out for sure.

Just for completeness here are some of reasons why you would use
Moq
over my simple take on it, which as I say, was more about just having a go at
creating the DSL / Method Chaining / Fluent Interface / Progressive Interfaces /
Parser Tree syntax that Moq
uses.

Anyway we digress, here is why you need to use
Moq
and not my simple example:

I only support mocking interfaces

I do not support mocking of classes

I do not support nested mocks

I do not deal with ref variables

Anyway now that we have all that no technical speil out of the way, let's
continue on with the 2nd example, which is my simple
Moq
clone, which I have dubbed "The Uber Example"

I should also point out that although simpler than
Moq,
my simple Moq
clone actually does work, which you can see by running the demo code associated
with this article.

Uber Example: The General Idea

As I have just stated what I have created is a very simple mocking framework
that uses an internal DSL to build up a mock object, that can be used in place
of a real object. I have also stated that my simple DSL allows a subset of what
normal mocking frameworks produce, but I am not worried about that, what I am
worried about is showing you all the techniques involved with how I got there
and some of the DSL like syntax that you could use in your own DSLs.

So now that we know that we are creating a DSL to create a mocking framework,
what sort of things could we expect to see?

Well if we consider what sort of things a typical object allows we could
conceivably have an initial list of the following

Mocking methods

Mocking properties

Raising events

Throwing exceptions

So that is the sort of thing we might want to include in our DSL/Semantic
Model. However as we look into each of these areas in more detail we can start
to think about the way in which a normal (ie non mocked object) would operate.
It could potentially allow all sorts of behaviour such as:

Throwing an error in a property setter

Only accepting certain inputs

Expect to only be called a certain number of times

You get the idea, so considering these things we can develop our DSL. What
I have chosen to do was imitate what I would consider to be a very well known clever
existing API (ie Moq),
but you can see how these extra concerns could influence your DSL design.

Uber Example: Semantic Model

In some ways the semantic model for this is very simple it just comes down to

Methods

Properties

Which is actually all that we can create using the DSL. What makes the
semantic model more interesting is that methods may needs to do things like
throw Exceptions, or raise events. So these considerations and the ability to
deal with these concerns then become part of the semantic model.

Uber Example: Common Techniques

This section will outline common techniques that are used throughout the "uber example" and these will provide
the building blocks for the subsequent sections

Common Techniques: Proxying

As this DSL is all about mocking, it should come as no surprise that there is
an internally held proxy object, which is used instead of the real object. You
may ask how this proxy comes about? Who creates it?

There are many free open source frameworks for dealing with dynamically
created proxies, in fact it is not that hard to come up with a simple one using
the .NET Remoting APIs. That said I have chosen to use Castle Windsor, which is
a tried and tested tool, I get it you could say.

So yeah we will be using a Castle Windsor proxy for the mock object generated
within this Uber Example DSL.

Common Techniques: Interception

To do the types of thing that the Uber Example is trying to do, we are
obviously into a quite niche domain, where we expect things to be returned from
a method when we do X, or we expect that when property Y gets set that its value will
be Z. This is all quite specific to the way a real object would work, but we
don't want to write real objects to deal with our wishes, what we want to do is
write some DSL code that specifies the behaviour, such as:

When Method X is called return List<Foo>

So how do we go about that? Well luckily, there are a number of frameworks on
the market that allow us to do this. This is really just Aspect Oriented
Programming (AOP) when you think about it, and how does AOP work? Well, in .NET
land, it usually works via method interception, but how do we do that?

Well, like I say, there are many free Open Source frameworks for doing AOP with
.NET. For this article, I have chosen to use Castle Windsor, which uses a
technique called interception, which is made possible by the use of
Interceptors.

Quite simply, Interceptors
allow you to hook into the method calling pipeline and decide
what return values/parameters etc., should be passed (if at all) to the base
method implementation.

This is a very powerful technique. By using
Interceptors, we are able to pretty much deal with any of the mocking
requirements. In an
Interceptor we can do the following:

Do a callback for a method being called

Increment a counter for the number of times a method has been called

Work our whether a method call was valid

Raise an Exception if we were asked to, during a particular
method/property call

Raise an event during a particular method/property call

This technique is used throughout this Uber Example, where the following
Interceptors are used:

Common Techniques: Finding Event Names

The actual raising of events is a tricky beast to deal with, and was one of
the most challenging aspects of all of the work I did for this article. Why is
the raising event(s) code so difficult you may ask? It is down to the fact that I
wish to be able to identify the event to raise in a strongly formed way, that is
I do not wish to pass around magic strings.

That is the problem really, but why is that an issue?

Well, in .NET, the only thing you can do with an event in an external class
that makes use of the event is to add or remove handlers using the +=
or -= operators. So how by simply using these operators can we
obtain the actual event name?

This turned out to be quite a difficult problem, and one that did not come to
me straight away. The following diagram illustrates what I managed to come up
with eventually.

In words what happens is that we accept a delegate that takes T,
which just happens to be the same type as the mock object we are building. So
that part I hope is OK. The next thing we do is create a extremely short
lived Castle dynamic proxy
of type T, which will only be used to analyze calls made to it via
Castles interception feature that only serves to grab information about what
methods/events etc., were called on the proxy. I have called this the
PredictiveInterceptor.

So we now have a proxy object in place and a PredictiveInterceptor
which will store methods (and event add_XXX/remove_XXX are really
methods) invoked on the proxy. OK, cool, so now all we do is call the original
delegate (Action<T> passing in the short lived proxy we created and then find
out what methods were called on it). It's that simple.

It did take me a while to come up with that one, anyway that is essentially how this Uber Example works when raising events.

Uber Example: Properties

Obviously since we are dealing with a DSL for creating mock objects, we are
going to have to provide some way of dealing with setting up properties for our
mock using our DSL, but just what sort of thing should we allow for.

Properties can accept values and return values, and can throw Exception(s)
and also raise events, and can be called once, twice n-many times of never. As
such here is what I came up with that the DSL needs to support for properties:

We need a way of setting up what values a property should return

We need a way that a Setter should throw an Exception. I feel it would
be quite strange for a Getter to throw Exceptions (ok it might but it would
be rare)

We need a way that a Setter can raise an event. I feel it would be
extremely strange for a Getter to raise an event

So that is what our DSL will support. Now let's continue to have a look at
how this is achieved.

Properties: Returning Values

We can setup a property using the SetupProperty method (which is part of the method chaining methods of the Mock class) as follows:

Properties: Throwing Exceptions

Having the ability to throw an Exception from a property probably only makes
sense in a Setter of a property, after all who wants an Exception thrown each
time they try and read a property. So how is this achieved? Well, it turns out
this is very simple. We simply have a DSL fragment that accepts either a generic
Type for the Exception to throw, which is created using
Activator.CreateInstance,
or we accept a pre-populated Exception object.

In either case, we store the
Exception against the PropertyData<T>builder and we make use of this
Exception in the
PropertyInterceptor via the use of an ISupportExceptions interface that the PropertyData<T>builder class also implements.

We are then able to check if an Exception should be raised when we are setting a property within the PropertyInterceptor. As I have stated
on numerous occasions, I am using Castle Interceptors.
Here are the relevant parts of the PropertyInterceptor.

Properties: Raising Events

Now that I have gone through the basics of how an event name is deduced (see Common Techniques: Finding Event Names)
in a strongly typed manner, let's concentrate on looking at how the property building DSL fragment is able to actually deal with events.

It starts with the ability to accept the event in a strongly typed manner,
which we discussed above, but for completeness, here are the methods that accept
a standard event arguments signature to raise an event, and also a custom event
argument signature:

So that gets us a name to store for the event which we store in a List<EventWrapper> object for later use.

But how about raising these actual events? All we have done at the moment is say that when property XYZ setter is called, we want to
raise event ABC in a strongly typed way. But how do we actually call the correct event invocation list callback handlers when the property value does change?

To understand that part we need to recall that there is actually a EventInterceptor
at work which is enforced for the actual mock object the DSL is creating. Here is that code, it can be seen that we store the callback
delegates against an IEventRaisingAgent (this is the actual mock object the DSL is building):

So that is how we have a knowledge of what event callback delegates to call. But how do we actually call them? Well, for that
piece of the puzzle,
we need to jump back into the PropertyInterceptor, whose most relevant parts for events are shown
here:

It can be seen that this simply calls the DSL stored PropertyData<T>.RaiseEvents() method for the current property invocation. So all
we need to do to complete this wicked web is have a look at the PropertyData<T>.RaiseEvents(), which is as shown below:

Properties: Verification

Whilst working with the mock object that this Uber Example DSL builds, it is
not inconceivable that one may want to know how many times a certain property is
called. This is called Verification in this Uber example.

Here is the basic idea:

Every time we see a property getter/setter called, we increment an
internal counter within the mock object that the DSL represents

At the end of using the mock object created by the DSL (say in a
UnitTest), we can verify the property

So you can see the process is pretty simple really, we just store the number of times that we want a property getter/setter called directly on the property
builder, and the PropertyInterceptor verifies that the actual number of getter/setter calls to a particular property is also stored against the
property builder.

And here is some user code that deals with how to perform verification using the DSL:

Uber Example: Methods

Obviously since we are dealing with a DSL for creating mock objects, we are
going to have to provide some way of dealing with setting up methods for our
mock using our DSL. But just what sort of thing should we allow for?

Methods can accept values, can accept only valid values, and return values, and can throw Exception(s)
and also raise events, and can be called once, twice n-many times of never. As
such here is what I came up with that the DSL needs to support for methods:

We need a way of setting up what values a method should return

We need a way of setting up what values will be passed to the method

We need a way of calling back user code when a method is called

We need a way that a method should throw an Exception

We need a way that a method can raise an event

So that is what our DSL will support. Now let's continue to have a look at
how this is achieved.

One thing to note is that a lot of the functionality we are going to discuss
for methods is much the same as we previously discussed with properties, the
only difference being that for methods we will be using the following two data structures
instead of the ones we previously used for properties:

MethodData<T>: The method builder that is used internally
to create/store method data inside the semantic model

Methods: How the MethodData<T> Builders are Created

As most of the concepts for dealing with methods have already been covered by
the concepts we went over for properties, this section will only detail those
parts that are different from the properties we have already seen.

One immediate difference is that the mock object that underpins the Uber
Example DSL, has MethodData<T> builder objects that are built for
dealing with methods, rather than PropertyData<T> which we already
saw above, when dealing with properties. So how are these MethodData<T>
builder objects created in the first place?

Where both the Setup methods take an Expression which holds the method to call, which is obtained using the
GetMethodFromExpression helper method shown above.

Each of the following sub sections make use of these MethodData<T>
method builder objects which are returned by the mock Fluent interface that were obtained using this code.

Methods: Validating Arguments

One cool thing that we might like to allow in the DSL to do is to specify argument
checkers that can be applied to the method's argument values, such that when the method is
actually called, the method invocation will run through any argument checker
that were initially setup on the mock for the particular method prior to the method invocation. If any of
the argument checkers that were setup in the DSL for the currently invocation
fails, an InvalidOperationException is thrown, indicating that an invalid argument
was supplied to the method being invocated.

So how does this work? Well, as before, we start by looking at the actual DSL syntax, which is something like this:

It is a dead simple class, so how do these argument validators get created?

The creation of them actually happens in the Setup methods that we saw earlier within the Mock class, where they both
end up calling this helper method that returns a List<IArgumentChecker> object which represents the argument checkers for the method that the DSL
is currently building.

Or we create an IArgumentChecker that will make use of the original Predicate<T>
that was passed into the DSL method builder. So now that we have some
IArgumentChecker(s) created and associated them with the MethodData<T>
method builder, how do these get used to actually do the validation? Well, the
answer to that is pretty simple, we simply intercept any method call and
run the arguments passed to the method through the List<IArgumentChecker>
associated with the method invocation and see if the are all valid. If they are
not, an InvalidOperationException is raised. Here are the relevant
parts of the MethodInterceptor:

You may ask why there are two types of callback methods chaining DSL methods
available, and you would be right to ask this question. The reason is that since
we do not know what the method signature is in advance of it being mocked, we
don't know what type of callback delegate would be needed. Sure we could make a
dynamic delegate using Reflection.Emit which is an avenue I looked into, but that
dynamic delegate would also need to be known about up front in order to get the
WithCallBack MethodData<T> builder to work correctly.

It seemed like a chicken and egg type of thing. This is why I have opted for
two separate callbacks, there
is WithCallback which deals with 1 or 2 arguments, and then there is
WithPropertyBagCallback which creates a dynamic property bag type object with
any number of parameters. When I finally looked at what
Moq
did, I can see this is in fact an issue Moq
only got around by including delegates that took up to 20 arguments. Whilst I
can see why they did that, I did not like it that much.

Anyway let's continue to look at the two MethodData<T> builder
methods that deal with callbacks, shall we?

WithCallback

Here is the MethodData<T> builder
code that deals with the simple cases where we know the argument types and
number of arguments:

OK, so now that we have all these callbacks stored against the MethodData<T> builder,
it is just a question of doing the actual callbacks. As before this is done in
the MethodInterceptor, where this is the most relevant part of the
MethodInterceptor code:

Methods: Interception

As I have stated on numerous occasions, I am using Castle
Interceptors. The interceptor associated with method interception is called
MethodInterceptor and I showed you a complete listing of that earlier.

Methods: Demonstration

To illustrate all these features, let's consider the following user DSL code:

Uber Example: Miscellaneous

One of the final things I would like to mention is that it is also possible to raise events directly from the mock object. This is done by the use
of the following methods, which allow the DSL to specify either a standard event or a custom event signature.

As before, the technique used is to use the extremely short lived proxy that has its calls intercepted to obtain/store the actual event information. This works the same
as previously discussed. What is different with raising the events directly on the mock object, is that we are able to raise the event right then and there, we do not need
to wait for a property setter or method call to occur to work out whether the DSL provided for those properties/methods specified an event to raise.
The DSL portion is literally saying "Raise Event_XYZ NOW".

The code to obtain and raise the events directly from the mock is shown below. It actually really boils down to a few simple steps:

Run the very short lived proxy where the original portion of the
expression tree (x => x.Changed += null) is essentially
played and intercepted against the short-lived proxy to obtain an event name
from the short lived proxy MemberInfo.

Look to see if we have any events on the mock that match the name of the
event just requested.

If we find some events for step 2, just invoke the delegate handlers for
the InvocationList for the found event name.

Uber Example: Demo, Please Examine It

The demo illustrates this far better than my words I feel, please have a play with it. The Uber demo is "SimpleMock.TestApp". Have a play,
I hope that you can grasp how it works from the examples that I have included in the demo code.

That's it

Anyway that is all I wanted to say this time, I hope you have enjoyed this article, I know I have enjoyed writing this one, as it was not obvious how to do certain
things and required a lot of thinking at some points in the article/code. If you too enjoyed it, could you maybe give it a vote or let me know what you thought,
or even better both? Many thanks.

By the way my next article will be a full rich client/server side code all the way to the database n-tiered article, which will be used to demonstrate various aspects/techniques
people have been asking me to talk about for ages, where I have been putting it off due to the sheer amount of work....The time has come though, so that is what I will
be working on next.

Oh, by the way, it will be a Zombie explorer application, charting zombie activity around the globe, and it will involve some sort of map component, should be cool.

I always enjoy reading your articles. Your insights, demonstrations and excerpts are always educational, clear and well composed. I greatly value your contributions to these forums as they are second to none - I'm an avid follower.