Principal Consulting Software Developer at Haumohio.com, based in beautiful Christchurch, NZ. Writing about devops, F#, and software dev in general.

Dec 8, 2015

The trips and traps of creating a Generative Type Provider in F#

This isn't the story I was originally intending to write for 2015’s F# advent calendar. I was originally going to talk about my currently ongoing adventures with F# vs. Linux vs. Azure, but it doesn't yet have a satisfactory ending — so stay tuned!

Only add the root type(s) to a generated assembly, and only after all child types have been added to it

To store instance values — add a private field to each instance and access it via a central register of field definitions

The Problem

Neo4J has a fantastic query language called Cypher, with an accompanying library for .NET. However, in order to use cypher a programmer needs to write a lot of the query as a string.

db.Cypher.Match("p:Person").Where("p.born=1973").Return("p")

As you can see the return type (Person) and the query filter are unchecked strings even on the simplest of queries, and the return type is dynamic which can be troublesome in F#.

Wouldn't it be great if we could have something that give us node and relationship names and types, similar to the way that the SQL type provider gives us tables and columns.

The Vision

When the Neo4J client tried to populate the property using reflection it just sees the base type without any generated propertiesSo, let’s follow the pattern that the SQL type provider give us for instantiation.

We create a schema at compile-time against an active Neo4J database, and then connect to one with a “matching” schema at run-time.

Getting Started

If you've never delved into the realm of Type Providers, get the “Provided Types” helpers from somewhere like this and read Mavvn’s blog and this blog on functional flow. This provides the framework for building your own provider.

Why Generative?

So, I started off building a simple Erased type provider. But it only got me as far as expressing node and relationship names. This was great - I got data. But I really wanted to get return types with named properties as well, so I could use it as a useful object store.

The difference between this and a Generative one is that the erased one is compiled into the user’s assembly when used and is somewhat ephemeral. A generative provider’s type has it’s own assembly, and therefore the provided types can be referenced by C#, and (importantly in my case) serialized to from JSON/XML.

From the Top

The core of the top level of the code is to compose the node labels, relationship names, node property names, and node proxy types as members of the schema type that we’re trying to provide.

I created a few simple “helpers” to mutate and pass on the type-structure (basically an apply) like addMember(s) and addIncludedType to help with the functional style.

The simple stuff : Names of things

This is the node, property, and relationship names. It’s simple because they are all implemented as a static properties that return their own name as the value, just like you find in the Getting Started tutorials.

… but I had to define by own type (with the property names spelled exactly right) to serialize into , or use dynamic typing.

Filling types of thin air

So, I jumped into creating erased proxy types that could be used in the Return command’s type parameter.

Of course, the properties couldn't be static as the values needed to be different for each instance. I tried a number of increasingly nasty techniques to store values such as a hidden static dictionary with a combined key of object ID and property name, with mixed results.

Finally I cobbled something together for a single named property and got a sort-of unexpected result. The results returned the correct number of obj objects with no values! This is because the erased types are actually just the concrete type that the provided type is based upon, with a bit of decoration.

Lesson 3: Erased types look like their base types from the outside

My next gambit was to base the proxies on a type that has more stuff to hang values off. I created a record type with a name, id, a property hashmap, and getters/setters and based the proxy types on this.

Lesson 4: Provided types can’t be Records

At least that was true at the time — there has been discussions and this may or may not have been changed now. In any case, I changed it into a class and tried again. This time I got back the correct number of my special type, but with no values set, and came to the conclusion that the generated properties weren't being activated by the Neo4J client’s deserialization routines.

After a lot of reading (and re-reading) through Dave Fancher’s post I got something going. The trick was to add the created types to the created assembly once, and only after everything had been finished.

Lesson 6: Only add the root type(s) to a generated assembly, and only after all child types have been added to it

The Instance Variables

This was the bit I got stuck on earlier, and still caused me some headaches, but Dave Fancher came to my rescue here too with private fields.

Lesson 7: To store instance values — add a private field to each instance and access it via a central register of field definitions

I was on the right track by storing “stuff” in a mutable property list by type. But I also needed a private field attached to each instance. Then I could create a non-static property with a getter and a setter that alters/accesses this field via the list of field definitions.

You can see in the example above that the name of the type “Person”, as well as the data-shaped type schema.Proxies.Person (and it’s properties — e.g. name & born)has been provided to give us some compile-time confidence that the code matches the data. As an added bonus we can use a F# filter function in the Where clause, too.

Moving on

I started packaging this for Nuget, but have gotten around to finishing it, so you’ll need to compile it to use it. Also, feel free to send me a PR if you feel like packaging things for everybody (or any other improvements too, for that matter).