A scalable pattern for building metadata

In the latest version of Quino -- version 1.8.5 -- we took a long, hard look at the patterns we were using to create metadata. The metadata for an application includes all of the usual Quino stuff: classes, properties, paths, relations. With each version, though we're able to use the metadata in more places. That means that the metadata definition code grows and grows. We needed some way to keep a decent overview of that metadata without causing too much pain when defining it.

In order to provide some background, the following are the high-level requirements that we kept in mind while designing the new pattern and supporting framework.

Manage complexity

A simple model should be easy and straightforward to build, with no cumbersome boilerplate; complex models should support multiple layers and provide an overview

Leverage existing knowhow

Our users don't want to learn a new language/IDE in order to create metadata; neither do we want to provide support for our own metadata-definition language

Support modularization

Modules can be used to hide complexity but are also sometimes necessary to define hard boundaries in the application metadata

Support extensibility

Interdependent modules and overlays will need to refer to elements in other modules; there needs to be a standard mechanism for defining and accessing metadata elements that doesn't rely on string constants1

Support refactoring

Rely on convention and name-matching as little as possible to avoid subtle errors

Support introspection

Developers that stick to the pattern should be able to maximize efficiency using common navigation and introspection2 tools like Visual Studio or ReSharper.

Definition Language

Quino metadata has always been defined using a .NET language -- in our case, we always use C# to define the metadata, using the MetaBuilder or InMemoryMetaBuilder to compose the application model. This approach satisfies the need to leverage existing tools, refactoring and introspection.

Since Quino metadata is an in-memory construct, there will always be a .NET API for creating metadata. This is not to say that there will never be a DSL to define Quino metadata but that such an approach is not the subject of this post.

Modularization

Quino applications have always been able to define and integrate metadata modules (e.g. reporting or security) using an IMetaModuleBuilder. Modules solved interdependency issues by splitting the metadata-generation into several phases:

Add classes and foreign keys

Add paths between classes (depends on foreign keys)

Add calculated properties and relations (depends on paths)

Add layouts (depends on all properties)

In this way, when a module needed to add a path between a class that it had defined and a class defined in another module, it could be guaranteed that classes and foreign keys for all modules had been defined before any paths were created. Likewise for classes that wanted to define relations based on paths defined in other modules.

The limitation of the previous implementation was that a module generator always created its own module and builder and could not simply re-use those created by another generator. Basically, there was no "lightweight" way of splitting metadata-generation into separate files for purely organizational purposes.

There were also a few issues with the implementation of the main model-generation code as well. The previous pattern depended heavily on local variables, all defined within one mammoth function. Separating code into individual method calls was ad-hoc -- each project did it a little differently -- and involved a lot of migration of local variables to instance variables. With all code in a single method, file-structure navigation tools couldn't help at all. The previous pattern prescribed using file comments or regions that could be located using "find in file". This was clearly sub-optimal.

The new pattern

The new pattern that can be applied for all models, bit or small includes the following parts:

Model generator

As before, there is a class that implements the IMetaModelGenerator interface. This class is used by the application configuration and various tools (e.g. the code generator or UML generator) to create the model.

Model elements

Metadata that is referenced from multiple steps in the metadata-generation process is stored in a separate object (or objects) called the model elements. (E.g. classes are created in the AddClasses() step and referenced in the AddPaths, AddProperties and AddLayouts steps.) The model elements typically has two properties called Classes and Paths.

Metadata generators

Module generators still exist, but there are now also metadata generators that are lightweight, using a metadata builder and elements defined by another generator (typically a module generator or the model generator itself).

This may sound like a lot of overhead for a simple application, but it's really not that much extra code. The benefits are:

Models, modules and lightweight parts all use the same pattern, with the same phases and method names. That makes it far easier to know where to look for a definition

Since the pattern is the same, it's easy to move functionality from one module to another or to split one module into multiple lightweight parts without doing a lot of refactoring

A small model will naturally grow to a medium or large model, all while using the same pattern. There is no moment during development where you have to do a major refactoring in order to get organized: the pattern will naturally support a clean coding style.

Building a model, step by step

But enough chatter; let's take a look at the absolute minimum boilerplate for an empty model.

The code above is functional but doesn't actually create any metadata. So what does it do?

It uses the generic MetaBuilderBasedModelGeneratorBase to indicate the type of Elements that will be exposed by this model generator. The elements class is created automatically and is available as the property Elements (as we'll see in the examples below). Additionally, we're using a ModelGeneratorBase that is based on a MetaBuilder which means that the property Builder is also available and is of type MetaBuilder.

It includes the DemoCoreGenerator which is a dependent generator -- it's lightweight and uses the elements and builder from its owner. The exact types are shown in the class declaration; it can be read as: get elements of type DemoModelElements and a builder of type MetaBuilder from the generator with type DemoModelGenerator. The initial generic argument can be any other metadata generator that implements the IElementsProvider<TElements, TBuilder> interface.

The model generator overrides AddMetadata to include the metadata created by DemoCoreGenerator in the model.

Even though it's not very much code, you can create a snippet or a file template with Visual Studio or a Live Template or file template with ReSharper to quickly create a new model.

Step one: define the model

Now, let's fill the empty model with some metadata. The first step is to define the model that we're going to build. That part goes in the AddMetadata() method.3

OK, now we have a model with two entities -- companies and people -- that are related to each other so that a company has a list of people and each person belongs to a company.

Step five: add translations

Now we'd like to make the metadata support German as well as English. Quino naturally supports more generalized ways of doing this (e.g. importing from files), but let's just add the metadata manually to see what that would look like (unaffected methods are left off for brevity).

Note that I created a local variable for both company and person. I did this for two reasons:

The code is shorter and easier to read

There are fewer references to the Elements.Classes.Person and Elements.Classes.Company properties. It's useful to keep the number of references to a minimum in order to make searching for usages with a tool like ReSharper of maximum benefit. Otherwise, there's a lot of noise to signal and you'll get hundreds of references when there are only actually a few dozen "real" references.

Step six: using private methods

You can see that the metadata-generation code is still manageable, but it's growing. Once we've filled out all of the properties, relations, translations, layouts and view aspects for the person and company classes, we'll have a file that's several hundred lines long. A file of that size is still manageable and, since we have methods, it's eminently navigable with a file-structure browser.

If we don't mind keeping -- or we'd rather keep -- everything in one file, we can see more structure by splitting the code into more methods. This is really easy to do because we're using the elements to reference other parts of metadata instead of local variables. For example, let's move the class initialization code for the person and company entities to separate methods (unaffected methods are left off for brevity).

Step seven: using multiple generators

While this is a good technique for small models -- with anywhere up to five entities -- most models are larger and include entities with sizable metadata definitions. Another thing to consider is that, when working with larger teams, it's often best to keep a central item like the metadata definition as modular as possible.

To scale the pattern up for larger models, we can move code for larger entity definitions into separate generators. As soon as we move an entity to its own generator, we're faced with the question of where we should create paths for that entity. A path doesn't really belong to one class or another; in which generate should it go?

Well, we thought about that and came to the conclusion that the pattern should be to just create a separate generator for all paths in the model (or multiple path-only generators if you have a larger model). That is, when a model gets a bit larger, it should include the following generators (using the name "Demo" from the examples above):

DemoCoreGenerator

DemoPathGenerator

DemoCompanyGenerator

DemoPersonGenerator

The DemoCoreGenerator will create metadata and assign elements like the display languages. It's also recommended to define base types like enumerations and very simple classes4 in the core as well. Obviously, as the model grows, the core generator may also get larger. This isn't a problem: just split the contents logically into multiple generators.

For the purposes of this example, though, we only have a single core and a single path generator and two entity generators. Since these generators will all be dependent on the model's builder and elements, the first step is to define a base class that will be used by the other generators.

You'll note that we only moved code around and didn't have to change any implementation or add any new elements or anything that might introduce subtle errors in the metadata. Please note, the classes are all shown in a single code block above, but the pattern dictates that each class should be in its own file.

Step eight: integrating external modules

So far, we've only worked with generators that are dependent on the model generator. How do we access information -- and elements -- generated in other modules? For example, let's include the security module and change a translation for a caption.

This approach works well with any module that has adhered to the pattern and exposes its elements in a standardized way.5 In this case, the core module includes the security module and retains a reference to its elements. Any code that uses the core module will now have access not only to the core elements but also to the security elements, as well.

Another major benefit to using this pattern is that the resulting code is quite self-explanatory: it's no mystery to what the Elements.Security.Classes.User.Caption is referring.

One last thing: folder structure

The previous pattern had a single monolithic file. The new pattern increases the number of files -- possibly by quite a lot. It's recommended to put these new files into the following structure:

[-] Models
[+] Aspects
[+] Elements
[+] Generators

The "Aspects" folder isn't new to this pattern, but it's worth mentioning that any model-specific aspects should go into a separate folder.

That's all for now. Happy modeling!

Naturally, the IMetaModel is always available and any part of the generation process can access metadata in the model at any time. However, the API for the model is quite generic and requires knowledge of the unique identifier or index for a piece of metadata.↩

By introspection, we mean that if metadata is accessed through .NET code structures -- like properties or constants -- we should be able to find all usages of a particular metadata element without resorting to a "find in files" for a particular string.↩

It doesn't have to go there. The DemoCoreGenerator could also set up the builder (since it's using the same builder object). To do that, you'd override AddCoreElements() and set up the model there. However, it's clearer to keep it in the generator that actually owns the builder that is being configured.↩

Simple classes generally have few extra properties and no layouts or short description classes.↩