Validation – and other new features in the .NET API stack

During the long and rainy summer here in Amsterdam our team has been working hard to expand the .NET API to support more advanced usecases like full profile validation. Though validation may be The Most Wanted Feature for many of you, it is certainly not the only exciting new feature we like to showcase, so I’ll give you a brief overview on what’s coming up in the newest version of the .NET API in the sections below.

Poco-less parsing

We all love the (generated) .NET classes like Patient and Observation to manipulate and create FHIR data, including the parsers and serializers that turn them into xml and json. In fact, this is what the .NET API started out with some years ago. There is a growing class of tools however, that only need a subset of information, and for which the all-or-nothing nature of working with POCOs is a hindrance. Some of the reasons you may want to start working with poco-less parsing are:

Performance Our generic FHIR REST server Spark does not really need to parse data into POCOs just to index and store the data into its internal format.

Flexibility Tools may want to work with partially incorrect or incomplete resources, which would never be parseable into the generated classes. Our profile editor Forge, for example, would gladly try to read StructureDefinitions that are incomplete or incorrect so the author can correct and save them using that tool.

Version independence You may like to work with FHIR data from multiple FHIR versions, and write code that deals with the differences. With the POCO classes you would commit yourself to a specific FHIR version. Using Poco-less parsing allows you to do that. This will be used by the registry tool Simplifier to support profiles in both DSTU2 and STU3.

The central abstraction for working with FHIR data without using POCOs is IElementNavigator, which includes a set of extension methods to do LINQ-to-Xml-like navigation on the data:

Many of the newers parts of the .NET API are built on top of this interface and since implementing IElementNavigator is pretty straightforward, you could implement this interface top of your own (proprietary) model and have it participate in the functionality the .NET FHIR stack offers, like validation. Out of the box, we provide an implementation for POCOs, XML/Json and probably RDF.

FhirPath evaluation

In the meantime, work has been going on on a HL7-designed navigation and extraction language called FhirPath (formerly known as FluentPath). FhirPath looks a lot like XPath and can be used to walk trees, extract subtrees, formulate invariants, just like XPath. In fact, almost all XPath invariants in the spec have now been replaced by their FhirPath equivalent.

In parrallel, we have been adding a FhirPath compiler and execution environment to the API. It’s built on top of IElementNavigator described above, so you could now say:

Since we’ve also implemented IElementNavigator on top of the POCOs, you can now quickly select any part of an in-memory instance or run FhirPath invariants on top of POCOs.

Underneath, a FhirPath compiler will turn these FhirPath statements into ordinary .NET lambdas, so if you execute the same statement multiple times, you’ll be running native lambdas, not re-interpreted strings.

Retrieving conformance resources

Most of us will sooner or later need the metadata for Resources, called the conformance resources in FHIR. These allow you to get information about which elements are members of a given resource type, which datatypes exist, cardinality of elements etcetera. The .NET API has a new abstraction (based on previous work that’s been around for a while), the IResourceResolver.

It’s main method is ResolveByCanonicalUri() and it will get you the conformance resource (StructureDefinition, ValueSet, OperationDefinition etc) with a given uri. All core datatypes have uri’s like http://hl7.org/fhir/StructureDefinition/HumanName, but these could of course also be more specific profiles, like Argonaut’s ‘DAF Patient’.

We have provided implementations for locating these resources within a zip file (like validation.zip that is available on the FHIR website) and within a directory where your application is running. Resolvers can be cached and combined, so you can have a resolver that first tries your local directory, then the standard zip, then goes out to the web.

Of course, you can write your own implementations for these interfaces. We did so for the registry Simplifier, where we needed an IResourceResolver that resolves uris to resources using the database of profiles present in the registry.

These abstractions are used by the validator and the terminology modules to retrieve referenced profiles and valuesets when they are encountered.

Navigating StructureDefinition’s hierarchy

If you have worked with StructureDefinition, you know the pain of dealing with its flat list of Elements: even though the StructureDefinition expresses a (deeply) nested tree of elements, these are represented as a flat list (with hierarchical paths). You’ll generally need a lot of smart string manipulation to just find “the next sibling” for an element, or move back to its parent. To avoid this kind of code creeping all over the place in the API, we developed an internal class that we have now made public, the ElementDefinitionNavigator. It’s highly optimized for speed and low-memory use so you can now move around the hierarchy expressed by the StructureDefinition with ease:

Creating Profile snapshots

Maybe not a feature everyone will need, but if you are authoring profiles you will need at some point to turn your changes to a profile (“the differential”) into a snapshot, that can be used as input for e.g. rendering and validation tools. This is a pretty complex job, and Grahame Grieve, Chris Grenz and our own Forge developer Michel Rutten have been busy to make sure our tools can read each others outputs. Mind you, that means long nightly sessions on the subtleties of merging differentials into bases – but Michel packed it all in the new SnapshotGenerator. It builds on the resolver and ElementDefinitionNavigator above to do its work and using it looks deceptively simple:

// Regenerate the snapshot for the StructureDefinition
gen.Update(def);
[/code]

And yes, it will handle your re-sliced re-slices.

Terminology services

Terminology is a vast and complex area, and we have no intention to provide a full-fledged terminology server in the API, but there’s now at least ITerminologyServer and its lightweight, in-memory implementation LocalTerminologyServer. It supports ValueSets with define and compose statements, however it does not support filters. It’s designed to handle most common uses of ValueSet, but will happily throw NotSupportedException if you push it too far or you reach a certain maximum number of concepts in your expansion.

The LocalTerminologyServer depends on two pieces of new functionality:

The `ValueSet` class now has methods to search by code/system in an expansion

There is a new class ValueSetExpander, which would expand a valueset in-memory (with the caveats mentioned above)

Most likely we will still add another implementation called MultiStrategyTerminologyServer that would first use this local server, and when that fails, reach out to a real terminology server and use the FHIR REST Terminology operations to get what it needs.

Validation

That brings us to the most interesting addition for most people, the validator. It’s been a long-standing wish to add this to the .NET API, but thanks to support from UK’s NHS Digital, we have been able to work on this as well.

If you have been reading about the new features above, you can see these are all pillars that make validation possible:

A way to retrieve StructureDefinitions and other conformance resources to validate the instance against – the `IResourceResolver`

Generate snapshots for these definitions that serve as input to the validator

Be able to run constraints formulated in FhirPath

Be able to validate bindings using `ITerminologyService`

Mix these together, add some smart validation logic and you get the Validator.

It is not completely done yet -of course we left slice validation till the end- but it does most other constraints, including binding and simple extension validation. It handles Bundles and aggregation modes as well.

Learning more

While writing this blog I realized how much functionality we have added, and of course I had to leave lots of details out of the description above. Best way to get started with it is by running the validator demo and looking at the code and unit-tests. No, indeed, we have no online documentation about all of this yet. Another great way to learn more is to visit the upcoming HL7 FHIR DevDays where I will have a session on these (advanced) uses of the FHIR .NET API.

For now, let me get back to my Visual Studio and try to get it all polished and finished up for you.

Comments

[…] based on the generic IResourceResolver interface, as explained by Ewout in his recent article Validation – and other new features in the .NET API stack. The library exposes default implementations that support resolving resources from disk and […]