Archive for the ‘LINQ’ Category

About a month ago, we restructured re-motion’s folders and namespaces to make re-linq a top-level project within re-motion. This move had been a long time coming but the time was never right until now. It is also the first step towards pushing the re-linq source code into the CodePlex repository entirely. This second move will happen some time in the future.

What does this mean for re-linq’s…

…users? New namespaces and assemblies (Remotion.Linq instead of Remotion.Data.Linq). This is going to be the last foreseeable change in the namespace and assembly structure that will affect re-linq’s frontend (Remotion.Linq.dll).

…developers? The source code is now better isolated from the rest of re-motion, making it easier to work with.

Fabian had the news first, but here it is: We wanted to take re-motion to CodePlex already, but LGPL v3 is not a valid license there. We modified the license headers of re-motion from LGPL v3 to “LGPL v2.1 or later”, so we should be cool with CodePlex.

All the stuff, including the Subversion repository, is still here on re-motion.org, and CodePlex has only a few links and downloads. That’s a first step, and we believe that some people will learn about re-linq only through its presence on CodePlex.

Later, we plan to decouple re-linq from the rest of re-motion, since there is not a lot of shared code: re-linq uses almost nothing from re-motion, and it should therefore be a separate project, with a separate release cycle.

re-store (re-motion’s ORM) uses re-linq and will continue to do so. In upcoming releases, re-store will be treated just like any other downstream project from re-linq’s perspective.

As Fabian recently posted, re-linq just got its body parts reassembled. This is good news for everyone who wants to create a capable LINQ provider. Fabian has the details, but let me phrase it for mere mortals like myself.

Creating LINQ providers used to be a pain. It looked easy from a distance, since LINQ is similar to most query languages in many ways (including SQL, NHibernate’s HQL, Entity Framework’s Entity SQL or even the querying part of XQuery). How hard can it be to transform a LINQ abstract syntax tree (AST) to any of those? The problem is, LINQ won’t give you such a tree, but something entirely different that is really hard to translate to anything.

The main trick of re-linq is that it gives you exactly what you’d have expected in the first place: an AST that resembles the LINQ statement you wrote. Plus some infrastructure to make your transformations (including visitors and a few stock transformations you’re likely going to need) and other stuff (like creating additional queries for eager fetching).

Now finally creating a LINQ provider is as simple as you would think. You don’t know what the hack transparent identifiers are and how they make your live harder? Now you don’t have to. We believe it’s perfectly possible to create a very powerful LINQ provider in a few weeks. If you want to support every conceivable LINQ query, it’s going to be a bit more, depending on how different your target language is from LINQ. But still, there’s a lot in re-linq that helps you achieve even that.

This probably matters most for the NHibernate team. They just released a LINQ provider that supports basic query scenarios, but they plan to switch to a new one that uses their new HQL AST model. Using re-linq, they will hopefully be able to quickly create a provider that surpasses the old one’s capabilities by far. Steve Strong is already working on it.

I’m pretty sure that everyone who has ever tried to create a serious LINQ provider has suffered through the same stages:

Fascination: The way C# 3’s new features were combined to create query comprehensions are cool. Lambdas, anonymous types, extension methods, and all the rest are really useful language features on their own, and the query expression syntax (from … select) is really just a very thin layer on top of it (although they somehow managed to get Monads into the game).

Excitement: Via IQueryable and Expression<Func<…>>, we even get ASTs for the queries we write, and we can process them to generate SQL!

Hope: LINQ query expressions look much like SQL, so there must be a simple way to transform one into the other.

Frustration: The Queryable class makes a big mess out of those nice-looking query expressions. As we look at more complex queries and read the C# specs, we realize that the transformation from query expressions into method invocations follows some strange rules. They are very necessary for creating expressions that can be compiled and executed in C#, but unfortunately, we get the same bloated expressions for our ASTs. Now transformation to SQL doesn’t look so simple anymore. Transparent identifiers anyone?

Despair: System.Core does not contain any useful code to deal with those ASTs. LINQ to SQL does, but it doesn’t expose it (unfortunately, because LINQ to SQL is one oft the few LINQ providers that can deal with anything you throw at it).

Capitulation: We settle with a simple LINQ provider that understands just a primitive subset of LINQ (and throws exceptions for everything else), or we find that other LINQ providers might be good enough.

OK, now the last point is an exaggeration. Not everyone surrenders to the complexity of LINQ, or we would not have any useful LINQ providers. But few people actually get past this point, just have a look at Frans Bouma’s 15-part series Developing Linq to LLBLGen Pro. And honestly, it’s a real PITA that everyone would have to do this from scratch.

If we look at the nice syntax that query expressions have, it’s really a pity. Why can’t we have ASTs that resemble this beautiful syntax? After all, they are so much more like anything we’d want to transform them to (except for compiled code). SQL, XQuery, NHibernate’s HQL, Entity Framework’s Entity SQL, you name it.

Starting last year, we created a LINQ infrastructure that supports our own O/R mapper (re-store), and we started from exactly this point: To reduce complexity, we’d transform LINQ ASTs back into ASTs that resemble the original query expression syntax. (In some cases this is not the original syntax, since the extension methods of Queryable can also be called directly. But in compiled code this makes no difference, so we transform everything into query expressions.)

The result is re-linq. If you are thinking about writing a LINQ provider that supports more than just from/where/select, I’d like to encourage you to take a close look at re-link. Also, do not hesitate to contact us about missing features or chances to participate. (Unfortunately, providing public access to our issue tracker and a mailing list are still on our to-do list, but supply will follow demand!)

Fabian has written a short paper about re-linq (including open issues), and we’re going to publish more information in this blog soon. Stay tuned.