Interview: ClojureQL Overhaul

ClojureQL creator Lau Jensen recently took the
drastic measure of deleting his two-and-half-year long project
and starting from scratch. JAXenter caught up with him, to find out
why this was necessary, and what the overhaul has added to the
project…..

JAXenter: Can you introduce us to your
ClojureQL project?

Lau Jensen: ClojureQL is an implementation of
the primitives defined in Relational Algebra (RA) written in and
for Clojure. RA gives you a lot of power to express database
queries and relations without having to learn specific low-level
languages and their respective syntax, ie. SQL. With ClojureQL you
have a relatively small API, with a very consistent design, that
lets you express complex queries through very simple lisp
expressions.

JAXenter: After two and a half years, you
recently deleted the ClojureQL project and started from scratch.
What prompted such a drastic move?

Lau Jensen: The initial project was created
shortly after Clojure’s release to the public. Since that time both
Clojure and I have improved substantially.

The first project aimed at abstracting away all of the SQL
statements, both DDL (creation) and DML (queries/modification)and
we wanted to accomplish this for a myriad of backends. Because we
didn’t settle on using RA, we also had to design the api from
scratch. Our great ambitions also meant a great burden of work,
much greater than what our initial 2 man crew were able to carry
ourselves. Redefining ClojureQL also meant redefining our
ambitions: Primarily target SQL92 (PostgreSQL and MySQL), implement
primitives from RA and focus on DML. So esentially, there was no
way to continue the first ClojureQL, since the revised version was
totally different in every regard.

JAXenter: What improvements has going back to
the start brought to ClojureQL, which wouldn’t have otherwise been
reached?

Lau Jensen: Firstly and most importantly,
queries now compose giving the user great power of expression. We
only achieved this by using the primitives from RA. The elegance of
ClojureQL is the combination of Clojure and RA.

A very clean and modern implementation. The original ClojureQL
spanned about 20 files and even in its uncomplete state exceeded
3000 lines of code. The revised version spans only 5 files, where
the core.clj only amounts to 390 lines and the total of the project
is just over 1000 lines. This makes for a simpler and more easily
maintained project, which contributors are finding much easier to
work with.

Better Clojure integration. Because of advances in Clojure
itself (protocols and records) we are able to make a much tighter
integration with Clojure since we can now implement methods for
interfaces such as IDeref, IFn, IKeywordLookUp etc which previously
were not available. Right now our implementation of IDeref is a big
part of why composing queries looks so elegant.

JAXenter: And what are the next steps for the
new-look ClojureQL?

Lau Jensen: There are several things on the
roadmap, but its important to stress that this is much more than a
new-look, it’s a completely revised, rewritten and redesigned
project.

Firstly I expect the release of 1.0.0 will trigger many requests
for special SQL operators like WITH RECURSIVE etc, many of these
will likely make it into versions before 2.0.0.

Secondly I would like to simplify the way the compiler works,
possibly by internally redesigning the AST (Abstract Syntax Tree,
our internal representation of queries and relations). Right now it
is very possible for users to contribute entire compilers and plug
them into ClojureQL without needing any assistance from me. But
it’s not as easy as I would like it to be, because they have to
first read and understand how the AST works (which isn’t documented
yet) and secondly they have to read and understand the
sql92compiler.clj – the latter being the simplest of the two. If
most of the compiler work could be done in the AST, writing a
compiler would be as simple as rearranging the words of a sentence.
This is a likely milestone for 2.0.0.

Thirdly I would like to implement some high quality connection
pool management.

And finally, I would like to see more backends be contributed so
that people could enjoy all the features of ClojureQL on Sqlite,
Derby, H2 etc. Currently most things actually do work, but its the
final 10% that will eventually annoy users.

Lau Jensen is the founder of Best In Class, a Danish consultancy company which specializes in Clojure. Lau has used Clojure since its first public release and runs the blog at http://www.bestinclass.dk/blog.html which despite its narrow focus has been known to take in over 100.000 hits per month. Lau is also an instructor at the renowned Conj Labs.