Scala DSL tutorial - writing a web framework router

Recently released Play 2.0 framework brings new way of creating web services to Java community. It's nice and fun, but I dislike few components. One of them is the router with its custom routes definitions file, separate compiler and weird logic. As a Ruby developer I started wondering if it could be implemented in Scala as simple DSL. The requirements were quite simple:

statically compiled

statically typed

easy to use

extensible

it should provide (again, statically typed) reverse router

use type inference as much as possible

do not use much parentheses

Design

So basically, what is a router? It could be represented as PartialFunction[Request, Handler] and that's how it is implemented in Play. Let's get back to Play's original router for a second.

During compilation process, conf/routes file is parsed, converted to .scala files inside target/src_managed directory and then compiled to bytecode. There are two files generated: routing.scala and reverse_routing.scala. routing.scala is just one huge PartialFunction with every route described as case statement. reverse_routing.scala contains deep object structure to make calling e.g. routes.Application.index() possible. I do NOT like that.

Let's get started with How to build useful DSL in Scala.

End user interface

I have no idea what are the best practices with DSL design, I've never read a single book on that topic. This is the way that works for me.

Starting with the end result just feels natural and straightforward. First, describe what you want, then implement that - simple.

I started with very basic example, GET /foo that would route to Application.foo()

GET"/foo"Application.foo

looks quite nice. Unfortunately, it can't be implemented in Scala without using parentheses.

Note on operator syntax.

Method invocation like:

A.op(B)

can be written as

AopB

As well as:

A.op(B).opp(C)

can be written as

AopBoppC

But that syntax only applies to methods that take exactly one parameter, so: objectA method objectB.

In first DSL example (GET "/foo" Application.foo) the middle part is String object, so we can't apply this rule there. What about adding some middle words?

GETon"/foo"toApplication.foo// Or with parentheses
GET.on("/foo").to(Application.foo)

This can be compiled! GET can be an object that represents HTTP method, on is method, then "/foo" comes as parameter, then to is another method and finally Application.foo is a Function0[Handler]. Having that I made a mistake and started implementing it. Then I had to throw away huge part of code because it didn't met all requirements.

I dug deeper and came to path parameters. How to write a route that would match GET /foo/{id} and call Application.show(id)? Then I came up with an idea for:

GETon"foo"/*toApplication.show

That looked really nice. / as path separator, * as parameter placeholder and Application.show as Function1[Int, Handler]. Why this works? / is a method and * is an object. One might think that this would be equivalent to:

One last (for now) thing - reverse routing. Play's default router has a limitation that there can only be one router per action (and that sucks). If there is already route defined, why not assign that to val and user for reverse routing?

From now it will be possible to call routes.foo() or routes.show(5) and get nice paths.

The next part of this post will describe most parts of internal implementation. You might now finish and grab this library at http://github.com/teamon/play-navigator, but I strongly recommend reading about the implementation details.

Implementation

There are two hard parts: types and arity. Functions in Scala can have from 0 to 22 parameters. In Scala it is represented with Function0 to Function22. I will show later what are the consequences of this.

Here I defined two most common HTTP methods + ANY as catch-all method as objects with common parent type. Ok, now we might implement on method, but we don't know what argument it will take. Let's focus on "foo" / * part for a while.

There might be many variants of path:

"foo" / "bar" / "baz" "foo" / * / "blah" * / * / *

The good part is that we have finite set of types that comes as parts of the path. It's either static path or parameter placeholder. Said that, we can express this directly in Scala:

Here we have case class that wraps regular String and object * that has it's own type *.type. Unfortunately everything here is closely related, so I have to describe some more data structures now. As I previously said, Scala has 23 different types of functions (with different arity). I want type system to compare number of path placeholders with number of function arguments and raise an error if they do not match. To do that, we need different versions of RouteDefN. I'll reduce the number to just 3:

The reason for Self type parameter and withMethod method will be described a bit later.

Note that those RouteDefN's don't have type parameters (and I said that I want to check as much as possible during compilation). The fact is that RouteDefN knows only about it's HTTP method and path elements. It has nothing to do with handler function itself (yet).

The rule of / method is simple, if it gets Static part it stays within the same class, if it gets * it returns higher numbered route. RouteDef2 does not allow passing * since we don't have RouteDef3 class. To finish this part we need one more implicit conversion from String to Static

implicitdefstringToStatic(name:String)=Static(name)

Ok, now we have something like this:

GETonsomeRouteDef

This one is a bit tricky. How to make on method to return the exact same type as someRouteDef?

Let's get back to Method definition. It has on method that take type parameter R and calls withMethod on routeDef.

Yeah, types, types and even more types. Route0 takes RouteDef0 and function () â‡’ Out that has no parameters - simple. Route1 takes RouteDef1 and function (A) â‡’ Out, so A must be type parameter here. This syntax:

[A:PathParam:Manifest]

is just shortcut for

[A](implicitpp:PathParam[A],mf:Manifest[A])

Both PathParam[A] and Manifest[A] will be described a bit later (They will be, I promise)

By the way, as you probably already know Route2 takes RouteDef2 and function (A,B) â‡’ Out, so A and B must be type parameters here.

Here is where all compile time checks happen. Depending on RouteDefN, the to method can take only function with correct arity. And since RouteN needs some implicit parameters we have to pass them throught to method.

Here is the nice thing about all those types - we can simply add def apply to RouteN that will require correct number of arguments with correct types!

here foo is Route1[Int](RouteDef1(GET, Static("foo") :: * :: Nil), Application.show) and in the same time foo is (Int) â‡’ String

You see, typing == pure profit!

About PathMatcherN - this is yet another arity based stuff used to match request uri to correct route. Since I wanted to focus on DSL/user-side implementation I will not describe it in this post. Let's say that it is a function responsible for parsing/constructing urls.)

Only one thing left. Since all routes are type safe we need a type safe way to match paths to actions. The one way is to hardcode common types like Int or String, but that would be stupid. We already have type-aware routes, with incredibly powerful Scala's type system, why not use it to make something even more awesome?

What do we really need?

a way to parse path part (String) to our type

a way to convert path argument to String for reverse routing

That makes sense, but how to apply that to our router and how to make it extensible?

Type classes!

traitPathParam[T]{defapply(t:T):Stringdefunapply(s:String):Option[T]}

Here we define trait that provides two methods. apply is for converting T into String, and unapply is to parse String into our type T. Since parsing can fail it has return type of Option[T]

And this is the mystery about PathParam[A] in RouteN class definitions. Route classes just need to be aware of PathParam typeclass, so creating route for some type that does not have PathParam type class is forbidden by compiler.

Manifest[A] is special typeclass provided by Scala compiler for every type to provide type information at runtime. In play-navigator it is used to display list of routes with it's types (the route not found page).

Another (not included in play-navigator) might-be-useful example with java.util.UUID:

extensible - check, you can use your own types and use any Scala code to generate routes

it should provide reverse router - check, all routes are functions

use type inference as much as possible - check, no explicit type parameters on user-side

do not use much parentheses - check, it requires no parentheses

All green!

There are many other aspects of play-navigator that are not covered in this simple-yet-quite-long tutorial. If you think any part is missing some explanation or is wrong/could be made better feel free to contact me via twitter (@iteamon), teamon on #scala @ irc.freenode.net or in comments below.

You can see all the usage possibilities in README. Also the rest of implementation details such as path parsing, namespaces, integration with play etc is availible at github.

Disclaimer for HList entusiasts

I tried, didn't work as nice as I expected. The goal was to make end user api as simple as possible and with near to zero explicit type parameters.