In this post we will develop a simple type-safe request builder. Along the way, we will:

encode domain rules and constraints at compile time with implicits;

heavily use HLists;

use existing and implement a couple of new HList constraints.

The example in this post is a simplified version of a real request builder from my Scalist project. Implementation of intentionally omitted concepts, like response parsing and effect handling, can be found there.

Prerequisites

If some of those are missing, I encourage you to come back later: there’s a lot of good articles and talks on these topics out there in the web.

The problem

We will be developing a query request builder for and API that is capable of joining together multiple logical requests and returning all results in one single response. A great example is Todoist API: you are allowed to say “Give me projects, labels and reminders”, and the API will return all of them in a single JSON response.

Such an optimization is great from performance perspective. On the other hand, it imposes difficulties for a type-safe client implementation:

you need to somehow track the list of requested resources;

return value type must reflect this list and not allow to query for something else.

Those problems are not easy to solve at compile time, but, as you might have heard, almost everything is possible with Shapeless 🙂

The task

The task here is to implement a request builder with following features:

allows to request single resource like this:builder.get[Projects].execute

Let’s code!

Model

We’ll start with some simple things, that are required though. First let’s define a model to play with. Just some case classes from task management domain:

1

2

3

4

caseclassProject(name:String)

caseclassTask(content:String,isCompleted:Boolean)

caseclassLabel(name:String,color:String)

caseclassComment(taskId:String,text:String)

What we are actually going to request are lists of domain objects. But
get[Projects] looks better than
get[List[Project]], so let’s define some type aliases. Also, this is a good place to put an instance of our builder:

1

2

3

4

5

6

7

8

9

objectapi{

typeProjects=List[Project]

typeTasks=List[Task]

typeLabels=List[Label]

typeComments=List[Comment]

valbuilder=newBuilder

}

Good. We’ll get to the
Builder later. Now let’s define

APIResource typeclass

An instance of
APIResource[R] is a marker that
R can be requested with our builder.

1

2

3

4

5

6

7

8

9

10

11

sealedtraitAPIResource[R]

objectAPIResource{

importapi._

implicitvalprojectsResource=newAPIResource[Projects]{}

implicitvaltasksResource=newAPIResource[Tasks]{}

implicitvallabelsResource=newAPIResource[Labels]{}

implicitvalcommentsResource=newAPIResource[Comments]{}

}

A couple of comments here:

In a real case typeclass body will not be empty — it’s a good place to define some specific entity-related properties. A resource identifier, that is used to create a request can be a good example.

We mark
APIResource as
sealed here, because we know all the instances upfront and don’t want library users to create new ones.

Now we’re ready to implement the
Builder.

Single resource request

Builder call chain starts with
get method — let’s create it:

1

2

3

4

5

classBuilder{

defget[R:APIResource]:SingleRequestDefinition[R]=

newSingleRequestDefinition[R]

}

Quite simple for now: we allow the method to be called only for types that have an implicit
APIResource instance in scope. It returns a request definition that can be executed or extended further with
and method.

We will solve the execution task first. A
RequestDefinition trait defines
execute method for all request definitions:

1

2

3

traitRequestDefinition[R]{

defexecute(implicitev:MockResponse[R]):R=ev.value

}

There’s the
MockResponse thing I was talking about. It allows us to avoid implementing all the real machinery that is not relevant for the topic of this post.
Almost everything about this typeclass is simple and straightforward, but there’s an interesting thing that will show up later, so I have to put the implementation here to reference it.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

sealedtraitMockResponse[R]{

defvalue:R

}

objectMockResponse{

defapply[R](v:R):MockResponse[R]=newMockResponse[R]{

defvalue:R=v

}

implicitvalprojectsResponse=MockResponse[Projects](List(

Project("project1"),

Project("project2")

))

implicitvaltasksResponse=MockResponse[Tasks](List(/*...*/))

implicitvallabelsResponse=MockResponse[Labels](List(/*...*/))

implicitvalcommentsResponse=MockResponse[Comments](List(/*...*/))

}

Ok, we’re able to execute requests! Let’s see how we can solve the chaining task with
and method.

Chaining: 2 resources request

First, we have to decide, value of what type should be returned by an executed multiple resource request. Standard
List or
Map could do the job in general, but not with our requirements — precious type information will be lost.

This is where
HList comes in handy — it’s designed to store several instances of not related types without loss of any information about those types.

Next. When implementing a type-safe API, we have to put all domain constraints into declarations, so that they’re available for the compiler. Let’s write out, what is required to join two resources in one request:

both of them must have an
APIResource instance in place, and

their types must be different

Good, we’re ready to make our first step to a multiple resource request definition:

Also, you can notice the
HList in the result type parameter. It ensures the execution result to be precisely what we requested at compile time.

Now we’re all set up to dive into the most interesting and complex problem.

Chaining: multiple resources request

Here the task is to append a new resource request
R to a list
L of already requested ones. Again, let’s first define required constraints in words:

every element of
L must have an
APIResource instance in place;

R must have an
APIResource instance too;

L must not contain duplicates. Let’s call such a list “distinct”;

L must not already contain
R . In another words, result list must be distinct too.

That’s a lot. Let’s see, what implementation we can come up with here.
We will start from the
MultipleRequestDefinition class itself:

1

2

3

4

5

6

7

8

9

10

importshapeless.ops.hlist.LiftAll

importshapeless.{::,HList}

classMultipleRequestDefinition[L<:HList](implicit

ID:IsDistinctConstraint[L],

allAR:LiftAll[APIResource,L])

extendsRequestDefinition[L]{

// ...

}

Actually, we already have everything for our first requirement. A
LiftAll typeclass from shapeless ensures that every element of an
HList has a specified typeclass instance.
In our case, implicit
allAR:LiftAll[APIResource,L] constraints
L to have an
APIResource for each element.

Implicit
ID:IsDistinctConstraint[L] will ensure that all elements of
L are different (requirement #3). There’s no
IsDistinctConstraint in shapeless 2.3.0, so we will have to implement it ourselves. We’ll come to that later.

That’s it for the class definition. Let’s move on to the
and combinator:

1

2

3

4

defand[R](implicit

NC:LNotContainsConstraintR,

AR:APIResource[R]):MultipleRequestDefinition[R::L]=

newMultipleRequestDefinition[R::L]

Requirement #2 is trivial here.
NotContainsConstraint for requirement #4 will have to be implemented by us too.

All right, so we have two HList constraints to implement. Let’s see how it’s done.

Implementing HList constraint

In general, a constraint is implemented as a typeclass, that provides instances for and only for objects, that meet the constraint.

Define a base case: implicit constraint instance for
HNil or an
HList of known length like 1 or 2.
Base case depends on the nature of the constraint and can involve additional constraints for
HList element types.

Define the inductive step: implicit function, that describes how new elements can be added to an arbitrary
HList, that already meets the constraint.

We will start with
NotContainsConstraint. The typeclass definition is quite straightforward:

1

2

3

4

5

6

importscala.annotation.implicitNotFound

importshapeless._

@implicitNotFound("Implicit not found: NotContains[${L}, ${U}]. "+

"HList already contains type ${U}")

traitNotContainsConstraint[L<:HList,U]

U is the type that
L must not contain to meet the constraint.

Let’s define the base case. Here it’s simple:

HNil doesn’t contain anything.

In general we want constraints to stay same under any circumstances, so it’s usual to define implicit rules right in the typeclass companion object:

1

2

3

4

5

objectNotContainsConstraint{

implicitdefnilNotContains[U]:NotContainsConstraint[HNil,U]=

newNotContainsConstraint[HNil,U]{}

}

Seems logical: for any type
U we state that
HNil doesn’t contain it. Heading over to inductive step, it can be expressed in words this way:

Given an HList that doesn’t contain
U, we can add any non-U element to it and get a new HList, that still doesn’t contain
U.

Let’s encode it.

1

2

3

4

5

6

7

8

9

objectNotContainsConstraint{

//...

implicitdefrecurse[L<:HList,T,U](

implicitev:LNotContainsConstraintU,

ev2:U=:!=T):NotContainsConstraint[T::L,U]=

newNotContainsConstraint[T::L,U]{}

}

Here we require an HList
L, that doesn’t contain
U (is guaranteed by implicit
ev:LNotContainsConstraintU) and a type
T, that is not equal to
U(
ev2:U=:!=T ). Given those evidences, we can state that
L::T doesn’t contain
U. We do it by supplying a new typeclass instance
newNotContainsConstraint[T::H,U]{} .

Nice, it works!
I hope it’s transparent here how implicit resolution can or can not find a constraint instance for an HList: we start with
HNil base case and go to the list head. If implicits chain is not broken along the way by a duplicate element — we get a constraint instance for the whole list.

Now we’re going to implement
IsDistinctConstraint in a similar manner. And our fresh
NotContainsConstraint is going to help us here!

Base case is quite simple:

HNil is a distinct list.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

importscala.annotation.implicitNotFound

importshapeless._

@implicitNotFound("Implicit not found: "+

"shapeless.IsDistinctConstraint[${L}]. "+

"Some elements have the same type.")

traitIsDistinctConstraint[L<:HList]extendsSerializable

objectIsDistinctConstraint{

implicitdefhnilIsDistinct=newIsDistinctConstraint[HNil]{}

//...

}

Inductive step is quite simple too:

If an HList
L is distinct and it doesn’t contain type
U, than
U::L is a distinct list too.

1

2

3

4

5

6

7

8

9

objectIsDistinctConstraint{

// ..

implicitdefhlistIsDistinct[U,L<:HList](

implicitd:IsDistinctConstraint[L],

nc:NotContainsConstraint[L,U]):IsDistinctConstraint[U::L]=

newIsDistinctConstraint[U::L]{}

}

Tests show that everything works as expected:

1

2

3

4

5

implicitly[IsDistinctConstraint[HNil]]// ok

implicitly[IsDistinctConstraint[Int::HNil]]// ok

implicitly[IsDistinctConstraint[Int::String::Boolean::HNil]]// ok

implicitly[IsDistinctConstraint[Int::Int::HNil]]// error

implicitly[IsDistinctConstraint[Long::Int::Int::HNil]]// error

Wiring everything up together

Now, when we’ve done all the preparation work, it’s time to get our builder to work.
We’ll try it out in a REPL session. Single request case goes first:

There’s no implicit mock response for our HList! We will have to add some implicits into
MockResponse companion to help it join our results:

1

2

3

4

5

6

7

8

9

10

11

objectMockResponse{

implicitvalhnilResponse=MockResponse[HNil](HNil)

implicitdefhlistResponse[R,L<:HList](

implicitR:MockResponse[R],

L:MockResponse[L]):MockResponse[R::L]=

newMockResponse[R::L]{

defvalue:R::L=R.value::L.value

}

}

After all those constraint tricks this simple typeclass extension should be transparent to you. We basically supply a
MockResponse instance for any combination of
MockResponses.

Important note: although the problem we’re solving here looks artificial, it is not — in a real world we will have to propagate requested types through all network & parsing machinery, that obtains the result. It is the only way to keep the compile-time safety.
And, similarly to our example, some tools (probably implicit) will be required for joining several results in an HList.

Finally, we get everything working! Notice the result types and how HList allows to select only requested types.

Incoming data validation is a problem that every API developer faces at some point in time. In this small article I’ll show, how shapeless tags can be used to express custom validation rules for Play JSON deserializator.

The problem

We’re going to receive this as a JSON field of incoming request and have to validate the credentials against some rules:

card number is 16 to 19 digits after removing all whitespace chars (that’s why we made it a
String, not a
Long)

cvv is 3 to 5 digits

How can we implement this? An experienced API designer would shout: “Those are not strings!”, and introduce some types. Completely valid point, a model like this:

1

2

3

caseclassCardNumber(value:String)

caseclassCVV(value:String)

caseclassCreditCard(number:CardNumber,holder:String,cvv:CVV)

would do the job. Single drawback is you’d have to implement custom serialization/deserialization for
CVV and
CardNumber to stay with simple JSON strings. By default, an instance of this type would serialize like:

1

2

3

4

5

6

7

8

9

{

"number":{

"value":"1234567812345678"

},

"holder":"Card Holder",

"cvv":{

"value":"666"

}

}

Anyway, this is still good design. But what if we want them to be
String’s? For any reason, like we’d have much cleaner code that uses this
CreditCard class.
Let’s set this as a requirement and see what we can do.

Simple strings will require defining custom
Reads for all types they belong to. Like if we have card number in some other type T, we’d have to duplicate that rule in
Reads[T]. We don’t want that.

Here is where tags come nicely into play.

Tags

As a quick intro, a tag is a marker for an existing type that creates a new type with following properties:

Values of the new type can be used as the values of original untagged type.

Values of the original type can’t be treated as tagged ones. Such code won’t compile.

So it works! 🙂
We left our strings almost untouched while not losing Play
Reads granularity.

Thanks for reading!

UPDATE. Note on Play route url binders.

A nice catch from Doug Clinton in comments: this trick won’t work with Play route parameter.
I’ll quote Doug:

The problem is that the generated routes file uses
classOf[T] when creating its invoker.
classOf expects a class type, and won’t compile when the parameter type is
@@[String,IdTag] , which does not have a runtime class.

Thank you for addition, Doug!

Acknowledgement

I want to say a big “thank you” to Denis Mikhaylov (aka @notxcain) for introducing this concept to me.