Scala @ Scale, Part 2: Compose Yourself!

Function composition is an extremely useful tool for creating modular, testable programs. One of the most natural applications of functional composition that I’ve found is creating a lightweight, composable HTTP request builder, and with that objective in focus, this post will equip you with the tools you need to simplify your HTTP clients.

In a microservice-based environment, distributed services need to communicate with each other over the network. Many times the protocol of choice is HTTP. What do you do when you need to expose an API for a given service? Create a client of course! Clients are great and should be used, but the issue with clients is that adding new endpoints or changing existing methods can be both tedious and error prone. It’s tough to remember exactly how to accomplish these tasks every time you need to hop into the client to make changes. Ever had to skim through a bunch of code just to remember how to add a new method to the client? Questions that frequently come up are:

I forget how to create an HTTP request… how do I do that again???

What’s the base set of information required by the API for this specific service?

What method should I copy and paste to get exactly what I want?

These issues arise due to your HTTP client not having a composable and expressive DSL (domain specific language) that feels natural to use. That’s the problem that we’re here to solve today. What if I told you that we can build a DSL that feels so natural that you’ll (probably) never again need to copy/paste code every time you add a new endpoint/method to the client? By the end of this post, we will have a set of functions that allows us to express HTTP requests like this:

How nice is that?? It is VERY easy to remember because it’s so natural to use. Building a DSL like this requires no external libraries either. Bonus!

Usual Disclaimer: This series assumes knowledge of Scala. Most readers without Scala knowledge should be able to follow a lot of the code, but some nuances may be missed.

Let’s jump right in. Function composition is all about being able to compose functions together to accomplish a task. Building HTTP requests is a great use of this since you are usually working (or “building”) on a single request object. The request object has many fields that allow you to specify how the request should be made and what data needs to be sent in that request.

Let’s first define our HttpRequest model. We’re using a custom, overly simplistic one for purposes of demonstration, but the actual model can come from anything (most likely the library you’re using to make the requests like Akka-Http or http4s):

Now that we have our super-sophisticated HttpRequest class, let’s define the types to be used for request composition:

// The type for building requests. RequestBuilder is simply
// a function that takes an HttpRequest and returns a
// modified HttpRequest. Modifications can be whatever
// you want as long as they adhere to this definition.
type RequestBuilder = HttpRequest => HttpRequest
// The type used to signify lifting of values into RequestBuilder (see usage below)
type HttpUnit = () => RequestBuilder

OK so what’s going on here. First we define a type for RequestBuilder. This defines the operation for our composable functions. Each one will receive an HttpRequest object, modify it however we want, and return a copy of the modified request. Already sounds “chain-able” doesn’t it? Let’s get to the fun stuff. Next we’re going to implement the functions we can use to compose and build our HTTP requests:

Sorry for the big code dump, but it’s a necessary evil to give context to what’s happening. The HttpRequestBuilder object contains various functions for creating/modifying http requests. The entry points for new requests are the HTTP verb methods (GET/POST/PUT/DELETE). We have a private lift method defined which allows us to — you guessed it — “lift” the initial request values into the context of a RequestBuilder. Once we have a RequestBuilder, we can use the various modifier methods to work on the request.

At this point, we can actually generate a function that will resolve to an HttpRequest when applied. Let’s check it out:

Nice! So we see that it’s building the request the way we want, but we’re obviously missing an important part to all of this: Actually executing the request! So let’s build a mock HTTP executor that will allow us to “execute” that lovely request we’ve built:

// Very simple mock Http Executor interface. This can be whatever you want.
trait HttpExecutor {
/**
* The exec method takes an HttpRequest and a mapper function. The
* mapper function simply takes the response string and can do whatever
* it wants to it (e.g. returning a case class from the response)
*/
def exec[A](httpRequest: HttpRequest)(f: String => A): Future[A]
}
// Implementation of the executor
object BasicHttpExecutor extends HttpExecutor {
/**
* This exec method takes a fully formed HttpRequest and a transformer function
* and "executes" it to return a result (as a Future).
* We built additional logic in for testing. If the HttpRequest has a mock status code
* set that is not 200, we fail the future with an error.
*
* On success we just format the request data in a friendly, human readable format.
* In a real-world scenario this would be some kind of actual response like
* JSON or XML that would be passed to a parser.
*/
override def exec[A](httpRequest: HttpRequest)(f: String => A): Future[A] = httpRequest.headers.get("x-mock-status-code") match {
case None | Some("200") => Future.successful(f(
s"""
Method: ${httpRequest.method}
Endpoint: ${httpRequest.url}
Data: ${if(httpRequest.body.isEmpty) "N/A" else httpRequest.body}
Query Params: ${httpRequest.query}
Headers: ${httpRequest.headers}
""".stripMargin))
case Some(code) => Future.failed(new RuntimeException(s"Received error from API: $code"))
}
}

Again this is just a mock executor service. In real life, you’d be using an executor from whatever HTTP lib you’ve chosen (Akka, http4s, etc.). It doesn’t matter which one you choose; the overall implementation is the same. In our mock executor, we check a special HTTP header called “x-mock-status-code.” If it’s not defined or set to 200, we consider the request a success. In that case, we just return a successful future with the serialized http request information. If the header is set to a non 200 code, we fail the future and return it with an exception.

Using our example from above, we know how to compose the functions together, and then apply it to receive the HTTP request. So now we could do something like this:

That will in fact work as you’d expect, but it’s kinda ugly and unintuitive. I’d love a way for us to just call exec() directly on the result of our RequestBuilder. When using basic types (instead of instances of classes), we can augment or “pimp” out our types to add an exec method using implicit classes. Implicit classes are outside the scope of this article, but a quick google search will provide all the info you want. Here’s how we ultimately want to use the request builder:

Oh that’s nice! Instead of having to explicitly call exec on the HttpExecutor, we can just pimp out the RequestBuilder type to be able to handle it for us. Here’s what the implementation looks like for that:

And there we have it! We built our request using RequestBuilder and friends, mocked out an executor service for it, and created an implicit class to execute any instance of RequestBuilder. At this point you have enough knowledge and code to venture on and create your own request builder to make client generation great again! If you’re hungry for more, read on for an actual client implementation using RequestBuilder.

Look at you, still hungry for more client generation goodness! OK. Well we’re going to create a client for our awesome “Foo Service.” We’ll make an aptly named “FooClient” for it so that all our other services can use Foo as much as they want. Let’s first build out some of the case classes we’ll need to handle authentication and responses:

Our Foo service expects credentials on every request to identify who’s using it (set as headers on the HTTP request. We also create a FooHost case class to hold connection information as well as a response class for saying hello to the user. Let’s get into the actual client now:

Really simple client. Notice the addCredentials method. We are able to reuse the more powerful addHeaders function to extend the base RequestBuilder for the specific use case in FooClient. Even though this is a very simple example, you should already be able to see how extensible this is. Let’s see how this client is used!

Boom! And there we have it. A fully fledged client using the RequestBuilder types. If you got this far, you now have the tools to build a composable, type-safe, expressive DSL for building up HTTP requests regardless of what HTTP library you’re using. Stay tuned for Part 3 for more Scala @ Scale goodness!

For More Scala-Related Articles . . .

If you’re interested in other Scala-related articles based on the experiences of Threat Stack developers, have a look at these:

As a senior platform engineer at Threat Stack, Dave is responsible for leading the design and implementation of solutions that scale to handle the massive amounts of data received by Threat Stack Cloud Security Platform (CSP)™. He utilizes his knowledge and experience of large-scale architectures to create systems that are both flexible and stable in the presence of ever-changing data and requirements.