Extensible Design with Protocols

I wrote some code this week that reinforced the power of protocols as a tool for
software design. The term “protocol” can mean many things in the world of
software. Let me clarify that I’m using protocol to mean the mechanism used by
some languages (Elixir, Clojure, etc) to achieve polymorphism. Used properly,
protocols allow you the provide users of your code with a set of standard
behavior as well as a clear contract for implementing that behavior on standard
or custom types.

In this post I’ll provide an introduction to protocols and then describe several
uses of protocols that lead to extensible design. The examples in this post are
written in Elixir but should be equally useful in other languages (after all,
Elixir credits Clojure as inspiration for its implementation of protocols).

An Introduction to Protocols

Protocols are a mechanism for achieving polymorphism. To say it more plainly,
protocols let you call a single function (or set of functions) while allowing
the subject of the function to dictate the way in which the function is
implemented. I know, I know, that’s still confusing. Let’s be more concrete and
also provide an example.

A protocol feels very similar to an interface in languages like Java. It
consists of (at least) two pieces. First, there is the definition of the
protocol itself. This is essentially a template of functions that must be
implemented for any type that the protocol can act on. Suppose we’d like to
introduce a protocol that determines if a collection is empty. Our protocol
could look something like this:

defprotocolEmptydodefempty?(collection)end

Our protocol has a single function called empty?. For us to actually use this
protocol, we must provide some implementations. Let’s do so for List and
Map.

We can check do see if our RedBlackTree is empty in the same way we check
Lists and Maps.

Empty.empty?(%RedBlackTree{...})

So What?

Why does this matter? How can we use it to write better libraries and
application code? Story time.

This week, I was working to prep my library
Scrivener for the upcoming major
release of Ecto. A pull request came in
that was unrelated to the work I was doing – someone was interested in
extending Scrivener to paginate Lists as well as Ecto queries. My goal with
Scrivener has been to keep the library small and focused. This idea had me both
excited and concerned. I wanted to provide a library where I could focus on the
functionality I needed while allowing individuals in the community to easily
extend the library for their own needs. Protocols to the rescue.

As you can see from the function @spec, this takes an Ecto.Query and a
Config and returns a Page. That Page struct contains the page’s entries as
well as information about the total number of pages, the current page number,
etc. This works. It’s great. But when the new PR came in focused on adding
pagination for Lists, I was concerned. Will I need to add some kind of pattern
matching around the first argument? Will I be stuck maintaining pagination logic
for every type of database and collection under the sun? And then it hit me:
protocols. I made a very simple change.

This single change means my library is now massively easier to extend while
giving up none of the existing functionality. The individual who asked about
adding List pagination was now free to do the work without needing to change
Scrivener itself and could release the new functionality as a companion library.
You could imagine the code looking something like this:

After including this companion library, users of Scrivener can interact with it
via the exact same API, but passing in a list instead of an Ecto query. Very
powerful indeed.

A Few Other Examples

Two other great examples of using protocols for extensible APIs are the
Poison JSON library and the built-in Enum
module.

Poison implements JSON encoding via a protocol called Poison.Encoder. The
library ships with implementations for all applicable standard types (Maps,
Lists, etc) and allows you to easily implement your own encoders for custom
types.

The Enum module in the Elixir standard library is another great example. The
functions in the Enum module are implemented in terms of the Enumerable
protocol. This means that if you implement the Enumerable protocol for your
custom collection, you get all the functionality in Enum for free.

So They’re Just Interfaces?

Protocols are very similar to interfaces with one extremely important
distinction. An interface author must rely on the consumer of their code to
implement the interface for their domain objects. A protocol author can
implement the protocol for you on any existing standard or custom types as
well as allowing you to implement the protocol for types that you deem
applicable.

This means that I was able to introduce a protocol into the Scrivener codebase
without having to ask the Ecto team to implement the protocol for me.
Fundamentally, protocols allow safe extension of existing code even if it is not
owned by the author of the protocol. Interfaces, on the other hand, force the
users of your API to implement functionality directly on their domain objects to
achieve polymorphism. It is hard to overestimate the impact of this subtle
distinction.

Consider Protocols Judiciously

I’d urge you to consider protocols as mechanism for providing extensibility in
your libraries and application code. However, it’s important to not overuse
them. The temptation to generalize early with protocols is ever-present and
problematic. Elixir itself was a victim of this, initially providing a protocol
for “dictionary like objects” (Maps, Lists, Keywords) and eventually
removing the protocol entirely and saying “just use Maps”.

Use protocols for extensibility, not for making data operations generic. When
working with a concrete type, treat the data as that type.