RESTful Conversations

MAR 15, 2015

My blog posts related to IT strategy, enterprise architecture, digital transformation,
and cloud have moved to a new home: ArchitectElevator.com.

ABOUT ME

Hi, I am Gregor Hohpe, co-author of the book Enterprise Integration Patterns. I like to work on and write about asynchronous messaging systems, service-oriented architectures, and all sorts of
enterprise computing and architecture topics. I am also a technical director in Google
Cloud's Office of the CTO.

As indicated a good while ago I spent some time thinking about patterns that instead of following a message through
multiple systems, looks at the message exchange over time between a (mostly) fixed
set of systems. I call these message exchanges "conversations", partly because it
avoids collision with terms like "orchestration" or "choreography" and partly because
it makes for a nice bridge between communicating systems and communicating humans.
"interactions" would have been another good name, but I found it a tad too generic,
even though I did register interactionpatterns.com.

Vocabulary aside, my writing efforts on EIP Vol 2 fell very much asleep around 2008,
first due to lack of time, but then also to my fear that with the rise of REST, stateful
conversations between systems would step into the background as most integration problems
are now solved with a simple POST or GET. A presentation and conversation at Stuttgart
University's IAAS (not IaaS!) PhD Seminar setup by Frank Leymann enlightened me that this is not at all true: systems following the REST architectural
style very much engage in conversations.

To REST or not to REST

I am probably the last person to admonish everyone that neither using XML or JSON
nor HTTP verbs alone makes an interface "ReSTful" in the intended sense. First up,
REST is an architecture style, not a specific technology or encoding. Mary Shaw and Dave Garlan defined an architectural style as "defining a vocabulary of components and connector
types, and a set of constraints on how they can be combined". [side note: in my ranking
of proudest moments, being listed as a SATURN 2015 keynote speaker alongside Mary Shaw vies for 1st place with Grady Booch's mention of our book at OOPSLA 2005]. Second, ReSTfulness is not a binary property. Leonard Richardson's REST Maturity Model describes multiple levels of maturity that make an API actually ReSTful. Martin Fowler
wrote a nice explanation of this model, including a halo to be achieved in the REST Nirvana, the mismatching religious metaphors
being my contribution. Not quite coincidentally, simply sending POX over HTTP gets
you only maturity level 0, the lowest possible one.

Application State

The connection between REST and conversations starts at level 3 of the model, where
we encounter what must be one of the worse acronyms in recent IT history: HATEOAS, which is supposed to stand for "Hypertext as the Engine of Application State". I
could not actually make out who blessed us with this acronymial masterpiece -- it
does not appear in Roy Fielding's thesis or blog. If it was up to me, I would change its meaning to Hideous Acronym That Elaborates On Application State.

The unlucky name aside, the idea behind level 3 of the maturity model is that application
state is expressed through hypermedia (read URI's). In reality this means that a service
response describes possible follow-on on resources to acted upon by the client in
form of embedded URI's, aka "links". Essentially, this set of URI's defines the legal
state transitions of the application (and thus the allowed states) to the client as
Roy Fielding explained in a somewhat emotional post from 2008: "From that point on, all application state transitions must be driven by client selection
of server-provided choices that are present in the received representations or implied
by the user's manipulation of those representations". As the application state transitions drive the conversation a client can engage
in, this representation of application state translates fairly directly into a description
of allowed conversations. Compared to other ways of describing conversations, such
as choreography or orchestration, describing conversations with links embedded in
response messages is a dynamic and relatively loosely coupled mechanism. Speaking
of loose coupling, my favorite quote from Roy's post is "There is so much coupling on display that it should be given an X rating" -- I'll make sure to borrow that some time.

After the customer has just ordered a drink the next action he or she can take is
to submit a payment. Correspondingly, a URI for the payment resource is included in
the response message under the "next" element. The semantics of "next" URI's are indicated
by the "rel" attribute http://starbucks.example.org/payment, meaning client and server have to share a common vocabulary and semantics of resources
such as "payment". The set of the resources and associated actions is also the vocabulary
for the conversation to be had: you can place an order, make a payment etc. The links
do not show, though, that you can also potentially update your order -- this is handled
by the HTTP protocol via the OPTIONS verb or the expect header.

Conversation State

The inclusion of "next" links in responses defines the set of valid conversations
from the perspective of the service provider. Unlike choreography and orchestration approaches, the conversations are dynamically defined at the time of interaction as opposed
to an a priori definition. This approach reduces coupling (and thus avoids an X rating),
however, it does not allow the upfront creation of client workflows that model allowed
conversations. Instead, the client has to start interacting with the service and see
what it is being offered to do next. For example, constraints that define valid conversations,
such as whether a customer can pick up a drink before payment is made or whether the
drink is started to be made before a payment is received, are represented as a set
of links embedded in multiple response messages and not known upfront. This means
that clients cannot make strong assumptions on message order -- the service is in
the driver's seat. Once again, reduced coupling means fewer assumptions, but you (or your code) have to be more
flexible, which can bring complexity or at least a different way of thinking about
process flow. If (slightly) stronger coupling is allowed, a REST service could describe
legal transitions upfront through a special media type. The transitions are then mapped
to concrete resources via the URI's embedded in next elements.

This relatively simple example also makes it nice and clear that REST services are
indeed engaged in conversations just like traditional web services, and maybe even
more so because the individual interactions tend to be simpler, causing services and
clients to engage in longer conversations to accomplish their ultimate goal.

Conversation Patterns for REST Services

Having learned that REST services engage in conversations we should hope that some
of the Conversation Patterns I started to document a while ago still have relevance. In fact, one can assume that many new patterns are waiting
to be "mined" from the magnitude of REST services out there, from OpenStack to the
Amazon API's. This should motivate me to pick the topic of conversation patterns
back up and make a push towards EIP 2.

Florian Haupt, a PhD student at IAAS Stuttgart presented conversation pattern candidates at the PhD Seminar and documented them
in his upcoming paper A conversation based approach for modeling REST APIs, which he is presenting at WICSA 2015 in May. His interest in conversations stems from his work in model-driven development
for REST services. Modeling languages are particularly useful if they include higher-level
constructs for frequently recurring solutions. Such solutions are not patterns per se, but could be an indicator that there are related patterns to be harvested. His conversational
constructs originated from analyzing REST services, but, like all good patterns, are
not limited to REST services alone.

Florian shared the example of a long-running tasks, which in the REST world defines three resources: a manager, a task, and a result for the client to interact with. This could well be an implementation strategy to
a polling or callback conversational pattern. A second example, a Try-Confirm-Cancel sequence, is indeed a common conversation pattern that is as useful with REST services
as with any distributed system that lacks global transactions. In my paper on Conversation Patterns I deal with these conversations under the chapter "Reaching Agreement", which is
still incomplete.

Conversation Patterns Quo Vadis?

The realization that conversation patterns are as relevant as ever begs the questions
where I am with EIP 2: Conversation Patterns. I did post the paper I workshopped at EuroPLoP 2008 on-line, but I think I really need two things: a good vehicle to share work in progress
and a community to provide input and feedback to give me energy. For the former, you
may have noticed small changes to the Web site's navigation and the renaming of the
pattern catalog to "Messaging Patterns" in preparation of making room for another,
work-in-progress, catalog. Alas, the site is still being built by Ant 1.7 and XSLT,
which may soon find a place in the Computer History Museum, but is not a great tool for content management or collaboration. For EIP we used
a simple mailing list and archives for discussion because our Wiki was flooded by
spam. Maybe mailing lists are becoming "retro" and actually work? There is no writing
on conversation patterns without engaging in conversations!

Gregor is a technical director in Google Cloud's Office of the CTO. He is a frequent
speaker on asynchronous messaging and service-oriented architectures and co-authored
Enterprise Integration Patterns (Addison-Wesley). His mission is to make integration and distributed system development
easier by harvesting common patterns and best practices across many different technologies.www.eaipatterns.com