Versioning RESTful Services

I’ve talked about this in various venues and also cover it in my Pluralsight REST Fundamentals course, but the topic of how to version RESTful services has been popping up a bunch recently on some of the ASP.NET Web API discussion lists, and my friend Daniel Roth asked if I could serialize some of that presentation content into a blog post – so here goes.

First, note that while the focus here is on RESTful services and not just HTTP services, the same principles can potentially apply to HTTP services that are not fully RESTful (for example, HTTP services that do not use hypermedia as a state transition mechanism).

When talking about versioning, the most important question to ask is “what are you wanting to version?” The logical extension to this question is, “what is the contract between your service and client?” This is naturally important since the contract is the thing you want to version.

In the “old world” of Web services, the contract was the service. Service actions (and associated semantics) along with data formats and other metadata were covered by the definition of the service, which was exposed as a single URL (the service, that is – I’m grouping together all RMM L0 services here). As such, when it came to the question of how to version the service, the answer was generally pretty simple: if the contract is the service, and the service is exposed as a URL, then the solution is to version the URL. As such, you’ll see a lot of this if you browse around –

(Not trying to pick on NuGet here – it just happens to be a service API that I’m pretty familiar with at the moment)

It doesn’t take much imagination to see how unwieldy this can get after even a few iterations – especially when you’re clients are interacting with the service by generating strongly typed proxies and then pretending that there is no network (yes, I am picking on WCF here).

So how is this different for RESTful services?

Well, we should start by again asking the question, “What is the contract for a RESTful service?” The answer is, IMHO, the uniform interface. The uniform interface is made up of 4 constraints –

Identification of resources

Manipulation through representations

Self-descriptive messages

Hypermedia as the engine of application state (HATEOAS)

While all of these constraints are important to understand and take into consideration for the overall design of a RESTful system, I’m going to highlight the first 2 with respect to versioning.

First, let’s look at resources. A resource is any named information or concept – this is nice from a philosophical perspective, but maybe a little less helpful in how it enables us to think about versioning, so let’s look at it another way (as Fielding describes). A resource is a 0..n mapping between an identifier and a set of entities that changes over time. Here’s a concrete example:

I have 3 daughters: Grace, Sarah, and Abigail. However, this was not always the case – in fact, during the time period before Grace was born, the resource map of my family looked like the following:

As you can see, I had defined (in my mind) a bunch of named concepts – and at that point in time, they didn’t map to any actual entities (e.g. kids). Now, when Grace was born, the resource map looked like the following:

As you can see here, all of my named concepts map to a single entity – Grace. But what about when Sarah came along? Then the map changed to the following:

As you can now see, my “children” collection resource maps to multiple entities, and “youngest child” now maps to Sarah rather than Grace.

The point here is that the resource *concept* has not changed here – and more importantly, though the specific entity mappings have changed over time, the service has done this in a way that preserves the meaning of identified domain abstraction (e.g. children).

A representation, on the other hand, is an opaque string of bytes that is effectively a manifestation of a resource. Representations can come in many different formats and the process of selecting the best format for a given client-server interaction is called content negotiation. The self-descriptive message constraint of the uniform interface adds that the information needed to process a representation, regardless of format, is passed in the message itself.

I wanted to give this brief explanation of resources and representations because it’s important to have a clear understanding of what they are so that you can know when to version them. So let’s get back to versioning…

So, if the contract for a RESTful service is the uniform interface, then the answer to the question of how to version the service is “it depends on which constraint of the uniform interface you’re changing.” In my experience, there are 3 common ways that you can version (I’m sure there are more, but these are the 3 that I’ve come across most regularly).

Versioning Strategy 1: Adding content to a representation

In the case where you’re adding something to a representation – let’s say that you’re adding a new data field “SpendingLimit” to a customer state block as follows:

{

“Name”: “Some Customer”,

…

“SpendingLimit”:”unbounded”

}

In this case, the answer to the versioning question is to just add it. Now, this assumes that your clients will ignore what they don’t understand. If you’ve written clients in such a way that you can’t make that assumption, then you should fix your clients J – or perhaps you need to look at the next strategy…

Versioning Strategy 2: Breaking changes in a representation

In the case where you’re either removing or renaming content from an existing representation design, you will be breaking clients. This is because even if they are built to ignore what they don’t understand, by making this sort of change on the server, you’re changing what they already understand. In this case, you want to look at versioning your representation. HTTP provides a great facility for doing this using content negotiation. For example, consider the following:

GET http://localhost:8800/bugs HTTP/1.1

User-Agent: Fiddler

Host: localhost:8800

accept: text/vnd.howard.bugs.v1+html

This request gives me the following response fragment – as you can see, I’m working from an HTML base media type:

HTTP/1.1 200 OK

Content-Length: 1107

Content-Type: text/vnd.howard.bugs.v1+html

…

<div id=”links”>

<h2>Navigation</h2>

<a href=”/bugs” rel=”index”>Index</a><br/>

<a href=”/bugs/backlog” rel=”backlog”>Backlog</a><br/>

<a href=”/bugs/working” rel=”working”>Working</a><br/>

<a href=”/bugs/qa” rel=”qa”>In QA</a><br/>

<a href=”/bugs/done” rel=”done”>Done</a>

</div>

Now what if, for some reason, I needed to change the link relationship values? Remember that based on the hypermedia constraint of the uniform interface, my client needs to understand (e.g. have logic written against) those link relationship values, so renaming them would break existing clients. However, in this case, I’m not really changing the meaning of the resources or the entities that the resources map to. Therefore, I can version my representation and enable clients that know how to work with the newer version to request the newer version using the HTTP accept header as follows:

Therefore, this request:

GET http://localhost:8800/bugs HTTP/1.1

User-Agent: Fiddler

Host: localhost:8800

accept: text/vnd.howard.bugs.v2+html

Will now give me the new response format:

HTTP/1.1 200 OK

Content-Length: 1107

Content-Type: text/vnd.howard.bugs.v2+html

…

<div id=”links”>

<h2>Navigation</h2>

<a href=”/bugs” rel=”index”>Index</a><br/>

<a href=”/bugs/backlog” rel=”backlog”>Backlog</a><br/>

<a href=”/bugs/working” rel=”active”>Active</a><br/>

<a href=”/bugs/qa” rel=”testing”>Testing</a><br/>

<a href=”/bugs/done” rel=”closed”>Closed</a>

</div>

One other thing that I want to mention here – you’ve probably noticed that I’m using the language of representation design and representation versioning as opposed to content type design/versioning. This is deliberate in that many (most?) times, you’re going to design your representations completely on top of existing content types (e.g. xml/json/html/hal/etc). Without wading into the custom media type debate in this post, my point is that when I’m talking about versioning the representation here, I’m talking about versioning the domain-specific aspects of your representation that your client needs to be aware of.

Versioning a representation over an existing media type will look slightly differently from what’s shown above in that you’ll pass a standard media type identifier in the HTTP accept header along with an additional metadata element to identify your representation-specific aspects and then do content negotiation based on the combined description. There are several different ways to add the representation-specific metadata, including media type parameters and custom HTTP headers.

Versioning Strategy 3: Breaking the semantic map

In both of the prior strategies, all changes, breaking and non-breaking, have been related to the representations. This is a really good thing as it enables the resources (and more importantly, the URL resource identifiers) to remain stable over time. However, there may be occasions – hopefully rarely – when you need to break the meaning of a resource and therefore the mapping between the resource and its set of entities. As an example, as I get older and my kids grow up and leave, let’s say that I start returning my pets as children. In this case, I’ve changed the meaning of the “children” resource and thereby broken that aspect of the contract between my client and service. The solution then, in this case, is to version the resource identifier itself.

If I did a lot of this sort of resource versioning, it is very possible that I could end up with an ugly-looking URL space. But REST was never about pretty URLs and the whole point of the hypermedia constraint is that clients should not need to know how to construct those URLs in the first place – so it really doesn’t matter whether they’re pretty or ugly – to your client, they’re just strings.

So to summarize

This post ended up being longer than I had planned, so here’s the summary.

In REST, the contract between clients and services is the uniform interface

How you version depends on what part of the uniform interface you’re changing

If you’re adding only, go ahead and just add it to the representation. Your clients should ignore what they don’t understand

If you’re making a breaking change to the representation, version the representation and use content negotiation to serve the right representation version to clients

If you’re changing the meaning of the resource by changing the types of entities it maps to, version the resource identifier (e.g. URL)

In my opinion in your second approach you are confusing representation with entity structure. Representation is about the format in which you send the information or your client expects to receive it (xml, html, json, etc), in this case the resource that you are pointing to. Which means if I request the same resource but with different representation (xml, json, etc) at the same instant of time then my resources should point to the same exact entity structure and contents but with a different format, the thing that changes is the way it is dressed.

If you add a new field to the entity your resource is mapping to (like in the first approach you describe: adding SpendingLimit to customer entity), what you are really doing is creating a new version of your entity with its own set of properties (in this case with all the properties from the previous plus a new one), then you need to create a new resource (or redefine the mappings) to map to that entity as you do in your children example when having a new child.

This is similar to what Roy Fielding describes in his blog when talking about bulk operations: …If you find yourself in need of a batch operation, then most likely you just haven’t defined enough resources… (http://roy.gbiv.com/untangled/2008/rest-apis-must-be-hypertext-driven#comment-743). In the sense that changes to the information structure (in this case number of entities mapped by the resource) is about identifying those entities by mapping them to a proper resource.

Based in all that, in my opinion and experience versioning a REST API in terms of changing entity structure should be done in the URI, but at the level of resource id rather than at the level of API, which means later and not earlier in the URL.

Doing it this way allows the client to request specifically the resource they want which maps to the entity they need. Without having to mess with headers and custom media types which is really problematic when implementing in a production environment.

Must take issue with the “before Grace was born, the resource map of my family looked like the following” showing a Grace node (not subscribing to “a boy named Sue” or “the grace of God” unconditionally), e.g. AFTER birth of Sarah the Sarah node appears.

Aaron

haha, no worries. I like beer a great deal, but I live in WI. So…. there’s that.

howarddierking

clearly, I’ve done a terrible job here. I will publish it today (will go isolate myself in a coffee shop) and also owe you a beer or 2…

Aaron

Ok, last attempt. Wondering if you ever posted something somewhere that I missed? Thanks!

Cort Schaefer

I think I am finally catching an understanding of using the accept value to version the representations. And I can certainly see the value of being able to version 1 specific method vs having to accept all changes that might belong to a new service version. That said — I would really like to hear/read what you have found out in relation to NuGet API. I just took a look and didn’t see another post that seemed to cover it — hate to pester, but is it still coming?

howarddierking

Just got back from vacation last night – so between that and the forthcoming holiday, should have something in a week or so…

Aaron

Hi Howard, just wondering where you’re at with your post describing your thoughts on all of this?

Aaron, these are some really great comments. As it turns out, I’ve been thinking a lot about this recently and have been having some conversations around the general weaknesses of purely server-driven content negotiation (for many of the reasons you mentioned). Rather than noting my thoughts in comments, I’m going to just do a new post describing where my thinking is at the moment regarding conneg and what that means for versioning/metadata/linking. Much of this thinking has been coming out of the NuGet API redesign effort, so should be able to provide some concrete examples behind the concepts.

Aaron

Howard, this was a fantastic read and a nice compliment to your pluralsight video as well. Where things break down for me (and others that I know) is when we make the assumption that our consumers will be able to specify content types or version numbers in a request for a resource. Maybe you can chime in here. Drawing from your example, let’s say that your uri is as follows:

api/children/grace

Performing a GET, with accept: text/json against the above api returns a data representation of Grace. So you might get back {“Age”:10, “Height”:4, … }

Performing a GET, with accept: image/jpg against the above api, will return an image of Grace.

So me as a consumer, might write a web page to consume that api. Assuming that I want the image representation of Grace, I would write the following HTML:

Where this breaks down is that I cannot specify the accept type in the image tag, so how can I be sure that the server will default my request to image/jpg?

Further, If i wanted version 2, I cannot specify that either. So I feel like there might actually be some validity in using a version based uri scheme for these types of scenarios.

Many times, consumers of your apis will just be the average browser user, who doesn’t have the first clue about HTML, so the only way for these users to access version 2 of grace’s image would be via api/v2/children/grace/image. Certainly they’re not going to figure out a way to specify a version type or accept type in the header.

So unless I’m missing something, it just feels like we have to consider our audience when designing our restful services.

I would REALLY love your thoughts on this. I think where RESTFul design practices break down for people is around the disparity in which they are consumed.

Thanks a ton!

Matt D

“but it seems an illogical default” – it’s not an illogical default… it would be more illogical to return version 2.1 definitions.

Matt D

Hi Howard,

Just got pointed to this from Mike Wasson’s blog post. A couple of queries on implementing this.

You’d presumably want to accept some standard mime type as well? Isn’t this slightly manipulating content negotiation more than the intended usage – your response isn’t going to be a different content type as such. If going this route, wouldn’t your preferred accept type be Content-Type: text/html;apiVersion=1.1?

In your closing you stipulate (I’m paraphrasing) that if you’re changing the meaning of the resource, change the URL. In most mature production systems this would mean having both versioning strategies in play – content type negotiation and URLs. Isn’t REST, and the API strategy in general, supposed to promote inferable APIs which is a little tricky if I have to reference /api/v2/person and supply a specific content type? i.e. I’d have to provide both the v2 in the URL and a v2 in the content type otherwise the next ‘breaking change to the representation’ (bullet point 2 in closing) that is made by the api would also break my client code as the logical default content-type, if using text/html, would be the ‘latest’ version of the api? Obviously this could be fixed by not deafulting to the latest version of the api but it seems an illogical default; my point is that mixing the two seems somewhat fraught.

Thank you! I believe I have a lot to learn, but you provided a bunch of useful information.

Anonymous

so there are a few questions in here, some related to versioning, some related to HTTP, and some related to REST in general. I’ll try to address a couple points here, but very strong urge you to read the Fielding dissertation for a more comprehensive treatment of REST, as well as other architectural styles that REST compares/contrasts with.

“When you are using the accept header for versioning, aren’t you essentially extending the URI to include the header?”

Not really. At the meta level, in this example the accept header (though content negotiation can be based on any bit of metadata, not just the accept header) is used to make a statement about the version of the *representation* whereas the URL identifies a resource. Practically, this means that that set of URLs that make up a service’s resource model can remain stable across multiple versions of changes in the representation (including the addition of new capabilities).

One thing that used to keep me from fully getting the concept of versioning representations like this was when I used to equate representation with entity (e.g. I have a bug representation, a customer representation, etc..). When I redefined representation (and the representation design) to include everything that could show up in the representation (Mike Amundsen calls these ‘state blocks’ in his book http://shop.oreilly.com/product/0636920020530.do), versioning the representation made a lot more sense wrt both the value in doing it and a practical approach.

“you are now requiring the REST server to now read the HTTP headers, instead of just the request”

At the risk of sounding like I’m nitpicking, there is no such thing as a REST server – REST is just a style of distributed systems architecture. I am using an HTTP server, though, and am quite confident that it knows how to read and process HTTP headers

“it should be implemented over a protocol that can transfer a meaningful representational state, such as HTTP”

Based on my reading waaayyy too much into this statement, I think you’re conflating a couple different things together (specifically, the self-describing message constraint of REST with the definition of representation state with the definition of HTTP) – the dissertation should help here better than my comments on your comments.

‘Technically, this could be achieved in the HTTP request: ” HTTP/1.1″‘

I don’t see how you could realize the self-describing message constraint (or the stateless constraint or the caching constraint, etc.) of REST with a protocol version, resource identifier, and one piece of control flow data. The data elements in the RESTful style include resource metadata, representation metadata, and control data – which HTTP implements as headers. As I remember it, Fielding’s only issue with this design was that all of the different types of metadata were reduced to the same flat list of headers in HTTP, making it less apparent what header was what type of metadata. One comment he did make that validates something you mentioned was that he agreed that cookies were generally contrary to the principles of the RESTful style, as they reduced the visibility (e.g. self-describing messages) and in practice, tend to be used in ways that also undermine the stateless constraint.

Past that, I’m not sure that we’ve gotten carried away with using a major component of a protocol that was meant to be used as an application protocol.

When you are using the accept header for versioning, aren’t you essentially extending the URI to include the header? I don’t see much of a difference between GET /bugs/v2 and GET /bugs with a header with v2, except you are now requiring the REST server to now read the HTTP headers, instead of just the request.

At the University of Minnesota, we have recently been discussing REST architecture in the Software Engineering program. I haven’t read Roy Fielding’s dissertation on REST yet, but I think I should. In my understanding of REST, it should be implemented over a protocol that can transfer a meaningful representational state, such as HTTP. Technically, this could be achieved in the HTTP request: ” HTTP/1.1″. But I think we took it too far by using HTTP headers (authentication, cookies, and in your example accept).

What do you think about this?

Anonymous

2 separate issues (‘things’ getting unwieldy and pretending that there is no network) – sorry if the way the post was worded made it seem like I was conflating the two.

Anyways, here’s the scenario I was thinking of wrt the ‘unwieldy’ comment. Because the unit of versioning is the service in the Web services world, then most of the changes that you make to to the service (e.g. a service operation) will require you to increment the service version. Coupled with the fact that services tend to be more coarsely grained means that you can very quickly end up with myhost/api/v2*10^8 (ok, that may be a *little* dramatic). Having this many versions is confusing to clients, but more importantly can become a pain to maintain on the server (e.g. how many interfaces does your service class end up implementing over time?).

There are plenty of problems with frameworks that enable you to pretend that there is no network (e.g. work with services as if they were local objects via proxies) – but in this post, I was thinking of the difficulty that can happen when the service versions change and clients need to rev to support the new service version – even if the client in question only needs the functionality of 1 operation/resource. In this case, even though the client is only using one new capability (or 1 bug fix), the client needs to test for everything that has changed in the new version.

Denis Dollfus

Howard, I’m a bit confused about “It doesn’t take much imagination to see how unwieldy this can get after even a few iterations – especially when you’re clients are interacting with the service by generating strongly typed proxies and then pretending that there is no network”

I don’t get your point here…. “pretending there is no network”. Proxies will keep on hitting http://nuget.org/api/v1/ and everything is fine.. or what do I miss?