5 Golden Rules for Great Web API Design

Ever found yourself wondering “what were they thinking?” when integrating a web service via its API? If not, you’ve been far luckier than I have.

Any software developer knows how easy it is to let a project devolve into spaghetti code, and web APIs are no less prone to resulting in a tangled web. But it doesn’t need to be that way. In truth, it’s possible to design great web APIs that people will actually enjoy using, and that you’ll enjoy creating as well. But how? The answer to that question is what this post is all about.

Perspective

Most of the time when you’re building solutions, you’re designing for end users who are not programmers, or who are generally not technically sophisticated. You’re giving them a graphical interface and, if you’ve been doing your job right, you’ve gleaned a pretty good idea from them of what they need the interface to do.

But API development is different. You’re designing an interface for programmers, probably without even knowing who they are. And whoever they are, they will have the technical sophistication (or at least will think they have the technical sophistication) to point out every little flaw in your software. Your users are likely to be as critical of your API as you would be of theirs, and will thoroughly enjoy critiquing it.

And therein lies part of the irony, by the way. If anyone should understand how to make a web API that’s easy-to-use, it’s you. After all, you’re a software engineer just like the users of your API, so you share their perspective. Don’t you?

Well, while you certainly understand their perspective, you don’t necessarily share their perspective. When you’re developing or enhancing your API, you have the perspective of an API designer whereas they have the perspective of an API user.

API designers typically focus on questions like “What does this service need to do?” or “What does this service need to provide?”, while API users are focused on “How can I use this API to do what I need?”, or more accurately, “How can I spend the bare minimum of effort to get what I need out of this API?”.

These different questions lead to two vastly different perspectives. As a result, the necessary prerequisite to designing a great API is to shift your perspective from that of the API designer to that of the API user. In other words, continually ask yourself the questions you would naturally ask if you were your own user. Rather than thinking about what your API can do, think about the different ways it may need or want to be used and then focus on making those tasks as easy as possible for your API’s users.

While this may sound easy and obvious, it’s astounding how infrequently APIs appear to be designed this way. Think about the APIs you’ve encountered in your career. How frequently do they appear to have been designed with this perspective in mind? Web API design can be challenging.

So with that said, let’s proceed and talk about the 5 Golden Rules for Designing a Great Web API, namely:

Rule 1: Documentation

Documentation. Yes, I’m starting here.

Do you hate documentation? Well, I can empathize, but put on your “user perspective” hat and I’ll bet that the one thing you hate more than having to write documentation is having to try to use an undocumented API. I rest my case.

The bottom line is that, if you want anyone to use your API, documentation is essential. You’ve simply got to get this right. It’s the first thing users will see, so in some ways it’s like the gift wrap. Present well, and people are more likely to use your API and put up with any idiosyncrasies.

So how do we write good documentation?

The relatively easy part is documenting the API methods themselves; i.e., example requests and responses, along with descriptions of each of the elements in both. Fortunately, there are an increasing number of software tools that facilitate and simplify the task of generating documentation. Or you can write something yourself that introspects your API, endpoints, and functions, and generates the corresponding documentation for you.

But what separates great documentation from adequate documentation is the inclusion of usage examples and, ideally, tutorials. This is what helps the user understand your API and where to start. It orients them and helps them load your API into their brain.

For example, if the developers of Twilio were to list out every class, every method, and every possible response to their API, but didn’t bother to mention that you can send an SMS, track a call, or buy a phone number through their API, it would take a really long time for the API user to find that information and understand it cohesively. Can you imagine sorting through a giant tree of classes and methods without any insight into what they were used for, other than their name? Sounds terrible right? But that’s exactly what so many API providers do, thereby leaving their APIs opaque to anybody but themselves. The Rackspace CloudFiles developer and API guide is one such example; it’s difficult to get your bearings unless you already understand what they’re doing and what they’re providing.

So write concise tutorials that help get the developer up and running quickly, with at least a skeleton of what they’re trying to do, and then point them in the direction of the more detailed, fully-documented list of functionality so they can expand on what they have.

Once you’re done with your documentation, be sure to validate that it makes sense to people other than yourself. Send it out to other developers in your network, give them no instruction other than pointing them to the documentation, and ask them to follow a tutorial or build something really basic in about 15 minutes. If they can’t have a basic integration with your API in 15 minutes, you have more work to do.

For some noteworthy examples of excellent and detailed documentation, check out Twilio, Django, and MailChimp. None of these products are necessarily the best in their markets (although they are all good products), yet they do distinguish themeselves by providing some of the best documentation within their markets, which has certainly facilitated their wide acceptance and market share.

Rule 2: Stability and Consistency

If you’ve ever used Facebook’s API, you know how often they deprecate and completely rewrite their APIs. No matter how much you respect their hacker culture, or their product, their’s is not a developer-friendly perspective. The reason they are still successful is because they have a billion users, not because their API is great.

But you probably don’t have the luxury of such a mammoth user base and market share, so you’re going to need have a much less volatile API, keeping old versions running and supported for quite a long period of time. Maybe even years. So toward that end, here are some tips and tricks.

Let’s say, for example, that your API is accessible via the URL http://myapisite.com/api/widgets and provides its response in JSON format. While this may seem fine at first blush, what happens when you need to modify the format of the JSON response? Everyone that’s already integrated with you is going to break. Oops.

So do some planning ahead, and version your API from the outset, explicitly incorporating a version number into the URL (e.g., http://myapisite.com/api/widgets?version=1 or http://myapisite.com/api/widgets/v1) so that people can rely on version 1 working and can upgrade to any subsequent version when they’re ready to do so. If you need to phase out a prior version at some point, go ahead, but give plenty of notice and offer some sort of transition plan.

A good URL scheme will include major versions in the URL. Any change to the output format or supported data types should result in bumping up to a new major version. Generally, it’s acceptable to keep the same version if all you are doing is adding keys or nodes to your output, but to be on the safe side, any time the output changes, bump a version.

In addition to being stable over time, APIs need to be internally consistent. I’ve seen many APIs that change parameter names or methods of POSTing data, depending on the endpoint that is being used. Instead, you should handle common parameters globally within your API and use inheritance or a shared architecture to reuse the same naming conventions and data handling consistently throughout your API.

Finally, you need to record and publish a changelog to show differences between versions of your API so that users know exactly how to upgrade.

Rule 3: Flexibility

Garbage in, garbage out (GIGO) is a well known mantra to most programmers. As applied to web API design, this guiding principle tends to dictate a fairly rigid approach to request validation. Sounds great, right? No mess, no problem.

Yet as with everything, there needs to be some balance. As it is not possible to anticipate every way that users will want to employ your service, and since not every client platform is consistent (i.e., not every platform has very good JSON support, a decent OAuth library, etc.), it’s good to have at least some degree of flexibility or tolerance with regard to your input and output constraints.

For example, many APIs will support a variety of output formats, like JSON, YAML, XML, et. al., but will only support specifying the format in the URL itself. In the spirit of remaining flexible, you could allow this to also be specified in the URL (e.g., /api/v1/widgets.json), or you might also read and recognize an Accept: application/json HTTP header, or support a querystring variable such as ?format=JSON, and so on.

And while we’re at it, why not allow for the format specified to be case-insensitive, so the user could specify ?format=json as well? That’s a classic example of a way to alleviate unnecessary frustration for the user of your API.

Another example is allowing for different ways of inputting variables. So, just like you have a variety of output formats, allow for a variety of input formats as well (e.g., plain POST variables, JSON, XML, etc.). You should at least be supporting standard POST variables, and many modern applications support JSON as well, so those two are a good place to start.

The point here is that you shouldn’t assume that everyone shares your technical preferences. With a little research into how other APIs work, and through dialog with other developers, you can glean other valuable alternatives that are useful and include them in your API.

Rule 4: Security

Security is obviously one of the most important things to build into your web service, but so many developers make it ridiculously hard to use. As the API provider, you should be offering usable examples of how to authenticate and authorize when accessing your API. This should not be a difficult issue that an end user spends hours working on. Make it your goal that they either don’t have to write any code, or it takes them less than 5 minutes to write it.

For most APIs, I prefer a simple token-based authentication, where the token is a random hash assigned to the user and they can reset it at any point if it has been stolen. Allow the token to be passed in through POST or an HTTP header. For example, the user could (and should) send an SHA-1 token as a POST variable, or as a header in a format such as “Authorization: da39a3ee5e6b4b0d3255bfef95601890afd80709”.

Also, choose a secure token, not a short numeric identifier. Something irreversible is best. For example, it’s relatively simple to just generate out an SHA token during user creation and store it in the database. Then, you can simply query your database for any users matching that token. You could also do a token generated with a unique identifier and a salt value, something like SHA(User.ID + "abcd123"), and then query for any user that matches; e.g., where TokenFromPost = SHA(User.ID + "abcd123").

Another very good option is OAuth 2 + SSL. You should be using SSL anyway, but OAuth 2 is reasonably simple to implement on the server side, and libraries are available for many common programming languages.

If the API you have made is supposed to be accessible on a public website via JavaScript, you need to also make sure you validate a list of URLs per-account for the token. That way, nobody can go inspect the calls to your API, steal the token from your user, and go use it for themselves.

Here are some other important things to keep in mind:

Whitelisting Functionality. APIs generally allow you to do basic create, read, update, and delete operations on data. But you don’t want to allow these operations for every entity, so make sure each has a whitelist of allowable actions. Make sure, for example, that only authorized users can run commands like /user/delete/<id>. Similarly, all useful headers that are sent in the user’s request need to be validated against a whitelist as well. If you are allowing Content-type headers, verify that whatever the user sends in actually matches a whilelist of supported content types. If it doesn’t, then send back an error message such as a 406 Not Acceptable response. Whitelisting is important as a lot of APIs are automatically generated, or use a blacklist instead, which means you have to be explicit about what you don’t want. However, the golden rule of security is to start with absolutely nothing, and only explicitly allow what you do want.

Validate access to resources. In every request, you need to verify that a user is in fact allowed access to the specific item they are referencing. So, if you have an endpoint to view a user’s credit card details (e.g., /account/card/view/152423), be sure that the ID “152423” is referencing a resource that the user really is authorized to access.

Validate all input. All input from a user needs to be securely parsed, preferably using a well-known library if you are using complicated input like XML or JSON. Don’t build your own parser, or you’re in for a world of hurt.

Rule 5: Ease Of Adoption

This is really the most important rule in the bunch, and builds on all the others. As I mentioned during the documentation rule, try this out with people that are new to your API. Make sure that they can get up and running with at least a basic implementation of your API, even if it’s just following a tutorial, within a few minutes. I think 15 minutes is a good goal.

Here are some specific recommendations to ease and facilitate adoption of your API:

Make sure people can actually use your API and that it works the first time, every time. Have new people try to implement your API occasionally to verify that it’s not confusing in some way that you’ve become immune to.

Keep it simple. Don’t do any fancy authentication. Don’t do some crazy custom URL scheme. Don’t reinvent SOAP, or JSON, or REST, or anything. Use all the tools you can that have already been implemented and are widely accepted, so that developers only have to learn your API, not your API + 10 obscure new technologies.

Simplify any necessary signup. If you are not developing an open source API, or if there is a signup process of any sort, make sure that upon signup, a user is very quickly directed to a tutorial. And make the signup process completely automated without any need for human interaction on your part.

Provide excellent support. A big barrier to adoption is lack of support. How will you handle and respond to a bug report? What about unclear documentation? An unsophisticated user? Forums, bug trackers, and email support are fantastic starts, but do make sure that when someone posts a bug, you really address it. Nobody wants to see a ghost town forum or a giant list of bugs that haven’t been addressed.

Web API Wrap-up

Web services and their APIs abound. Unfortunately, the vast majority are difficult to use. Reasons range from poor design, to lack of documentation, to volatility, to unresolved bugs, or, in some cases, all of the above.

Following the guidance in this post will help ensure that your web API is clean, well-documented, and easy-to-use. Such APIs are truly rare and are therefore that much more likely to be widely adopted and used.

About the author

Jordan is a top-notch architect, developer, sysadmin, and entrepreneur with the passion and experience to help businesses solve complex problems. He is an expert full-stack developer, bringing projects from concept to completion, with a proven track record of delivering powerful, stable, and comprehensive solutions. [click to continue...]

Comments

Pedro Werneck

This is sensible and reasonable advice for a quick-and-dirty RPC-style API, but there are a few anti-patterns when considering long-term evolution. I wouldn't adopt these rules for anything that should stay online for more than a year or so.

Jordan Ambra

I'd love to have the chance to respond to your concerns. Would you please be more specific?

Ron Barak

Great and clear article. Thanks.
One erratum:
As a result, the ↝the↜ necessary prerequisite

Jordan Ambra

Thanks for finding the bugs Ron :)

Pedro Werneck

What concerns me the most is that API versioning is presented as a rule for consistency and stability, while it implies the exact opposite. When you put a version number in the URI, you're implying that you will break consistency and backwards compatibility at some point. API versioning should be the last resort to deal with issues involving long-term evolution and backwards compatibility, not the first. The Web itself isn't versioned, why Web APIs should be? It may be used to ease server-side implementation, but not to drive interaction with clients.
For long term evolution, it's much better to let the API itself provides URIs, and let them drive the interaction, and documentation should reflect that. The examples given put too much effort on describing URIs, while they should describe link relations and keep URI semantics transparent. You don't care about the semantics of an URI when you click on a link, you only care what the relationship is, where it gets you to.
Also, it suggests API versioning as a solution for changing formats and data types, but that should be done by versioning the media-type itself, which brings us to the rule about flexibility. It considers appending a file extension or query parameter to the URI on par with the Accept header, while the latter is a lot more rich and flexible. By following rules 2 and 3, when you change the format of a single resource, you're forced to bump the version of the whole API. That's a compatibility nightmare. Instead, clients should request the version they want on a resource-basis, with the Accept header. If you have issues with creating custom media-types, you may use a custom header, but there's no need to tie the datatype version to the API version.
Again, this is what concerns me when considering long-term evolution of an API. On attending short-term demands, being consistent is more important than having a design informed by long-term vision, and these rules are reasonable enough for that.

Jordan Ambra

OK, do you have any concerns other than versioning of the API via URIs?
Here's the issue that we are trying to solve: someone implements an API client now in 2014, and we're doing the best we can to provide them with an interface that works well and is the absolute best that we know how to do. But in 2015, our business changes and we now support additional functionality and one of our data types changes to reflect that. We'll say we went from a string value to a list of string values in both the API input and API output.
So, how do we address that? Never allow for string values in an API in the first place, and only do lists and make everyone support that in their clients? It seems reasonable to me that instead, you would recognize that change is most definitely going to happen, and accept it, and build the ability to adapt to change very visible and very actively into your API.
One such way to handle change is to version your API whenever backwards compatibility becomes impossible. An easily visible way to do that is to put it in the URI or as a header.
As for versioning per-resource, I totally agree that's an acceptable way of doing it. It's not necessarily the best way, because it relies on the client to figure out the versions everywhere and you end up with version explosion, whereas if you bump a version on the server, you can simply fall back to a previous version rather than implement a copy. Neither way is perfect, and you can end up with the same issues with compatibility nightmare and version explosion with both solutions.
Either way, thank you for your thoughts. They are certainly worth considering during a buildout.

matheusml

I don't like this approach with api versioning: "http://myapisite.com/api/widgets/v1"
I'd rather put the version right after 'api', like this:
"http://myapisite.com/api/v1/widgets/"
Because, probably the user have a BASE_URL constant somewhere,
which is "http://myapisite.com/api/v1/", and to change the api version,
he just simply need to change that.

Pedro Werneck

As I said before, I wouldn't recommend adopting any of these rules for a long-term project. The versioning of API via URIs is a main concern here, since its usage reflects more on other shortcomings, like the API interaction being driven by out-of-band information, the generic media-types, etc.
The problem you describe is a simple change in media-type. That's addressable by proper usage of the Accept and Content-Type headers. By using API versioning to address that problem you're not only trying to solve an issue that's already solved by the HTTP protocol, you're also losing a lot of value you'd get from the proper usage of URIs in the first place, since you're conflating identification with interaction.
I recommend reading this: https://www.mnot.net/blog/2011/10/25/web_api_versioning_smackdown
The client doesn't necessarily need to know which versions he needs everywhere. When you enter a website like Gmail, does it make a difference for you what version it is? You hit your homepage, where you follow links that lead you to the version you should be using. Since the server is supposed to drive the client's interaction through links, it can direct the client to the versions it should be using.
Even when that's not the case, the client knowing which versions he needs isn't a problem at all, because that's the only out-of-band information it should have anyway. This simply can't lead to the same issues with compatibility one would have by using URI versioning, since in this case you don't have identification coupled to interaction, which is the cause of problems in that approach.
It's simple to do things in easily visible ways, but that does't always translate very well into long-term evolvability. That's another ballpark.

Jordan Ambra

You could consider either approach. Front-loading the API version before all the resources means you are migrating the entire API at once. Putting it after the resource means the clients can update just how they interface with a single resource and leave the rest alone.
Either way is valid, depending on how large your API is.

Oto Brglez

Couple of months ago I came across a free book from Brian Mulloy from apigee titled "Web API Design - Crafting Interfaces that Developers Love". He describes what I think are the best good and bad patterns when designing / building APIs. Versioning, Error handling, formats, authentication, ect.. Book covers everything what I think that good API developer should know and like said its free. http://info.apigee.com/Portals/62317/docs/web%20api.pdf P.s.: Its language agnostic, so don't be afraid. :)

swampwiz0

I think that the problem in the development world is that there are far, far too many APIs out there, with a lot of employers officially wanting their applicants to know all of them (whether this is real or a way to shill for H1Bs, who knows.) There really should be just a few really good ones out there, with the others being relegated to the trash bin.

Rachel Gillevet

Hey Jordan,
Great post. I just wanted to reach out and let you know that I included it in my roundup of the month's best web design/development content. http://www.wiredtree.com/blog/the-monthly-round-up-junes-best-web-designdevelopment-cms-and-security-content/ Thanks for the valuable resource, our readers will appreciate it. Cheers!
Rachel

dongguangming

great article.i feel very useful.

Gabo Lugo

Thanks!

Shaiju Janardhanan

Great article!!

acpmasquerade

Using semantics and HTTP built-in features are good. However, I feel like versioning decreases documentation overhead and lowers the learning curve.

karthik G

What is the good practice ? to have individual token for each endpoint ? or to have one token for all endpoints under a domain?

Janynne Gomes

Great post!!!

John Roca

Hi I am John Roca. I'm starting to develop my own API for my site. Thank you for this post. This post gives me a lot of knowledge for my API development and implementation. Hope you continue to help our community with your post! Great Job!

Jordan is a top-notch architect, developer, sysadmin, and entrepreneur with the passion and experience to help businesses solve complex problems. He is an expert full-stack developer, bringing projects from concept to completion, with a proven track record of delivering powerful, stable, and comprehensive solutions.