Developing an API product: from REST to running containers: Part One: Overall picture

This post is the first in a series of blog posts on how we develop, deploy and operate an API as a product.

In this first post, we’ll discuss the goals we want to achieve with the proposed development and production setup. In follow up blog posts, we’ll discuss how we actually implemented the architecture, a development pipeline and production setup, and we'll discuss some of the tradeoffs and assumptions we made.

The API

Let’s start with the product we would like to build: a startups API. Three years ago, our colleague Omar Mohout started to painstakingly collect data of the Belgian startup scene. At that time, nobody really had a good view on the startup scene in Belgium, let alone how it was evolving. Omar started gathering that data, and has since published many reports and insights on it. (You can find some of his analyses here, here or here.) As a result, Omar receives every so often requests from others to have access to that data, so they can do their own analyses. We also got the question to provide the community with a map and visual tools, all based on that data set. We thought it would be a good idea to expose that data through an official REST API, instead of sending excel files back and forth.

A production setup that enforces the use of API keys, ensures that rate limits are respected, i.e. an api gateway.

Some simple dashboards and analytics on API usage.

A production setup that is robust, yet easy to maintain (did I mention that development and operations of the startup API would be done by 1 person, and that the API isn’t supposed to be a full time job?)

Implement the necessary security practices to ensure that access to our startup data is protected at all times.

To add to these requirements above, we wanted to take advantage of the state of the art in (cloud) software engineering and DevOps and make sure that we aim for:

developer parity: make the difference in setup between production and development as small as possible,making it easy to test, change and debug issues, should they arise.

git push based continuous delivery: a development workflow that allows for fast, agile iterations, and that leverages test driven development and CI/CD. Ideally, a simple git push should trigger building, testing and deploying next iteration of the application.

the whole setup should be fun and easy to work with.

Architectural view of your API

Technological Choices

With these requirements in mind, and with our own capabilities and preferences in mind, we made the following technological choices:

Implement the REST service in hapi (http://hapijs.com/), an open source webserver built in Node.js by Wallmart. Hapi comes out of the box support with for API’s and has excelent testing support through the lab library. We're used to developing in Node.js, so no learning curve here, hapi is well documented, which is a big plus.

Mongodb as the data store: REST services produce mostly JSON these days, I therefor went with mongodb, since mongodb stores JSON (well technically BSON) directly, and uses JSON for querying. Another plus is that we have some inhouse experience with small scale mongo stuff.

3scale.net as an API management platform. 3scale offers developer portals right out of the box, provides an API gateway service (both as a service as well as self hosted) and has a free plan to start with. Alternatives could be mashape or apigee.

Swagger for documenting the API. hapi has a swagger plugin that generates swagger documentation based upon the source code of the API itself, no extra work for me there (at least, that's the plan).

For deployment, Docker containers will be used, scheduled, managed and orchestrated via Rancher. Using docker containers will make the app portable, so running the app locally or in production would more or less be the same.

we’ll use Amazon Web Services as the infrastructure provider, but we decided not to go overboard with it and mainly use EC2 (virtual machines) and ELB.

Let's Encrypt for everything ssl related. It's a free, open and automatic Certificate Authority that get's a lot of traction lately, what's not to like.

Use an internal, Sirris hosted GitLab repository for our source code, linked to an Jenkins as a continuous integration and delivery server that builds, tests and deploys our code.

Conclusion

In this blog post we sketched the overall architecture we have in mind for the startups API. In the next instalment in this series, we’ll discuss the startups REST api itself, highlight some of our lessons learned when implementing the API on top of hapi and we’ll take our first steps in containerizing the application.

The carbon footprint of data centers increases further as ever more services are hosted in the cloud, leveraging the need for an efficient management of the available resources. State-of-the-art servers exhibit a linear relationship between resource ... read more

Coming Bootcamps

How to choose the right payment method and provider for your SaaS? In this seminar, Dominique Adriansens, founder of Twikey, and Nick Boucart (Sirris) will offer you practical advice on what to take into account when choosing a payment solution for y... read more

Virtualization and Cloud Computing are ranked the top 2 business driven technology priorities for CIO’s worldwide. But while delivering on agility requested by business, they must address the security risks, and assess Cloud Services against critic... read more

Get In Touch

Subscribe to Our Newsletter

* indicates required

Email Address *

First Name

Last Name

Latest Tweet

Contact Us

About Nebucom

Software as a Service is rapidly changing the way software companies bring their products and services to the market. Thanks to SaaS, companies can now reach a global audience instantly. In Belgium, many software builders are considering a transition towards a SaaS offering. Sirris, Agoria, iMinds and LSEC joined forces to help Belgian software companies make this transition, by organizing workshops and information sessions that help you understand the challenges and solutions for going Cloud and SaaS.