Give Codeship a try

Want to learn more?

An API doesn’t exist on its own. There are always two parties involved: The Client and the Server.

In Rails, our apps are often the ones acting as the server, and we typically know how to troubleshoot the issues that inevitably arise. We can tail the logs to see what the incoming request looks like (the path, headers, params), how we responded, etc… but with backend development going more and more to a microservices architecture, our Rails apps are going to more often have to act as the client in addition to the server.

More likely than not, being the client is going to mean having to make HTTP requests and parse JSON responses. More specifically, this means crafting the correct URL, HTTP headers for authentication, pagination, response format, and finally, the body of the request which is most likely either as form data (application/x-www-form-urlencoded) or as JSON.

Thankfully most of the more popular APIs, such as Twitter, Slack, and Stripe, already have client libraries written for them that encapsulate (hide) the actual lower-level HTTP request building and response parsing.

But when you have to connect to your internal services or lesser known APIs, which may not have a client library already built for them, well, we’re on our own. The rest of this article will explore some of the different options available in Ruby for dealing with HTTP and some techniques for organizing the code and troubleshooting when things go wrong or you want a bit more visibility into what is happening.

Which HTTP Clients Are Available in Ruby?

There are quite a few HTTP libraries available in the Ruby ecosystem for making HTTP requests, so I won’t be able to touch on them all. A few of the more popular ones are:

Net::HTTP: Part of the Ruby standard library. While it can accomplish everything you need it to, there is probably a reason why so many third-party HTTP libraries exist. Personally I don’t find it simple to work with and tend to opt for one of the other options in this list.

Curb: This gem provides bindings for the libcurl program (the same one you use from the command line with the curl command). It’s fast, mostly because a lot of the heavy lifting is being done outside of Ruby, and the libcurl program is fast.

HTTParty: A very popular library (22 million downloads) that wraps the Net::HTTP code, providing an easier API to work with.

HTTP: While not as popular as the other libraries, this one has been a favourite of mine recently when connecting to APIs. It provides a chainable interface for building HTTP requests and provides support for all of the usual functionality you’d expect from an HTTP library.

Excon: Another very popular library (25 million downloads) written in pure Ruby. It has a clean API and is easy to use.

I won’t show an example here of how to use each one of these libraries; you’ll be able to find a simple one on the home page of each of them. In the next section, we’ll be working through an example with the HTTP library and how we can encapsulate and organize our code.

Wrapping the HTTP client

It’s best to provide some sort of interface for interacting with these API endpoints or services and hiding all of the lower-level HTTP request and response details. Other programming shouldn’t need to be exposed to the specifics of the implementation, and that same implementation should be able to be changed/refactored without the need to change the public API.

Let’s take a look at how we might do this for an API. We’ll use the lcboapi.com API. It provides a nice little interface to access information relating to the beverages and stores of the LCBO (the governmental corporation in Ontario, Canada, responsible for the retail and distribution of alcohol for the province).

We’ll be working with four classes:

Lcbo::Products: The public interface we’ll be working with to fetch details about a specific product.

Lcbo::ProductRequest: Handles the HTTP request for a given Product ID.

Lcbo::ProductResponse: Handles the HTTP response and knows how to build an Lcbo::Product.

Lcbo::Product: The actual details of a product we fetch.

Before we dive into the implementation, let’s take a look at how to use it:

The Controller

This class acts as the public interface to fetch details about one or more LCBO Products. At this point, I’ve only implemented the fetch method, which will return the details for a single product. This class’ job is to build Requests and handle Responses; it controls the flow and knows what order the API calls must be made in.

The Request

Each Request class knows how to make a single request to the API (basically one endpoint). It constructs the HTTP request, filling in the Header details and any other information that needs to be included in that request. This is also potentially where you could introduce caching if it notices you have a local version of the response already on hand. It will respond with a Response object.

Logging Outgoing Traffic

Programming is easy when everything works first try (hint: never happens). APIs can be tricky because the HTTP requests may be expected to be formatted in a very specific way, with this header or that body. It can be tricky just by looking at the Ruby code to figure out what the final HTTP request (and response) actually looks like.

With httplog

httplog is a nice gem that monkey patches most of the main HTTP libraries to be able to log incoming and outgoing traffic to the console ($stdout or wherever you want). The information it provides may help you realize you spelled a header wrong or that something was missing.

With mitmproxy

An even more powerful tool is to use a proxy server to spy on your outgoing requests and their incoming responses. mitmproxy is a fantastic tool for gaining insight into the HTTP traffic your app produces (or your phone, or your computer).

To use mitmproxy, you’ll first need to install it and then download a certificate pem file. Once you have that, you can start your Rails server or run a Ruby script and set an ENV var to point to your custom SSL certificate.

Ensure mitmproxy is running on localhost port 8080 (default) by starting it with the command mitmproxy.

Once that is done, we can tell our HTTP library to route its traffic through mitmproxy, and then watch the information come in. mitmproxy allows you to filter the requests based on patterns, retry requests, and export them to a file to be shared or viewed later. It reminds me a little bit of the Chrome/Firefox Network inspector tab.

With the HTTP library, you will need to change this line: connect = HTTP to connect = HTTP.via('localhost', 8080)

Conclusion

Coming up with your own abstractions for communicating with APIs or microservices can be very valuable. It enables easier testing, encapsulates the low-level details, and provides reusable pieces of code. Also take some time to give httplogger and mitmproxy a try, in order to gain more insight into what the requests and responses actually look like.

I’d also like to thank @nwjsmith for introducing me to some of the libraries, techniques, and tools discussed in this article.

Subscribe via Email

Over 60,000 people from companies like Netflix, Apple, Spotify and O'Reilly are reading our articles. Subscribe to receive a weekly newsletter with articles around Continuous Integration, Docker, and software development best practices.

We promise that we won't spam you. You can unsubscribe any time.

Join the Discussion

Leave us some comments on what you think about this topic or if you like to add something.

Konstantin Ilchenko

Have you tried faraday gem? It’s a nice wrapper for http clients like NET::HTTP, Excon, Typhoeus, supports logging out of the box and has a lot of plugins. You can write same code for different clients, which is pretty convinient