Menu

Golang basics - fetch JSON from an API

This is a recipe in Golang for making a "GET" request over HTTP to an API on the Internet. We will be querying an endpoint provided for free that tells us how many astronauts are currently in space and what their names are.

If you're already familiar with Go, then you may want to skip the first section.

Pre-requisites

If you want the change to stick, add the line to .bash_rc or .profile.

In Go 1.8 onwards this step will not be needed and is automatic.

We are going to use the Golang standard library which has a number of useful components for HTTP and JSON.

A blank template

I have had good experiences using nano and vim, but Visual Studio Code is probably the easiest to setup with a good Go experience.

If you're using Visual Studio Code, let it install the plugins it suggests such as gofmt because they will ensure your code looks good.

Make yourself a folder for a new project (you can use your Github username here):

mkdir -p $GOPATH/src/github.com/alexellis/blank/

Now save a new file named app.go:

package main
func main() {
}

This program is effectively hello-world without any print statements. You can type in go run app.go or go build followed by ./blank.

Parsing JSON

Let's go on to parse some JSON, in Go we can turn a JSON document into a struct which is useful for accessing the data in a structured way. If a document doesn't fit into the structure it will throw an error.

Let's take an API from the Open Notify group - it shows the of people in space and their names.

We have hard-coded the JSON response from Open Notify so that we can work on one chunk of behaviour at a time.

The json.Unmarshall function works with a []byte type instead of a string so we use []byte(stringHere) to create the format we need.

In order to make use of a struct to unmarshal the JSON text we normally need to decorate it with some tags that help the std library understand how to map the properties:

type people struct {
Number int `json:"number"`
}

The property names need to begin with a capital letter which marks them as exportable or public. If your struct's property is the same in the JSON you should be able to skip the annotation.

Another thing to notice is that we pass the address of the new / empty people struct into the method. You can try removing the & symbol, but the value will be set in a different copy of the empty struct.

This may seem odd if you are coming from languages that pass parameters by reference.

Putting it together with HTTP

Now we can parse a JSON document matching that of our API, let's go on and write a HTTP client to fetch the text from the Internet.

Go has a built-in HTTP client in the net/http package, but it has a problem with long timeouts and there are some well-known articles recommending that you set a timeout on your request explicitly.

There are more concise ways of creating a HTTP request in Go, but by adding a custom timeout we will harden our application.

If you're used to coding in Node.js for instance you will be used to most I/O operations returning a nullable error - the same is the case in Go and checking for errors here has made the code quite verbose. You may want to refactor elements of the code into separate methods - one to download and one unmarshal the JSON data.

It may not always make sense to attempt to parse the HTTP response, one case is if the HTTP code returned is a non-200 number. Here's how you can access the HTTP code from the response:

fmt.Printf("HTTP: %s\n", res.Status)

Being a good citizen on the Internet

There's one more important line I wanted to highlight. I've set a User-Agent in the HTTP request's header. This lets remote servers understand what kind of traffic it is receiving. Some sites will even reject empty or generic User-Agent strings.

req.Header.Set("User-Agent", "spacecount-tutorial")

Taking it further

I wanted to keep this post focused but do checkout the links for reading up and come back soon for follow-up posts.

Next time

In the next post we'll approach the problem again with TDD (Test Driven Development) and unit testing in mind. We will also build out a Dockerfile using the Official golang Docker image.