Platforms

Geocoding Address Data in Batch Quantities with HERE and Golang

By Nic Raboy | 06 November 2018

If you’re a larger organization processing massive amounts of address information, you’re probably going to burn through your HERE API transactions pretty quickly and rack up a nice bill. The good news is that you can easily prevent this by batching your requests into fewer transactions using the HERE Batch Geocoder API.

With the Batch Geocoder API you can send up to one million addresses to be geocoded in a single transaction. This could potentially save you a lot of money in the long term, depending on the needs of your application.

We’re going to see how to use the HERE Batch Geocoder API with Golang to process a potentially large amount of data, check on the status, and download results to a file.

Building Native Data Structures for the HTTP Response

Rather than trying to use a generic Go interface{} for all of our responses and data, we should probably more accurately model our data so it is easier to work with in the long term.

If you take a look at the documentation, you’ll notice that all responses of the Batch Geocoder API are of the same format. This is good for us because it limits the work we need to do.

Create a main.go file somewhere in your $GOPATH and include the following:

A lot of the code above is boilerplate Go code, but we have two data structures. The first data structure is a model of the potential response. Per the documentation, the response is XML, which isn’t exactly the nicest format to work with. Lucky for us, Go makes it easy to switch between formats using XML and JSON annotations. The annotations allow us to map the response to an object with no manual intervention.

The Geocoder data structure will hold our app id and app code found in the HERE Developer Portal. To use this API you must have a developer account, which is free.

Developing Functions for the HERE Batch Geocoder API Requests

With the data models in place, we can focus on developing functions that will process our requests. As of now there is no Golang SDK for the HERE services, but REST is available and not difficult to use.

Using the HERE Batch Geocoder API is a multi-step process. This means that no single request will allow us to upload data and get a geocoded response back. Instead, there are a few things to do, the first of which is upload data and start the process.

Most of the above code is around constructing a request. The batch function does require a payload to be provided. The payload in this circumstance is the byte data for a CSV file. After setting the query parameters, a POST request is made, passing the CSV file in the request.

When the request is successful, the XML body that is returned is stored in the BatchResponse variable. Without getting too far ahead of ourselves, the final outcome for this request might look something like the following image:

If everything went smooth, you’d be given a RequestId and a message saying the request was accepted. The RequestId will be used in future requests for checking the status or downloading the results.

With the batch process running remotely, we’d want to check the status with a status function like this:

In the above code, the request is a little different. Instead of a POST request, we’re doing a GET request and the RequestId is expected. Assuming all went well, the response will be parsed into a BatchResponse object and returned. In the end, it might look something like the following image:

Between the status response and the batch response, they are more or less the same, with the exception that the status has information about the process. When the process has completed, you’ll have information about the data and be able to download it.

The final step in the process is to download a ZIP archive of the data. We can create a data function that looks like the following:

The CSV file is read and the byte data is passed as a payload to the batch function. The response is then converted into JSON and presented in the Terminal. It was pretty printed in the example images because of an optional Python tool.

In the above code, we pass the RequestId to the data function and the binary data that is returned, is printed in the Terminal. Using the binary data, we can save it to a file like so:

go run *.go -d yCQBgMSQgdXhxnwkdMqjMec3doxeZl2v >> output.zip

If you’d rather not pipe the response into a file, you could always implement some file logic within the Go application directly. From a scripting perspective, I thought it’d be more valuable to allow piping of the data.

Conclusion

You just saw how to use the HERE Batch Geocoder API in your Golang application. While batching requests might not be useful in every possible scenario, given the correct scenario, it could save your company a lot of money in API transactions.

Nic Raboy is an advocate of modern web and mobile development technologies. He has experience in Java, JavaScript, Golang and a variety of frameworks such as Angular, NativeScript, and Apache Cordova. Nic writes about his development experiences related to making web and mobile development easier to understand.