The Next AGRC WebAPI

This article will take 5 minutes to read

The current WebAPI has been in production serving the public for multiple years. It handles millions of requests for spatial information every month and is free to use with a simple to create API key. While it receives constant enhancements and bug fixes, there have been no breaking changes to the API, nor have there been any major architectural updates. That is about to change with a major update currently in the works.

The interesting and often wild office discussions about how to improve the WebAPI have not been backed by concrete efforts until now. Implementing these ideas is now possible since key technologies have made amazing advancements while others have matured. AGRC is ready to combine our ideas with new technologies to make the WebAPI even better. Work has already started on what will most likely be version 2 of the WebAPI. And we hope to bring the same improvements to version 1 without interruption.

The History

The WebAPI proof of concept was built using ASP.NET WCF. After getting positive internal feedback (and usage in production, of course), we wrote version 1 of the WebAPI using ASP.NET WebAPI 3. The ASP.NET WebAPI then merged with ASP.NET MVC, and we are currently running on ASP.NET MVC 5.

RavenDB is the user and key store and Redis is being used for analytics and reporting. Redis and RavenDB are both running as Windows services and the WebAPI and developer website are hosted as separate sites in IIS. Supplementary data is cached from Google Sheets to allow for easy modifications.

Reading that last paragraph may have made you think that it would be hard to replicate or upgrade the entire WebAPI system. If that’s the case–you are correct. The list of installation instructions and documentation for the version 1 WebAPI system are as frustrating as they are long. A design goal of version 2 is to eliminate that friction as much as possible.

The Future

The new framework we plan to use, ASP.NET Core, is a ground-up redesign of ASP.NET 4 with the important design goals of being cross platform, lean, and modular. From our experience, this framework is fast! The WebAPI in development is running on ASP.NET Core 2.2 preview 2. From our testing, it can handle much more throughput than the current system. A huge benefit of this framework is that developers at the AGRC can run or develop this code using Visual Studio Code and Visual Studio IDE on Windows, Mac, and Linux. This creates a great developer experience by allowing any and all development environments. The cross platform abilities enables this project to enter the world of Docker and Kubernetes.

Containers

Version 2 of the WebAPI is being built for containers. It will run without them, but building the correct runtime environment is much simpler using them. The system is currently comprised of four containers that will likely expand to six or more.

The ASP.NET Core WebAPI service

The ASP.NET Core developer website

The PostgreSQL database

The Redis database

The API Explorer website *

The Google sheet synchronization service *

The logging/reporting service (Kibana) *

*not yet built

ASP.NET Core runs very well in a container and will drive the WebAPI and developer websites. We chose to replace RavenDB with Postgres as the user and key store. This change is because PostgreSQL runs amazing in a Docker container for development. It isn’t quite durable enough for production, but since the major cloud providers offer cloud SQL equivalents to Postgres, the WebAPI can use a managed cloud database for production.

Redis remains our choice for in-memory caching and also runs very well in a container. The main goal of using containers is that docker compose can start the entire system with one command. Using Docker, anyone can play with the system without having to install additional software.

Orchestration

Our version 2 WebAPI will run many containers at the same time, and managing how those containers work together is what Kubernetes was designed to do. Kubernetes allows AGRC to scale the WebAPI automatically or using a CLI tool. Kubernetes will update our containers when AGRC makes changes to the code and can do so without any downtime. It can also monitor and manage the health of our containers to maintain our SLA.

To recap, AGRC is writing code and organizing it into separate containers. The containers are being managed by Kubernetes, but they still need compute infrastructure to run on, so why stop there?

We have taken it one step further with Terraform. Terraform allows AGRC to template and script the infrastructure that Kubernetes and the containers run on. To build on the magic of running the WebAPI with one command, with Terraform, we can have the WebAPI running in the Google Cloud Platform with one command–all of the nodes, networks, etc. We are very excited for this level of predictability and declarative configuration to run this important AGRC system.

The Minutia

There are a few notable changes to the WebAPI source code that happened during this transition that only a software developer would appreciate.

Like the command pattern, testing your handlers is straightforward because of that clean separation. Another highlight of MediatR are its behaviors. Behaviors allow handlers to be be chained together to create a pipeline where they operate on the input and output of other handlers. This allows for simple extensibility without modifying the original code. Registering the behavior with your dependency injection container will make sure it is used. Read more about the mediator pattern and its benefits on this great blog.

Given the WebAPI’s history, we were very happy to take advantage of new language features, like replacing all of the C# 5.0 TPL Task continuations with async and await. We are moving away from config files for our application secrets in favor of environment variables and the Options pattern. This allows for a seamless experience when running locally (with or without docker) and in Kubernetes.

Google Sheets is a lynchpin in the version 1 of the WebAPI since required data is loaded from a handful of sheets on application startup. If the Google Sheets API is unavailable, the WebAPI will be in an unhealthy state.

In new version 2, we imported a current snapshot of the sheets data into Postgres and are creating a microservice to keep the data synchronized. This way the application is guaranteed to start in a healthy state, and the semi-static reference data can be updated when it changes–either on a schedule or when notified via web hooks. This will be a major improvement over the version 1 WebAPI.

Version 1 of the WebAPI contains a canary endpoint to make sure the required services are available. ASP.NET Core makes these health checks a first class feature. We can now write health checks that not only let Kubernetes and other monitoring systems know that we are in a healthy state but when things go wrong, we will know exactly what dependency is not functioning properly.

It is a great time to be a software engineer. There are so many services and software to improve our effectiveness, make us more efficient, and help us deliver a better overall experience to our users. If you have not tried any of these software, you should strongly consider it.

It has been a fun and exciting experience developing the next version of the WebAPI. After all, if the work you are doing isn’t enticing and fun, maybe it’s time to make a change.