SQR-015: Creating Microservices for api.lsst.codes

The site api.lsst.codes is intended to be a unified front-end for
API services designed for programmatic consumption to support LSST Data
Management. This technote will document how to use the tools that
SQuaRE has provided to more easily write and deploy these API
microservices.

In general, a microservice will be a way to extract data from an
underlying back-end source. It may be used to extract from multiple sources
and correlate and aggregate those sources. It also may be used to
reformat or reduce that data in a way that is sensible to consume
programatically.

You may ask, “why not just hit the backend? What value does the
microservice provide?” There are two, somewhat complementary, answers.
First is that if you put the service behind a unified framework and
enforce certain standards for discoverability and output format, you can
make writing tools to query your services a great deal easier. Second
is that the first use case for this is in a chatops service, and we
wanted to make the chatbot, in so far as possible, a presentation-layer
front end and keep data manipulation and reduction logic out of it.
This is particularly true since (if we stick with Hubot) the chat logic
is written in CoffeeScript. Python is already part of the DM technical
stack, which CoffeeScript is not, and therefore likely more familiar to
DM developers.

This document assumes that you will write your microservice in Python 3
and Flask (assuming you’re working in Python, it’s nice if it works
under Python 2 as well), and that you will use the apikit module
(source at GitHub) to provide the required metadata
routes.

However, there’s also a Quick Start Guide if you don’t really care how
all this works, as long as it works.

What’s actually going on in the project you just created with
cookiecutter? The rest of this technote is an answer to that
question, and will show construction (from scratch) of a toy example of
a microservice that provides a more useful interface to an underlying
backend service.

We will begin with the metadata routes any api.lsst.codes application
must provide.

The metadata must be presented as a JSON object. All fields are
type str.

{nameversionrepositorydescriptionapi_versionauth}

The fields name, description, and version are arbitrary.
Semantic versioning is strongly encouraged for version.
api_version should reflect the version of the API in use (at time of
writing, 1.0). repository is the URL for the source of your
project. If you want your microservice to be published on
api.lsst.codes its source must be publicly available. We extremely
strongly recommend hosting it on GitHub.

The auth field is constrained. It must be one of none,
basic, or bitly-proxy. These represent the three choices
available to the microservice for authentication to GitHub (we have
standardized on GitHub as a canonical source of authentication data,
since LSST is fairly fundamentally coupled to GitHub and it functions as
a widely available OAuth2 provider):

none: no authentication required.

basic: HTTP Basic Auth. Typically used with a GitHub username and
token, although if you didn’t have two-factor authentication enabled
at GitHub you could use a password here as well.

bitly-proxy: Authenticate through the Bitly OAuth2 proxy.
Typically used with a GitHub username and password, and basically
converts two-factor authentication back into username-and-password
authentication.

The good news is, if you’re writing in Python and your application is a
Flask app, you don’t need to implement the
metadata route. Just use apikit.

The apikit module is documented at GitHub. apikit has
two classes: apikit.APIFlask and
apikit.BackendError, and two functions:
set_flask_metadata() and add_metadata_route().

The apikit.APIFlask class is what you should generally use: it
is a subclass of a Flask application (flask.Flask) that
already has metadata added and the route baked into it.

If you have an existing Flask application, you might want to use
apikit.set_flask_metadata() on that application rather than the
apikit.APIFlask class. You will find add_route_prefix()
useful to add additional routes to the metadata. That is helpful, for
instance, for Kubernetes Ingress resources, which provide routing but
not path rewriting, which makes it your responsibility to ensure the
metadata is available at /{{app_name}}/metadata as well as
/metadata.

The apikit.BackendError class is useful with Flask decorators
to return diagnostic information when something goes wrong with your
application. You’ll see it in the example below.

Let’s pretend that you have a service living at
https://myservice.lsst.codes, which you want to turn into a microservice
(that is, put an api.lsst.codes-conformant API wrapper around) using
apikit. Your service uses the Bitly OAuth2 proxy to use GitHub as its
authentication source, so you need to leverage that.

We’ll say that this is going to go in a directory
uservice_mymicroservice, and we will package it for installation via
setuptools. The server itself will, imaginatively, be called
server.py. (This mirrors the setup you would get if you used
cookiecutter to create the service.)

Having done that, we need to create the microservice as an instance of
apikit.APIFlask. This class takes the same arguments as the
object returned by metadata, with the following exception: auth
becomes an object with two fields, type and data, unless auth is
one of None, the empty string, or the string none. The type
field must be one of the strings none, basic, or bitly-proxy.

If auth is an object whose type field is none, auth.data is
the empty object, or omitted completely. Otherwise auth.data is an
object with two fields, username and password. If auth.type
is bitly-proxy then auth.data must have a third field,
endpoint, which is the start point of the OAuth2 proxy data flow
for the underlying service. Usually this is
https://service.host/oauth2/start.

The api_version field has a sane default (currently 1.0) and can
normally be omitted.

This creates a Flask application which presents the service metadata on
/metadata, /v1.0/metadata, /mymicroservice/metadata, and
/mymicroservice/v1.0/metadata/, as well as all of those with
.json appended.

Now, in order to actually access your data, you’re going to need to make
your requests within a session with the appropriate authentication.
Let’s assume that your caller is going to send you HTTP Basic
Authentication headers, and you’re going to use those as username and
password to the proxy.

You’ll need a place to store the session. Fortunately, Flask provides a
mechanism for this: the app.config dict.

Since this application is eventually going to run under Google Container
Engine using an Ingress TLS terminator and router (well, this is our
current state, and it is our assumption that it will be that way
long-term, anyway), you want the actual application root to return a
200 very quickly, because the Ingress controller will be pinging it
often to determine service health (GCE’s Ingress defines a successful
healthcheck as getting 200 from an HTTPGET/.

Finally, let’s add the actual service. In addition to the routing and
fetching logic, you will need to peel the authentication headers out of
the inbound request and create a session with them, if you don’t already
have a session with the correct authentication information.

Let’s say you have decided that your microservice interface will respond
to GET/mymicroservice/jobname/metric to retrieve the named metric about
jobname (for instance, GET/mymicroservice/buildmyapp/time
to get back data about how long a build took).

We’ll pretend that your backend service is ill-behaved, and does the
following annoying things:

It wants its arguments as parameters on the HTTPGET rather than
as a request body or a path on the GET URL.

It returns the requested metric as a plain text value, rather than
wrapped in JSON or XML or anything sane.

Therefore, you call it with GET/api?metric=metric&job=jobname and
what you get is what you get, which you hope is ASCII text, or maybe
UTF-8, but it’s not like the other side is going to guarantee that to you.

Flask provides a nice decorator service for pointing routes to
functions. You’ve seen it above with the healthcheck route: just put
@app.route atop the function definition.

# Route it to the root too, in case we want to put it behind nginx# or HAProxy or something that can do path rewriting.@app.route("/<jobname>/<metric>")@app.route("/mymicroservice/<jobname>/<metric>")defget_metric_for_job(metric=None,jobname=None):"""Retrieve the metric and format it with JSON for return."""# Create a custom error if metric or jobname are not specifiedifmetricisNoneornotmetricorjobnameisNoneornotjobname:raiseBackendError(reason="Bad Request",status_code=400,content="Must specify metric and jobname.")# If we have authorization on the request, try to use itifrequest.authorizationisnotNone:inboundauth=request.authorizationcurrentuser=app.config["AUTH"]["data"]["username"]currentpw=app.config["AUTH"]["data"]["password"]# If we are already using this user/pw, don't bother.ifcurrentuser!=inboundauth.usernameor \
currentpw!=inboundauth.password:_reauth(app,inboundauth.username,inboundauth.password)else:raiseBackendError(reason="Unauthorized",status_code=401,content="No authorization provided.")session=app.config["SESSION"]# This is going to end up in the same function where backenduri# is defined. See below.url=backenduri+"/api"params={"metric":metric,"job":jobname}resp=session.get(url,params=params)ifresp.status_code==403orresp.status_code==401:# Try to reauth_reauth(app,inboundauth.username,inboundauth.password)session=app.config["SESSION"]resp=session.get(url,params=params)ifresp.status_code==200:# Success!rdict={"metric":metric,"jobname":jobname,"value":resp.text()}returnjsonify(rdict)else:raiseBackendError(reason=resp.reason,status_code=resp.status_code,content=resp.text)

defserver(run_standalone=False):# Refer to the earlier pieces of this document for the code# fragments that need to be inserted in place of the comments.## APIFlask instantiation to create the application goes here...# ...then add SESSION to the config dict...# ...next, add an error handler...# ...then, your healthcheck...# ...finally, your actual route.## And now a bit of new code, to run the service if invoked standalone:ifrun_standalone:app.run(host='0.0.0.0',threaded=True)

The imports go at the top of server.py, of course, and the
_reauth() function stands on its own, not nested inside server().

The only other thing you really need is to add a Python shebang and
invoke server() standalone if the script is run from the
command-line. Making standalone() its own function makes
setup.py a bit prettier.

Your service will eventually be set up to run as a Docker container
under Google Container Engine. This will require population of a
Dockerfile and deployment description files in kubernetes.
However, those files are not in scope for this document, and, in
general, are expected to be added by the DM SQuaRE team. (If you use
cookiecutter you will already have these files, and they will be
modified as needed by the SQuaRE team.)

If you, as a service author, want to stop after making the service
pip-installable with setuptools, that’s perfectly fine. SQuaRE will
take it from there.