This decade’s about to wrap up, so I decided to spend some time
describing my development workflow as the year nears its end.

What I find interesting in my setup is that it entertains
working on a local k8s cluster — mainly to keep in touch
with the systems that run in production.

Running k8s locally isn’t what
you’d want to do to begin with, but rather a natural path
once you start wanting to replicate the environment that runs
your live applications. Again, you don’t need a local k8s
cluster just ‘cause, so make sure you have a good reason
before going through the rest of this article.

Cluster setup

Once a monstrous task, setting up a local k8s cluster is
now as simple as installing a package on your system:
Docker for Win/Mac allow you to run this very easily, and
Canonical has made it possible on Linux through microk8s
(that’s my boy!).

One of the funny things about running on microk8s
(or snaps, in general) is how it will automagically
upgrade under your nose — sometimes with breaking changes.
There was a recent change that swapped docker for containerd
as microk8s’ default container runtime, and it broke some
local workflows (more on that later, as it’s easy to fix).
In general, you can always force a snap to use a particular
revision, so if anything’s funky just downgrade and let others
figure it out :)

I’d be keen to try k3s out, as it seem to provide an even more
lightweight way to run the local cluster. Built mainly for
IoT and edge computing, k3s is interesting as running
microk8s is sometime resource-intesive — once I’m done
working on code, I usually prefer to sudo snap disable microk8s
in order to preserve RAM, CPU and battery life (proof here).

In the past, I’ve also tried to work on a remote k8s cluster
in the GKE from my local machine, but that proved to be too
much of a hassle — the beauty of kubectl is that you don’t
really care where the cluster is running, but your IDE and
other tools work best when everything is present and running
locally.

Development tool

This has been fairly stable until late this year, when I
decided to switch things around.

I’ve historically used helm and a bunch of shell magic
to run apps locally: you would clone a repo and expect
an helm/ folder to be available, with the chart being
able to install a whole bunch of k8s resources on your
cluster. Then, a bash script run simply apply the chart
with a bunch of pre-configured values: you would run dev up
and what the script would do would simply be something
along the lines of:

I started off with Helm 2, and v3 brought in a few changes I didn’t want to go through

helm is perfect if I want to package a generic app made up of multiple resources (service, ingress, etc) and release it to the outside world. Locally, I probably don’t need all of that verbosity (chart.yaml and so on)

most of the templating I did on development was {{ .Release.name }}. What’s the point then?

Towards the end of this year I went back to the drawing board
and started to think what if there was anything else I could
use that was simple enough and gave me enough flexibility.
I knew I could use simple k8s manifests but it wasn’t clear
to me how I could integrate it into my workflow in a way
that made it simpler than using a chart — and that’s when I
gave skaffold another chance.

Skaffold is an interesting tool, promoted by Google, that
supposedly handles local k8s development — and I say “supposedly”
because I’ve tried it in the past and have been extremely
underwhelmed by its workflow.

Let me explain: whenever a chance is detected in your codebase, skaffold
wants to redeploy your manifests but, rather than simply
working on an application-reload logic, is instead happy
to:

re-build your local image

push it to a registry

update the k8s deployment so a brand new pod comes up

If you’ve made it so far you probably realized that
the whole operation doesn’t either come cheap nor fast
— you could be waiting several seconds for your changes
to take effect…

That was, until skaffold introduced file sync
to avoid the need to rebuild, redeploy and restart pods.
This feature is currently in beta, but it’s already working
well enough that I’ve decided to give it a shot, with very
positive results.

Now, rather than having an entire chart to mantain locally,
my development setup has a simple skaffold.yaml that
looks like:

That’s about it. Now my dev up is mapped to a simple
skaffold dev, and skaffold takes care of re-building
the image when needed, syncing changes locally and so on.
One of the advantages of using this tool is that it automatically
detects changes to the manifests and the Dockerfile, so
it re-builds the image without you having to trigger the
process manually (which wasn’t possible with Helm alone).

Another interesting benefit of using skaffold is the support
for base registries as well as build stages. The former
allows you to run a registry at any given URL, and tell
skaffold to prepend that URL to any image that’s being pushed
to the k8s cluster.

As I mentioned, I use microk8s, which doesn’t play very well
with locally-built images,
so I simply run the built-in registry on port 32000. Others
in my team simply run Docker for Mac which doesn’t need a registry
as any image built locally is automatically available to k8s.

This would mean that I would have to update the image field
of my deployments, manually, to localhost:32000/my_app, a
tedious and annoying operation (and I’d also have to make sure
those changes aren’t pushed to git). Skaffold frees you from the
drama with a simple skaffold config set default-repo localhost:32000,
a trick that will tell skaffold to parse all the manifests it deploys
and replace the image fields, prepending the URL of your own registry.
The feature is documented extensively here,
and it’s a life saver!

The support for build stages is another great trick up in
skaffold’s sleeve, as it allows to use the power of Docker’s
multi-stage builds in your development environment.

Believe me, skaffold has made my life so much easier and it’s a tool
I would gladly recommend. Before introducing file syncing I didn’t
want to get my hads dirty with it, as I did not find the development
workflow sustainable (re-build and re-deploy at every file change),
but right now it works much better than anything I could have come
up with on my own.

Hands on the code

Last but not least, we went over running a cluster as well as our
application — but how do we actually debug our code or run tests?

Ideally, we’d like a script that would be able to:

build and run your app (up)

execute commands inside the container (exec)

jump inside the container (in)

execute tests (test)

Wouldn’t it be nice to simply open a shell and run your tests with dev test?

Turns out, creating a simple wrapper over our wokflow is very
straightforward, and here’s a sample of the code one could write:

All this script does is to read the command passed to it
and run it as a bash function. As you see, up is mapped
to skaffold dev, and in is mapped to kubectl exec ... -- bash
(so that you can jump into the container and run whatever
command you’d like).

The actual dev I run locally is on github, under odino/k8s-dev,
and I believe I should credit Hisham for the original idea
— this is a script we’ve been using (and polishing) since ages.

If you’re wondering how does it look on the
terminal, here’s an asciicast where tests
are run succesfully (dev test), we update
the code to make the tests fail and then
we jump into the container (dev in), before
cleaning up (dev delete):

That’s a wrap

Oh boy, right on time to close 2019 with a splash!

Developing on a local k8s cluster isn’t super straightforward,
and I hope that by sharing my toolbox it should be easier for
you to set your environment up for a productive day.

It contains 160+ pages of content dedicated to securing web applications and improving your security awareness when building
web apps, with chapters ranging from explaining how to secure HTTP cookies with the right flags to understanding why it is
important to consider joining a bug bounty program.