Tuesday, December 06, 2016

Using Helm to install Traefik as an Ingress Controller in Kubernetes

First of all, just a bit of theory. If you want to expose your application running on Kubernetes to the outside world, you have several choices.

One choice you have is to expose the pods running your application via a Service of type NodePort or LoadBalancer. If you run your service as a NodePort, Kubernetes will allocate a random high port on every node in the cluster, and it will proxy traffic to that port to your service. Services of type LoadBalancer are only supported if you run your Kubernetes cluster using certain specific cloud providers such as AWS and GCE. In this case, the cloud provider will create a specific load balancer resource, for example an Elastic Load Balancer in AWS, which will then forward traffic to the pods comprising your service. Either way, the load balancing you get by exposing a service is fairly crude, at the TCP layer and using a round-robin algorithm.

A better choice for exposing your Kubernetes application is to use Ingress resources together with Ingress Controllers. An ingress resource is a fancy name for a set of layer 7 load balancing rules, as you might be familiar with if you use HAProxy or Pound as a software load balancer. An Ingress Controller is a piece of software that actually implements those rules by watching the Kubernetes API for requests to Ingress resources. Here is a fragment from the Ingress Controller documentation on GitHub:

What is an Ingress Controller?

An Ingress Controller is a daemon, deployed as a Kubernetes Pod, that watches the ApiServer's /ingresses endpoint for updates to the Ingress resource. Its job is to satisfy requests for ingress.Writing an Ingress Controller

Writing an Ingress controller is simple. By way of example, the nginx controller does the following:

As I mentioned in a previous post, I warmly recommend watching a KubeCon presentation from Gerred Dillon on "Kubernetes Ingress: Your Router, Your Rules" if you want to further delve into the advantages of using Ingress Controllers as opposed to plain Services.
While nginx is the only software currently included in the Kubernetes source code as an Ingress Controller, I wanted to experiment with a full-fledged HTTP reverse proxy such as Traefik. I should add from the beginning that only nginx offers the TLS feature of Ingress resources. Traefik can terminate SSL of course, and I'll show how you can do that, but it is outside of the Ingress resource spec.

I've also been looking at Helm, the Kubernetes package manager, and I noticed that Traefik is one of the 'stable' packages (or Charts as they are called) currently offered by Helm, so I went the Helm route in order to install Traefik. In the following instructions I will assume that you are already running a Kubernetes cluster in AWS and that your local kubectl environment is configured to talk to that cluster.

Install Helm

This is pretty easy. Follow the instructions on GitHub to download or install a binary for your OS.

Initialize Helm

Run helm init in order to install the server component of Helm, called tiller, which will be run as a Kubernetes Deployment in the kube-system namespace of your cluster.

Get the Traefik Helm chart from GitHub

I git cloned the entire kubernetes/charts repo, then copied the traefik directory locally under my own source code repo which contains the rest of the yaml files for my Kubernetes resource manifests.

# git clone https://github.com/kubernetes/charts.git helmcharts

# cp -r helmcharts/stable/traefik traefik-helm-chart

It is instructive to look at the contents of a Helm chart. The main advantage of a chart in my view is the bundling together of all the Kubernetes resources necessary to run a specific set of services. The other advantage is that you can use Go-style templates for the resource manifests, and the variables in those template files can be passed to helm via a values.yaml file or via the command line.

I copied the dashboard-ingress.yaml template file from the Traefik chart and customized it so as to refer to my application's web service, which is running in a Kubernetes namespace called tenant1.

# cd traefik-helm-chart/templates

# cp dashboard-ingress.yaml web-ingress.yaml

# cat web-ingress.yaml

{{- if .Values.tenant1.enabled }}

apiVersion: extensions/v1beta1

kind: Ingress

metadata:

namespace: {{ .Values.tenant1.namespace }}

name: {{ template "fullname" . }}-web-ingress

labels:

app: {{ template "fullname" . }}

chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"

release: "{{ .Release.Name }}"

heritage: "{{ .Release.Service }}"

spec:

rules:

- host: {{ .Values.tenant1.domain }}

http:

paths:

- path: /

backend:

serviceName: {{ .Values.tenant1.serviceName }}

servicePort: {{ .Values.tenant1.servicePort }}

{{- end }}

The variables referenced in the template above are defined in the values.yaml file in the Helm chart. I started with the variables in the values.yaml file that came with the Traefik chart and added my own customizations:

# vi traefik-helm-chart/values.yaml

ssl:

enabled: true

acme:

enabled: true

email: admin@mydomain.com

staging: false

# Save ACME certs to a persistent volume. WARNING: If you do not do this, you will re-request

# certs every time a pod (re-)starts and you WILL be rate limited!

persistence:

enabled: true

storageClass: kubernetes.io/aws-ebs

accessMode: ReadWriteOnce

size: 1Gi

dashboard:

enabled: true

domain: tenant1-lb.dev.mydomain.com

gzip:

enabled: false

tenant1:

enabled: true

namespace: tenant1

domain: tenant1.dev.mydomain.com

serviceName: web

servicePort: http

Note that I added a section called tenant1, where I defined the variables referenced in the web-ingress.yaml template above. I also enabled the ssl and acme sections, so that Traefik can automatically install SSL certificates from Let's Encrypt via the ACME protocol.

Install your customized Helm chart for Traefik

With these modifications done, I ran 'helm install' to actually deploy the various Kubernetes resources included in the Traefik chart.

I specified the directory containing my Traefik chart files (traefik-helm-chart) as the last argument passed to helm install:

2. Configure DNS records corresponding to Kubernetes ingress resources to point to the load balancer IP/hostname found in step 1

At this point you should see two Ingress resources, one for the Traefik dashboard and on for the custom web ingress resource:

# kubectl --namespace tenant1 get ingress

NAME HOSTS ADDRESS PORTS AGE

tenant1-lb-traefik-dashboard tenant1-lb.dev.mydomain.com 80 50s

tenant1-lb-traefik-web-ingress tenant1.dev.mydomain.com 80 51s

As per the Helm notes above (shown as part of the output of helm install), run this command to figure out the CNAME of the AWS ELB created by Kubernetes during the creation of the tenant1-lb-traefik service of type LoadBalancer:

Now, if you hit http://tenant1-lb.dev.mydomain.com you should see the Traefik dashboard showing the frontends on the left and the backends on the right:

If you hit http://tenant1.dev.mydomain.com you should see your web service in action.

You can also inspect the logs of the tenant1-lb-traefik pod to see what's going on under the covers when Traefik is launched and to verify that the Let's Encrypt SSL certificates were properly downloaded via ACME: