Service mesh implemented via iptables

Imaginary distributed app with services plugged into the service mesh

So last time I mentioned, that another Kubernetes compatible service mesh – Conduit – has chosen another approach to solve the problem. Instead of enabling the mesh at machine level via e.g. http_proxy env variable, it connects k8s pods or deployments to it one by one. I really like such kinds of ideas that make 180° turn on solving the problem, so naturally I wanted to see how exactly they did that.

So, what’s Conduit again?

As I said, Conduit is a service mesh for Kubernetes. Instead of letting application services to talk directly to each other, it forces them to send all traffic through the mesh. In return this allows us to collect all sorts of statistics and logs, adds more control over traffic routing and provides all other goodies that self respecting mesh should provide.

Installation

It’s also fairly easy to install. Basically, downloading the binaries via provided shell script and adding it to PATH is more than a half of a deal:

Install Conduit

Shell

1

2

curl https://run.conduit.io/install|sh

export PATH=$PATH:$HOME/.conduit/bin

The second part is to add its control layer to Kubernetes cluster (in my case – minikube), which is simply another shell command:

Register Conduit

Shell

1

2

3

4

5

minikube start# Create local k8s cluster

conduit install|kubectl apply-f-

# namespace "conduit" created

# serviceaccount "conduit-controller" created

# ...

Dissecting install script

Before we go further, it looks like conduit install simply generates YAML configuration for Kubernetes, which then gets applied by kubectl apply. I actually wonder, what’s inside.

Dissect conduit install script

YAML

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

conduitinstall|vim-

### Namespace ###

kind: Namespace

apiVersion: v1

metadata:

name: conduit

# ...

kind: Service

apiVersion: v1

metadata:

name: prometheus

# ...

kind: Service

apiVersion: v1

metadata:

name: grafana

# ...

Cool! So among the whole bunch of accounts, services and deployments they also use Prometheus and Grafana. I think I know how they are collecting and displaying the metrics now.

Checking the dashboard

Sure, there will be no data, but it still interesting to see how it looks. conduit dashboard does the trick:

That’s nice. All is up and running, all’s green. The most interesting part is in the bottom. It says that cluster has 4 k8s namespaces, but only one of them – conduit – is fully connected to the service mesh.

So they are monitoring conduit with conduit? Nice. And it also means that there will be some network activity to see after all. Clicking at conduit namespace and…

They do have metrics! Per deployment and per pods. OK, going further – clicking at grafana deployment:

And we see Grafana, displaying traffic data about Grafana. Almost a recursion.

Wiring up more pods

Let’s head back to the main page, to the list of wired/unwired namespaces.

conduit is completely wired, default and kube-public are empty, but kube-system is both unwired and has 9 pods in it. I don’t think that would be a good idea to do that in production, but for minikube – what if we try to connect some system pods to the mesh?

Connection to the mesh is done via command line (I think they also have API for that): conduit inject. It takes YAML configuration for Kubernetes object (e.g. pod or deployment) as an input and returns a new one, slightly modified, with its traffic configured to go through the mesh. Before I actually connect anything to it, let’s check what exactly gets injected into, let’s say, deployment.

Comparing before and after YAMLs

kube-system namespace, which is the only one in default minikube setting, that has any pods in it, has two deployments:

Look at that beauty. Apparently, they are adding two containers: one that seems to accept the traffic and resend it to controller pods (gcr.io/runconduit/proxy:v0.4.2, line 189) and the other one – Init Container, gcr.io/runconduit/proxy-init:v0.4.2, line 211 – to run before anything else does and to setup the whole thing. I wonder what’s inside.

So it’s written in Go. The source files are also right there and the great reveal is that unlike Linkerd with it routing a traffic via http_proxy env variable, Conduit does this via iptables. At least that’s what my understanding of Go and word iptables tells me. As a side note, code for the second Docker image – proxy – is written in Rust. Right language for right job, cool.

Attaching kube-system deployments to the mesh

This satisfies my curiosity for now, so it’s time to try to connect the only other namespace with pods to the mesh: kube-system.

Conclusion

So this is Conduit. Like with Linkerd and a service mesh in general, I can’t say that playing with it was an eye opener for me. But I do like nice concepts and interesting implementation and a service mesh with Conduit is definitely the one. I really love how much one could learn just from looking at their source code: mixing languages works not only in theory, but in real products; Rust is alive; Init Containers are cool; iptables are too. I wonder, however, has anyone did a benchmarking of how using a service mesh affects performance? That naturally is going to be the first question anyone will ask me if I try to ‘sell’ the idea.