kubernetes, sdn, linux,devops

[Container Network Interface] Implement Your CNI In Golang

Posted on2018-06-16|Edited on2018-10-03|Comments:

As we know, the kubernetes use the CNI to provide the network connectivity for its Pod unit and the cluster administrator can choose what kind of the CNI should be installed in the cluster. For example, if the only requirement is the overlay network, you can choose the flannel CNI and choose the calico CNI if you have the requirement of the BGP. In this post, we will learn how to write your own CNI in golang language. Actually, You can implement it with any language as you like.

Introduction

In order to help the develop to develop their own CNI, the CNCF had setup two projects for developers.

Those projects are based on the golang language and provide useful libraries for the developer to control the Linux network functions, such as IP, netlink and network namespace.

The ContainerNetworking/CNI provides the basic function for CNI implementation in golang and you can see the introduction of that project in its README

As well as the specification, this repository contains the Go source code of a library for integrating CNI into applications and an example command-line tool for executing CNI plugins. A separate repository contains reference plugins and a template for making new plugins.

IPAM

In the host-local, you just need to provide a configuration file to describe what subnet/gateway you want to use and it will allocate a unused IP address from that subnet for your CNI.And the dhcp will runs a DHCP client in each container and send a dhcp request to get a IP address from the dhcp server.

In this tutorial, we will implement a bridge CNI and explain those functions step by step.

Before We Start

Before we start to implement the CNI, we must know the interface/specification of the CNI.

Your CNI will be invoked when the container is ready to create or has been terminated.

Allocate resources for the container, including the IP address and the network connectivity.

Remove all resources you allocated before when a container has been terminated.

The caller will pass the following information into your CNI program

Command (What kind of the event you should care)

ADD

DELETE

VERSION

ContainerID (The target ContainerID)

NetNS (THe network namespace path of the container)

IFNAME (The interface name should be created in the container)

PATH (The current working PATH, you should use it to execute other CNI)

STDIN (The configuration file of your CNI)

Step By Step

For each step, you can find a corresponding folder in my github repo and there’s all golang files for each steps.

Step1

First, we need to provide two function for ADD and DELETE event which is used to allocate/recycle resource when the container has been start/terminated.

Use the go build to build the binary and assume our execution file is example and then we should provide a basic configuration which should contains useful information for our CNI.Maybe we call the file configuration its contents looks like

Use the aforementioned command to call the binary again and you should see the linux bridge test has been created.

If youu don’t have the brctl command, use the apt-get install bridge-utils to to install the bridge tools.

Step3

In the next step, we will creat a veth for connecting the linux bridge and the taget container.

The logical flow are

Get the bridge object from the Bridge we created before

Get the namespace of the container

Create a veth on the container and move the host-end veth to host ns.

Attach a host-end veth to linux bridge

This step is more complicate then previous steps. since we will handle the network namespace here.Fortunately, the CNI project has provided convenience function to handle the veth and it can cover the (3) action itom above.

First, we use the netlink.LinkByName method to lookup the netlink object.

First , we create a function handler which calls the ip.SetpuVeth to create a veth pair on caller’s network namespace and move one side of veth pair to its third parameter(hostNS)

When we call the netns.Do(handler), it will call the function handler in netns's network namepsace and pass the caller’s network namespace to the function handler.Which will result in that there will be a veth pair between the host’s network namespace and netns's netowkr namespace.

In order to store the information about that veth pair, we can use the current.Interface{} object to store the data.

First, we need to import the library

1

import"github.com/containernetworking/cni/pkg/types/current"

and then create a variable represent to host side network interface in the function handler.

There is one important thing we need to care is the OS thread. since we will switch the netns to handle the namespace things.We must make sure the OS won’t switch the thread during the namespace operations.

Use the function runtime.LockOSThread() in the golang predefined function init().

123456

funcinit() {// this ensures that main runs only on main thread (thread group leader).// since namespace ops (unshare, setns) are done for a single thread, we// must ensure that the goroutine does not jump from OS thread to thread runtime.LockOSThread()}

And you can see we have already set the IP address to the interface eth10.You can use the following command to mamually set the IP address to the linux bridge and use the ping command to check the network connectiviy between the host and the target network namespace.

12

sudo ifconfig test 192.0.2.1sudo ip netns exec ns1 ping 192.0.2.1

Summary

In this tutorial, we have implemented a simple Linux Bridge CNI (only Add function) in golang.

We create the linux bridge and use the veth to connect the linux bridge with the target netowrk namespace.Besides, we also fethc the information we want from the pre-defined config file which means we can more flexible to change the behavior of your own CNI implementation.

To make the problem simple, we don’t use any complicated method to acquire a unique address from the config but you can desing you own algorithm to do that.If you want to learn more about the IP related operations, you can go to the host-local to learn more.