How a Container Runtime is using CNI

The last couple of years I have been working with containers in low level. I
learned how they work under the hood and I got familiar with container
standards like OCI (Open Container Initiative) and CNI (Container Network
Interface). When it comes to the runtime spec, there are quite many resources
which explain in detail the runtime standard. On the other hand, there are not
many resources about CNI, and it has not been clear to me how the runtime
engine is using the CNI standard, as well as how difficult is to
write a network CNI compatible plugin.

Introduction to CNI

CNI stands for Container Networking Interface, and it targets to standardize
the interface between the container runtime engine and the network
implementation. It is a minimal standard way to connect a container to a
network.

When discussing about CNI, there are some information that I find nice to keep
in mind:

A plugin that implements the CNI standard is binary, not a daemon. It should always have
at least CAP_NET_ADMIN
capability when runs.

Network definitions or network configuration are stored as json files. These
json files are streamed to the plugin through stdin.

Any information that is only known when the container is
to be created (runtime variable) should be passed to the plugin via
environmental variables. Nevertheless, in the latest CNI it is also possible
to send certain runtime config via json on stdin, especially for some
extension & optional features. More info here.

The binary does not suppose to have any other input configuration outside the
above two.

The CNI plugin is responsible for wiring up the container, and is expected to
hide the network complexity.

its an advanced use case, but its also an interesting point because it means that the runtime essentially modifies the config JSON after the operators provides it, but before the plugin sees it.

Container Runtime

In a OCI/CNI compatible version, the Container Runtime [Engine] is a daemon
process that sits between the container scheduler and the actual implementation
of the binaries that create a container. This daemon does not necessarily need
to run as
root user.
It listens for requests from the scheduler. It does not touch the kernel, as
it uses external binaries through the container standards to actually create or
delete a container.

For example in kubernetes case, the Container Runtime can be cri-o,
(or cri-containerd), it listens for requests from the kubelet, the agent from
the scheduler located in each node through the
CRI
interface. Kubelet is instructing the Container Runtime to spin up a
container, and the runtime is executing by calling in a standard way the runc
(binary that implements the OCI-runtime specs) and
flannel (binary that implements the CNI
CNI). The above process is summarized in the following image.

Container Runtime needs to do the following in order to create a container:

Creates the rootfs filesystem.

Creates the container (set of process that run isolated in namespaces and limited by cgroups).

Connects the container to a network.

Starts the user process (entrypoint).

As far as the network part is concerned, important is that the Container
Runtime asks from the OCI-Runtime binary to place the container process into
a new network namespace (net ns X). The Container Runtime in the next step
will call the CNI plugin using the new namespace as runtime ENV variable.
The CNI plugin should have all the info in order to do the network magic.

How Container Rutime Uses CNI

We will give an example how a runtime uses CNI to connect a container to a bridge
using the bridge
plugin.
We will “emulate” the actions of the runtime using simple bash commands.

Provision Phase

Before a runtime can spin up a container it needs some server provisioning.
Which tool is being used (e.g bosh, ansible, manual scripts) is irrelevant. It
only needs to make sure that the required binaries are in place. In our simple
case we need the OCI-runtime binary (runc) and the CNI plugins binaries
(bridge, host-local).

We can either download the pre-built binaries from the repos or we can build
the binaries from source

From the result output, one thing that is a bit confusing is the DNS result
entry. CNI plugin do not actually apply the DNS entry (aka. write the
/etc/resolv.conf). The plugins simply return the value, and the Container
Runtime is expected to apply the DNS server.

Take Away Notes

CNI is about all the network-related actions that take place during the
creation or the deletion of a container, it is not about realtime changes (e.g.
policy enforcement). CNI will create all the rules to reassure a connection
from/to a container, but it is not responsible to set up the network medium,
e.g. bridge creation or distribute routes to connect containers located in
different hosts. CNI targets to hide the network complexity in order to make
the runtime code-base cleaner and in the same time to enable third party
providers to create their own plugins and integrate them easily to all
container orchestrators that are using a CNI compatible runtime.