Upgrading Kubernetes

Building on Open Source

Other Resources

Kubernetes network troubleshooting on CoreOS

Kubernetes networking issues can be debugged with familiar tools, once the isolated nature of the container abstraction is taken into account. This document explains some of the best places to start troubleshooting when network issues arise.

This one-liner finds the pod in the kube-system namespace whose k8s-app property is kube-dns, and uses xargs to format that pod's name for a kubectl port-forward command. The result is that local port 5300 is forwarded to the pod's port 53. This allows host DNS tools to look up Kubernetes hostnames at your machine's port 5300:

Debugging Docker bridge and other host networking issues on CoreOS

If you suspect the issue is actually with the host's networking, it may seem frustrating that CoreOS does not include some of the standard network utilities like tcpdump or nmap. However, CoreOS provides the toolbox, a special container that can install and run a complete userland, without needing everything installed on the base system.

Run the command toolbox and CoreOS will launch a container from the "Fedora" image, downloading it first if necessary. This container is executed with all kernel capabilities, mounts local filesystems for inspection, and attaches directly to the host network. After running the toolbox command, you will be at a shell inside this privileged container.

Once inside the toolbox container, install the desired tools:

# dnf install -y [package] [package] [package]...

Run debugging tools in the toolbox:

# tcpdump -i docker0

Exit the toolbox container by hitting Ctrl+] three times to stop it, leaving the base system unchanged. The downloaded container image will remain in the local image store on disk, but can be manually removed.

Debugging Kubernetes pod issues

Many pod network issues come down to container networking. Access to the container's network namespace is needed to troubleshoot these issues. Gaining access to a container namespace boils down to finding the container ID, and sometimes mapping it to the process ID running inside the container.

Finding the ID of the right container

You can get the ID of the container(s) in a pod with kubectl describe:

For rkt containers:

Attach to rkt containers by using the namespace entry command, nsenter, to invoke rkt and another container with the --net=host option. This new container thinks it is attaching to the host network, but the "host" network it sees is actually the network namespace of the target container. To find a target process ID in a rkt container:

$ rkt status [container ID]

Once you have the target process ID, spawn the rkt debugging container, passing the target PID to nsenter: