Docker container DNS resolution issue in air-gapped network

Problem

The scenario is where docker containers are running on a host system with Ubuntu, which is connected in an air gapped network, with the router configured with a list of local DNS servers. The host doesn’t have any manual DNS entries, though it is able to make DNS queries via the LAN’s upstream DNS servers. Containers running on Bridge mode, on the other hand, are trying to resolve from 8.8.8.8 instead of using LAN’s DNS. The expected behaviour is that containers should resolve from the LAN’s upstream servers.

Analysis

Here is a bit of related information to understand how Ubuntu DNS resolution works & how docker container’s DNS configuration is being handled:

In short, systemd-resolved daemon also acts as a DNS server by listening on IP address 127.0.0.53 on the local loopback interface. lo is hardcoded, it cannot listen on any other network interfaces such as docker0, eth0,en0, etc.

Any non-local DNS queries, in our case, will be forwarded to the upstream DNS server of the LAN. System gets this DNS server details during the DHCP connection, which will be updated in /etc/resolv.conf file dynamically.

/etc/resolv.conf file shouldn’t be edited manually, rather it should be symlinked to one of the following files as per your use-case, where you can specify hardcoded nameservers if there any:

/run/systemd/resolve/stub-resolv.conf

/usr/lib/systemd/resolv.conf

/run/systemd/resolve/resolv.conf

Container DNS

When docker containers are started in Bridge mode, docker daemon copies any non-localhost entries of /etc/resolv.conf, /etc/hosts and /etc/hostname files from the host to the container; if no non-localhost entries are found, container’s /etc/resolv.conf will be initialized with hardcoded nameserver 8.8.8.8.

fig 1: Docker DNS issue in air-gapped network

Whenever the host machine receives new DNS configuration over DHCP, it will be updated to the docker container once it gets restarted. When upstream DNS configuration changes, there is no way the running containers to know about the changes.

Solution

Hardcoding nameservers in instance doesn’t solve the issue, because whenever a nameserver change is required, it needs to be updated in every system of the network. All running docker container instances need to be restarted as well in order to get the updated DNS list.

One solution for this problem would be to enable host system to listen on docker interface apart from the local loopback interface. i.e., forwarding all DNS queries from the docker containers to the host’s 127.0.0.53.

Since systemd-resolved cannot listen on docker0 interface, we can use dnsmasq to bind to docker interface. dnsmasq will accept queries from the containers and relay to the systemd-resolvd.

$ sudo apt-get install dnsmasq

/etc/dnsmasq.conf

interface=docker0bind-interfaces

We also need to configure docker daemon to point all container DNS queries to forward to the docker host system.