Live Reconfiguration

Debugging etcd

Migration

Enabling HTTPS in an existing etcd cluster

This guide outlines the process of migrating an existing etcd cluster from HTTP communication to encrypted HTTPS. For added security, it also shows how to require TLS peer certificates to authenticate connections.

Prepare cluster components

Check insecure port availability

By default, etcd communicates with clients over two ports: 2379, the current and official IANA port designation, and 4001, for clients who may implement versions of the protocol older than 0.4. We leverage this quirk of legacy support to migrate a running cluster from insecure plain-HTTP communication (on the old port, 4001) to encrypted HTTPS (on the current port, 2379) without cluster downtime. We restrict the legacy communication on port 4001 to the local interface.

If you've configured flannel, fleet, or other components to use custom ports, or 2379 only, they will be reconfigured to use port 4001.

If etcd isn't listening on port 4001, it must also be reconfigured. If you used a Container Linux Config to spin up your machines, you can retrieve the --listen-client-urls value from /etc/systemd/system/etcd-member.service.d/20-clct-etcd-member.conf to verify the etcd ports:

In this case etcd is listening only on port 2379. Add port 4001 with a systemd drop-in unit file. Edit the line that starts with --listen-client-urls in the /etc/systemd/system/etcd-member.service.d/20-clct-etcd-member.conf file and append the new URL on port 4001 to the existing value retrieved in the previous step:

--listen-client-urls="http://0.0.0.0:2379,http://127.0.0.1:4001"

Run systemctl daemon-reload followed by systemctl restart etcd-member.service to restart etcd. Check cluster status using the etcdctl commands:

$ etcdctl member list
$ etcdctl cluster-health

Repeat these steps on each etcd cluster member.

Generate TLS key pairs

Copy key pairs to nodes

Assume we have three Container Linux machines, running three etcd cluster members: server1, server2, server3; with corresponding IP addresses: 172.16.0.101, 172.16.0.102, and 172.16.0.103. We will use the following key pair file names in our example:

server1.pem
server1-key.pem

Create the /etc/ssl/etcd directory, then copy the corresponding certificate and key there. Set permissions to secure the directory and key file:

Alternatively, copy ca.pem into /etc/ssl/certs instead, and run the update-ca-certificates script to update the system certificates bundle. After doing so, the added CA will be available to any program running on the node, and it will not be necessary to set the CA path for each application.

Repeat this step on the rest of the cluster members.

Using an etcd proxy

If you typically connect to a remote etcd cluster, this is a good time to configure an etcd proxy that handles the remote connection and TLS termination, and to reconfigure your apps to communicate through the proxy on localhost. In this case, you must generate a client key pair (e.g. client1.pem and client1-key.pem) and repeat the Copy Key Pairs step above with the client key pair files.

It is also necessary to modify your systemd unit files or drop-ins which use etcdctl in ExecStart*= or ExecStop*= directives to replace the invocation of /usr/bin/etcdctl with /usr/bin/etcdctl --no-sync. This will force etcdctl to use the proxy for all operations.

Configure etcd key pair

Now we will configure etcd to use the new certificates. Create a /etc/systemd/system/etcd-member.service.d/30-certs.confdrop-in file with the following contents:

On any one etcd cluster member, run each of the printed commands – except the last one. We will change the peer URL for the last etcd cluster member only after completing any proxy configuration below.

Check cluster health:

$ etcdctl member list
$ etcdctl cluster-health

If the cluster members report as expected, we can move on to configuring the local etcd proxy, if neeeded, then invoking the last of the printed command lines on a cluster member.

etcd proxy

An operating etcd proxy will automatically adopt new peer URLs within 30 seconds of each update (that is, the default [--proxy-refresh-interval][proxy-refresh] 30000).

etcd versions 2.1 and older

For etcd versions 2.1 and earlier, the awk filter to produce the peer URL change commands is different:

Apply the changes in the same manner described above, by running each of the printed commands, except the last one, on any one etcd cluster member. Finally, invoke the last printed command after completing any etcd proxy configuration in the previous section.

Change etcd client URLs

Edit the lines that start with --listen-client-urls, --advertise-client-urls, and --listen-peer-urls in the /etc/systemd/system/etcd-member.service.d/20-clct-etcd-member.conf file and append the new URL on port 4001 to the existing value retrieved in the previous step: