Live Reconfiguration

Debugging etcd

Migration

Getting started with etcd

etcd is an open-source distributed key value store that provides shared configuration and service discovery for Container Linux clusters. etcd runs on each machine in a cluster and gracefully handles leader election during network partitions and the loss of the current leader.

Application containers running on your cluster can read and write data into etcd. Common examples are storing database connection details, cache settings, feature flags, and more. This guide will walk you through a basic example of reading and writing to etcd then proceed to other features like TTLs, directories and watching a prefix. This guide is way more fun when you've got at least one Container Linux machine up and running — try it on Amazon EC2 or locally with Vagrant.

Setting up etcd

Container Linux's etcd-member.service systemd unit knows how to fetch and run the current etcd v3.x container image, providing etcd v3 without requiring the binary to be present in every default OS installation.

This is the human-readable config file. This should not be immediately passed to Container Linux. Learn more.

# This config is meant to be consumed by the config transpiler, which will# generate the corresponding Ignition config. Do not pass this config directly# to instances of Container Linux.etcd:name:my-etcd-1listen_client_urls:https://10.240.0.1:2379advertise_client_urls:https://10.240.0.1:2379listen_peer_urls:https://10.240.0.1:2380initial_advertise_peer_urls:https://10.240.0.1:2380initial_cluster:my-etcd-1=https://10.240.0.1:2380,my-etcd-2=https://10.240.0.2:2380,my-etcd-3=https://10.240.0.3:2380initial_cluster_token:my-etcd-tokeninitial_cluster_state:new

This is the raw machine configuration, which is not intended for editing. Learn more. Validate the config here.

Reading and writing from inside a container

To read and write to etcd from within a container you must use the IP address assigned to the docker0 interface on the Container Linux host. From the host, run ip address show to find this address. It's normally 172.17.0.1.

To read from etcd, replace 127.0.0.1 when running curl in the container:

Proxy example

Let's pretend we're setting up a service that consists of a few containers that are behind a proxy container. We can use etcd to announce these containers when they start by creating a directory, having each container write a key within that directory and have the proxy watch the entire directory. We're going to skip creating the containers here but the docker guide is a good place to start for that.

Create the directory

Directories are automatically created when a key is placed inside. Let's call our directory foo-service and create a key with information about a container:

Watching the directory

Now let's try watching the foo-service directory for changes, just like our proxy would have to. First, open up another shell on a Container Linux host in the cluster. In one window, start watching the directory and in the other window, add another key container2 with the value localhost:2222 into the directory. This command shouldn't output anything until the key has changed. Many events can trigger a change, including a new, updated, deleted or expired key.

Watching the directory and triggering an executable

Now let's try watching the foo-service directory for changes and - if there are any - run the command. In one window, start watching the directory and in the other window, add another key container3 with the value localhost:2222 into the directory. This command shouldn't trigger anything until the key has changed. The same events as in the previous example can trigger a change. The exec-watch command expects etcdctl to run continuously (for watch command you can use --forever option)

Test and set

etcd can be used as a centralized coordination service and provides TestAndSet functionality as the building block of such a service. You must provide the previous value along with your new value. If the previous value matches the current value the operation will succeed.