Each member of a swarm is considered a node. In this tutorial, each instance will be a swarm node, one as the master node and the two others as worker nodes. You'll submit service definitions to the master node, which will dispatch work to the worker nodes.

A service is the definition of the tasks to execute on swarm nodes. In this tutorial, you'll define three services, each starting a CockrochDB node inside a container and joining it into a single cluster. Each service also ensures a stable network identity on restart via a resolvable DNS name.

Step 3. Start the swarm

Take note of the output for docker swarm init as it includes the command you'll use in the next step. It should look like this:

copy

icon/buttons/copy

$ sudo docker swarm init --advertise-addr 10.142.0.2

Swarm initialized: current node (414z67gr5cgfalm4uriu4qdtm) is now a manager.
To add a worker to this swarm, run the following command:
$ docker swarm join \
--token SWMTKN-1-5vwxyi6zl3cc62lqlhi1jrweyspi8wblh2i3qa7kv277fgy74n-e5eg5c7ioxypjxlt3rpqorh15 \
10.142.0.2:2377
To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

These commands each create a service that starts a container, joins it to the overlay network, and starts a CockroachDB node inside the container mounted to a local volume for persistent storage. Let's look at each part:

sudo docker service create: The Docker command to create a new service.

--replicas: The number of containers controlled by the service. Since each service will control one container running one CockroachDB node, this will always be 1.

--name: The name for the service.

--hostname: The hostname of the container. It will listen for connections on this address.

--mount: This flag mounts a local volume with the same name as the service. This means that data and logs for the node running in this container will be stored in /cockroach/cockroach-data on the instance and will be reused on restart as long as restart happens on the same instance, which is not guaranteed.

Note:

If you plan on replacing or adding instances, it's recommended to use remote storage instead of local disk. To do so, create a remote volume for each CockroachDB instance using the volume driver of your choice, and then specify that volume driver instead of the volume-driver=local part of the command above, e.g., volume-driver=gce if using the GCE volume driver.

--stop-grace-period: This flag sets a grace period to give CockroachDB enough time to shut down gracefully, when possible.

--publish: This flag makes the Admin UI accessible at the IP of any instance running a swarm node on port 8080. Note that, even though this flag is defined only in the first node's service, the swarm exposes this port on every swarm node using a routing mesh. See Publishing ports for more details.

cockroachdb/cockroach:v2.0.2 start ...: The CockroachDB command to start a node in the container in insecure mode and instruct other cluster members to talk to each other using their persistent network addresses, which match the services' names.

The service definitions tell the CockroachDB nodes to log to stderr, so if you ever need access to a node's logs for troubleshooting, use sudo docker logs <container id> from the instance on which the container is running.

Now all the CockroachDB nodes are running, but we still have to explicitly tell them to initialize a new cluster together. To do so, use the sudo docker run command to run the cockroach init command against one of the nodes. The cockroach init command will initialize the cluster, bringing it into a usable state.

Step 7. Monitor the cluster

It's possible to access the Admin UI from outside of the swarm because you published port 8080 externally in the first node's service definition.

On this page, verify that the cluster is running as expected:

View Node list to ensure that all of your nodes successfully joined the cluster.

Click the Databases tab on the left to verify that insecurenodetest is listed.

Step 8. Simulate node failure

Since we have three service definitions, one for each node, Docker swarm will ensure that there are three nodes running at all times. If a node fails, Docker swarm will automatically create another node with the same network identity and storage.

To see this in action:

On any instance, use the sudo docker ps command to get the ID of the container running the CockroachDB node: