Deploying Mesos and Marathon on an Arm Cluster

In this post we'll walk through setting up a cluster of Arm machines. We'll use Mesos as the cluster manager, Marathon for container orchestration, Zookeeper as a key-value store, and Weave Net for the container overlay and DNS solution. The high level steps are listed below.

Compile Mesos.

Configure Zookeeper.

Install Weave Net.

Compile Marathon.

Setup Zookeeper.

Setup Weave Net.

Setup the Mesos cluster.

Setup Marathon.

Run a few containers via Marathon and confirm connectivity.

Hardware Setup

The instructions below were performed on two Softiron Overdrive 1000 computers running Ubuntu. Each machine has Quad-core A-57 AArch64 CPUs. The master node will be called od1000-1, and the agent node will be called od1000-0. A Mesos agent will run on the master node so that tasks can be scheduled on it as well. Running a Mesos agent on a master node can introduce security vulnerabilities on the cluster. It may not be wise to do this on production systems, but for the purpose of this blog, it's ok.

Aside from the Overdrive 1000, Mesos and Marathon have been deployed on other server and networking platforms like the Cavium ThunderX1 (48 AArch64) and Marvell MacchiatoBin (Quad-core A-72 AArch64). The instructions below should work on just about any AArch64 hardware.

The next step is different from the standard instructions posted on the Mesos website. The build will have to be executed twice. The first time around, it will create a number of config.guess files and then fail. The build will fail because these config.guess files aren't compatible with Arm. These files will need to be replaced with Arm compatible versions of the file. Once they are replaced, the build can be executed a second time, which will result in a successful compile.

To understand this further, first run the following command from the Mesos source root.

After all of the files are replaced, run the make command from above in $MESOS_ROOT/build again.

Once Mesos is compiled, the next step is to install Mesos.

The install command will copy the Mesos binary to /usr/local/bin/. Another important file that gets installed is libmesos. This will be copied to /usr/local/lib. libmesos is loaded by Marathon to interact with Mesos.

The ldconfig command will update the shared object caches so that the run-time linker can find libmesos.

Prepare Zookeeper

The Mesos source comes with a Zookeeper Java Archive (.jar) already included. Untar Zookeeper and create the config file by making a copy of the sample config that's included. Note the version of Zookeeper provided may be different from what's shown below.

Compiling Marathon Source

Once Marathon is compiled, it will start running automatically. There will be repeating connection failed errors that should look similar to the image below. At this point, kill Marathon with <ctrl-C>.

Cluster Setup

Starting Zookeeper

A single instance of Zookeeper for both Mesos and Marathon will be used. Zookeeper only needs to run on a single node. The master (od1000-1) will be used in this case.

Note that the peer names should be host names that can be found via DNS. IP addresses can also be used instead of host names. More than one peer connection can be made on a node. This is done by passing the peer names into the weave launch command like below.

The resources of this node should appear in the Mesos UI after running the above.

When the agent node is started, there are a few switches that are required.

--containerizers: This switch enables Mesos to translate its configs into Docker CLI options.

--resources: In Mesos, ports are considered resources just like CPUs, RAM, and disk space. When the agent is started, the ports that can be offered as resources must be specified. In the command above, all ports are offered as resources. This is ok for this blog, but offering all ports shouldn't be done in production. Make sure to only offer ports that are actually needed.

--docker_socket: Will tell Mesos to point the Docker client (i.e. Docker CLI command) towards the Weave Net Docker API proxy. Normally the Docker client will talk directly to the Docker Daemon, but since Weave will provide a DNS service, Mesos needs to issue Docker commands to the Weave Net proxy. This proxy allows for containers to get registered with WeaveDNS when they are created. After registration, the Weave Net proxy will forward the Docker command to the Docker daemon. Without this switch, WeaveDNS will not work.

Start Marathon on Master Node

Once the Mesos master and agents have been established, start Marathon.

MESOS_NATIVE_JAVA_LIBRARY: Tells Marathon where to find libmesos. This variable can be placed in ~/.profile to avoid having to set it in the shell that will run Marathon.

The JSON file describes two Marathon application groups (group1 & group2). Each group will run a single Marathon application. Each application will run two instances of an Ubuntu container that calls sleep. Each container instance uses .25 CPUs, 128MB of RAM, and 1GB of disk space. Thus, to run the 4 containers described in the JSON file, Mesos will require that 1 CPU, 512MB of RAM, and 4GB of disk space be available on the cluster. Mesos also requires that the requested resources of a single container instance (.25 CPUs, 128MB RAM, and 1GB Storage) must all be available on a single host for it to deploy on that host.

To support the Weave Net overlay, notice that a hostname parameter is set with the domain .weave.local. This is required so that WeaveDNS registers the host name. Without this, the containers cannot discover each other via host/service name.

Last, the sequence in which the containers startup is controlled by the group dependencies. In the example file above, /arm-demo/group2 is dependent on /arm-demo/group1. When this file is posted to Marathon, the containers in group1 will start up before the containers in group2.

Deploy and Confirm Connectivity

The Marathon HTTP API is used to request Mesos to startup the containers.

(master_ip) is the IP address or host name of the node (od1000-1) running Marathon.

After running this command, take a look at the Marathon UI. After a minute or two, the containers should show as running. The containers should startup quickly because their base image was downloaded in the prerequisites section of this post.

Open the Mesos UI. It should also show that the four containers are running.

Run docker ps on each node to see the containers.

OD1000-1 (master):

OD1000-0:

There are three containers running on each node. Two Ubuntu 16.04 containers and a Weave Net Router.

Notice that the container instances in each group have split across the two nodes of the cluster. That is, there's one ubuntu1 (group1) container on od1000-0, and the other is on od1000-1. The same is true for the two container instances of application group2. This is desirable, as it provides fault tolerance for each group.

Confirm that WeaveDNS has registered all four containers.

All four containers have registered with WeaveDNS. The containers with IP addresses of 10.40.x.x are on one node, while the containers with IP addresses of 10.32.x.x are on the other. The subnet for this overlay is 10.32.0.0/12, so these containers are all on the same VXLAN subnet as expected.

So far, it looks like there should be connectivity between containers. To be sure, confirm connectivity from the perspective of one of the containers.

Shell into a container on node od1000-1.

(container_hash) is the hash value of the container to shell into. It can be obtained by running the docker ps command.

From within this ubuntu2 instance, check DNS by running nslookup against host names ubuntu1 and ubuntu2.

There are two IP address entries for each host name as expected.

If nslookup is run more than once with the same host name, the order of reported IP addresses will change. This is the built in load balancing feature of WeaveDNS.

Ping all IP addresses from this ubuntu2 instance to confirm connectivity to all other containers.

Given the above, connectivity and DNS is confirmed across the overlay network.

Closing Remarks

We've proved that Mesos, Marathon, Zookeeper, and Weave Net are all functioning properly in an Arm cluster. With the exception of Weave Net, these are all components of the Datacenter Operating System (DC/OS) open source project. Given the above results, it's very likely that DC/OS will run on Arm without issues as well. Perhaps exploring DC/OS will be the subject of a future blog post. Last, aside from the special handling of the config.guess files when compiling Mesos, all of the instructions presented above are architecture agnostic. This is what's desired for a heterogeneous data center.

We encourage readers to get involved with the Works on Arm project. This project aims to further develop the Arm Software ecosystem in the data center. The idea of the Works on Arm project is to take Open Source projects, deploy them on Arm platforms, and debug any functional and performance issues found in the process. Much like what we've done above with Mesos/Marathon, all it takes to get started is to build and run the SW as intended by its developers.