My Life as a Sys Admin

Category Archives: Mesos

In my previous blog, i’ve explained how to manage Docker cluster using Mesos and Marathon. Now Mesos has released 0.20 with Native Docker support. Till Mesos 0.19, we had to use mesosphere’s Deimos for running Docker containers on Mesos slaves. Now from Mesos 0.20, Mesos has the ability to run Docker containers directly. As per Mesos Documentation we can run containers in two ways. 1) as a Task and 2) as an Executor. Currently Mesos 0.20 supports only the host (–net=host) Docker networking mode.

Install the latest Mesos 0.20 using the mesosphere’s Mesos Debian package. Once we have setup the Zookeeper and Docker, we need to make a few changes to enable the Mesos Native Docker support. We need to start all the Mesos-Slave with --containerizers=docker,mesos flag. For enabling this flag, we need to create a file containerizers in /etc/mesos-slave/containerizers with content “docker,mesos”. Mesos slave process, when starting up, will read the folder and enable this flag.

I was trying to start a container with Port 9999 via Marathon REST API. But the task was keep on failing. Up on investigating the logs, i found that the slave when sends the allocatable resource details to mesos master was total allocatable: cpus():0.8; mem():801; disk():35164; ports():[31000-31099, 31101-32000]. So the allowed port range was [31000-31099, 31101-32000]. So if we wnat to use any other custom port, we need to define the same in the /etc/mesos-slave folder by creating a config file.

Now, my new allocatable resource details sent to the Mesos master became total allocatable: ports():[31000-31099, 31101-32000, 9998-9998]; cpus():0.7; mem():801; disk():35164. The above config is crucial if we want to allocate our own custom range of ports. So this gave a good idea on how to control my resource allocation by creating custom configuration files. Now let’s restart Mesos-slave process.

Now we have the Mesos slave running with Native Docker support. But the current stable version of Marathon still don’t support the native Docker feature of Mesos. So we need to setup Marathon 0.7 from scratch. I’ve written a blog on upgrading Marathon from 0.6.x to 0.7. Once we have setup the Marathon 0.7, we can start using Mesos with the Native Docker support. Let’s create a new App in Marathon via its REST API.

With the new Native Docker support, Mesos is becoming more friendly with Docker. And this is a great achievement for people like me who are trying to use Mesos and Docker in our Infrastructure. A big kudos to Mesos and Mesosphere team for their great work 🙂

A few days ago Mesos 0.20 version was released with native Docker support. Till that, Mesos Docker integration was performed using Mesosphere’s Deimos application. But with the new Mesos, there is no need for any external application for launching Docker containers in Mesos Slaves. As per Mesos Documentation we can run containers in two ways. 1) as a Task and 2) as an Executor. Currently Mesos 0.20 supports only the host (--net=host) Docker networking mode.

The current stable release of Marathon is 0.6X. The debian package provided by the Mesosphere also provides the 0.6x version. The container format has been completely changed in 0.7.x version. So we cannot use the 0.6.x version along with Mesos 0.20. More details are available in the Marathon Documentation

The current Mesos Master branch is version := "0.7.0-SNAPSHOT". So we need to compile Marathon from scratch. The only dependency for compiling Marathon is scala-sbt. Get the latest Debian package for sbt from here. The current sbt version is 0.13.5.

The above build command will create an executable jar file “marathon-runnable.jar” under the “target” folder. We can use this jar file along with Marathon 0.6.x start script. If we check the upstart script of Marathon 0.6.x, it uses /usr/local/bin/marathon binary for starting the service. So we need to edit two lines in this file to use our latest version 0.7’s jar file.

First let’s move our jar file to say “/opt/ folder.

$ cp marathon-runable.jar /opt/marathon.jar

Now let’s edit the Marathon binary. Make the below changes in the binary file,

marathon_jar="/opt/marathon.jar" # Line number 21
and
exec java "${vm_opts[@]}" -jar "$marathon_jar" "$@" # -jar option added to use the jar file instead if the default -cp option. Line number 62

Now Marathon is running with the version 0.7 jar file. Let’s verify by creating a new Docker task. Make sure that the Mesos version 0.20 is running on the host and not the older versions of Mesos. More detail about the new container format is available in the Marathon Documentation page.

Here i’ve used Port number 9999. But by default, in Mesos 0.20, default port range resourced by Mesos Slave is [31000-32000]. Please refer my next Blog post on how to modify and set a custom port range.

With the new Native Docker support, Mesos is becoming more friendly with Docker. And this is a great achievement for people like me who are trying to use Mesos and Docker in our Infrastructure. A big kudos to Mesos and Mesosphere team for their great work 🙂

It’s almost 3 months since i’ve started with Mesos + Marathon. Yesterday, i was going through the Marathon Doc Site, and found that Marathon has a cool internal Event Bus. Currently Marathon has an ibuilt HTTP callback subscriber that POSTs events in JSON format to one or more endpoints. So i decided to give it a try to my Mesos test environment. More documentations of the Event Bus are available in the Documentation page.

Currently, in my test environment all the Mesos and Marathon are installed from the Mesosphere Debian packages. Whether we use the packaged init or upstart script, both of them are directly calling the /usr/local/bin/marathon binary. For the Event Bus with http callback, we need to enable two flags (–event_subscriber http_callback –http_endpoints http://host1/foo,http://host2/bar) while starting the Marathon service ie, –event_subscriber, the type of subscriber that we are going to use and –http_endpoints, endpoints corresponding to the subscriber.

From the marathon binary, i found the it looks for files under the /etc/marathon/conf/ folder, where each file is the name of the flag to be enabled and the content of the file is the flag values. So in our case, we need two files inside the “/etc/marathon/conf/”, 1) event_subscriber and 2) http_endpoints

From the above ps command, it can be seen that the flags are enabled properly. Now i’m going to start a docker container. So as per the Event Bus documentation, Callbacks are Fired every time Marathon receives an API request that modifies an app (create, update, delete)

This Event bus is really usefull as it notifies us on all the Event changes happening on our Mesos cluster. We can also buit an Event notification system based on these callbacks. Though the current Events has very minimal details, i’m sure that more Event types will get added soon into Marathon.

In my previous blog, i’ve described how to manage Docker cluster using Mesos and Marathon. It was using a single Mesos Master. But for production, we cannot go with a single Mesos master, as it will result in a sinlge point of failure. Mesos supports Multi master via Zookeeper. So in this blog i’m going to explain how to setup a Multi Master Mesos Cluster using Zookeper for Automatic promotion of Mesos master when the current Active Mesos Master fails. This time i’m going to setup 3 Zookeeper services and 2 Mesos Master on a single Vagrant box.

Here, since the 3 zookeeper services are running on the same host, i’ve assigned separate ports. If the zookeeper are running on separate instances, then we can have the same ports for all of the zookeeper nodes. There are two port numbers. The first followers use to connect to the leader, and the second is for leader election.

Now we can start the Zookeeper in Foreground using the ./bin/zkServer.sh start-foreground from each of the 3 zookeeper folders. netstat -nltp | grep 288 will display the port of the zookeeper service which is the current master among the cluster. We can also check the connectivity using the zookeepr client binary available in the zookeeper source folder. Once zookeeper cluster is UP, we can go ahead setting up Mesos.

Setting up Multiple Mesos Masters

Mesossphere team has already built packages for Mesos,Marathon and Deimos. So one of my Mesos master will be from the package. But i also wanted to play with the Mesos Source, so i decided to build Mesos from the source. So my second Mesos Master will be from the scratch.

Now we need to define the Zookeeper cluster details so that Mesos master and slave can connect. Edit /etc/mesos/zk and add zk://localhost:2181,localhost:2182,localhost:2183/mesos. More details about Zookeeper URL is available here. Also More details about installing Mesos from Mesosphere’s Debian package is available in their website.

So now we have one master ready, we can start the Mesos Master, Mesos Slave and Marathon. ANd esnure that they can connect to our Zookeeper cluster properly. We can also see the connection status in the stdout of the zookeeper process as they are running in foreground.

Now building the second Mesos master from scratch. First let’s download the Mesos Source Code.

This will launch a single container. We can check the status of the container via the Marathon UI as well as via RestAPI also. Now comes the critical part for Production, What happens when the master fails. Mesos Documentation says, if the current master fails, in a Multi master Mesos Cluster, Zookeeper will elect a new Master, in our case the second Mesos master service. And Mesos Claims that the running services will not get crashed and the status will be taken by the newly promoted master. In our case, we have a Docker container running as a long running service.

First i’ll stop one of the Mesos master service,

$ service mesos-master stop

Now immidiately on the stdout of the Zookeeper, we can see the logs corresponding to the Election process. Below is the same,

As per the above logs, the second Mesos Master running at port 5054 was promoted as New Mesos Master. This will be reflected to Marathon also. We can go to the Marthon UI and make sure that the Docker process that we have started using the old Mesos master is still running fine. We can even try making some scaling changes and can make sure that the new master can alter the process.

In my testing, i’ve tried stoping each of the service and ensured that only one Mesos master is running and also tried scaling the Apps from one Master service to make sure that the new Master was able to keep track of all these changes. The results were quite promising. Mesos + Marrthon + Docker indeed is killer combo. We can really built a cross vendor independent cluster with scaling capabilites.

Docker has became one of my favourite tool. It’s super cool and super easy tool to manage linux containers. LXC’s are around in IT world for some time, but by the entry of Docker last year, the wave started rising. Thanks to Docker team and Solomon Hykes for open sourcing such a wonderfull project. I’ve already mentioned a lot of stuffs about Docker in my previos blogs, so today im going explain how Docker can be used as a Cluster. There are some interesting tools like CoreOS, Helios etc for managing Docker as a cluster. But today i’m going to explain on how to set up a Docker cluster using Apache Mesos. CoreOS is a custom linux os which comes with SystemD. But the restriction is, we have to use that custom images of coreos.. Indeed CoreOS team open sourced some exciting tools like etcd fleet which works with CoreOS for managing Docker clusters. But Mesos is quite simple, we can install it via package, or even using tar balls available in thier Github repo onto most of the Linux Distro’s and it’s quite easy to configure also. Mesos is heavily used by Twitter to manage their data center’s. And now Mesosphere has opensourced a new tool called Mararthon which now provides a UI and a Rest API for maaging and scheduling Mesos Frameworks aka jobs, in this case containers as a service.

A few weeks ago, Mesos 0.19 was released which comes with an official support for Docker coantiners by intergrating Deimos into it. And a few days ago Marathon has released their new version 0.6.0 supports launching any task in a Docker container via Mesos 0.19+

Setting up Mesos Cluster

In this test setup, i’m going to setup both Mesos master/slave and Zookeeper on the same Ubuntu 14.04 vagrant node. First we can install the dependencies,

Marathon will now start listening to port 8080, We can access the UI from the browser via this port, also via rest API using the same port.

curl localhost:8080/help # gives us some details about the API's

I just went through the Deimos code, so under the hood they are using docker run with some default parameters like --sig-proxy, --rm, --cidfile, -v, -w and extra parameters that we are passing while creating the task via Marathon.

As of now, we still can’t pass details like Container image, Docker options via Marathon GUI. So we can use the Rest API for the time being. Below is a sample curl request for launcing a single container,

We can pass custom options to the docker run command via “options”. After making the curl request, we can check the syslog, as mesos will be logging into syslog by default. We can even see the Docker run command on the same.

A more detailed report about the Docker job which we have launched can be viewed via the default Mesos GUI listening on port 5050 on the Mesos master. Now we can test the scalability of the Job. Currently we have only one container running. So now we can try scaling say adding one more node. We can do it in two ways, like via PUT request using curl or using GUI

Now we can use the docker ps command to see if the new container is launched or not. Also we can see that status in UI also.

Similarly, we can scale down also. I’ve tested the same and all seems to be good. Marathon ensures that the docker process will be running. So incase if the process crashes Marathon will restart the same and ensures that the instances are up and running as per our configuration. There are a few other Open Sourced Mesos Scheduler’s like Apache Aurora, Airbnb’s Chronos. But for my requirement marathon is pretty straight and simple and also provides a very good Rest API layer for managing containers. Mesos, Marathon and Docker are still young, but provides a killer combination for managing clusters built over Docker containers.