Category: Docker Compose

Docker CE 19.03.0 Beta 1 went public 2 week back. It was the first release which arrived with sysctl support for Docker Swarm Mode for the first time. This is definitely a great news for popular communities like Elastic Stack, Redis etc. as they rely on tuning the kernel parameter to get rid of memory exceptions. For example, Elasticsearch uses a mmapfs directory by default to store its indices. The default operating system limits on mmap counts is likely to be too low, which may result in out of memory exceptions, hence one need to increase the limits everytime using sysctl tool. Great to see that Docker Inc. acknowledged the fact that kernel tuning is required sometimes and provides explicit support under Docker 19.03.0 Pre-Release. Great Job !

Wait..Do I really need sysctl?

Say, you have deployed your application on Docker Swarm. it’s pretty simple and it’s working great. Your application is growing day by day and now you just need to scale it. How are you going to do it? The simple answer is: docker service scale app=<number of tasks>.Surely, it is possible today but your containers can quickly hit kernel limits. One of the most popular kernel parameter is net.core.somaxconn. This parameter represents the maximum number of connections that can be queued for acceptance. The default value on Linux is 128, which is rather low.

The Linux kernel is flexible, and you can even modify the way it works on the fly by dynamically changing some of its parameters, thanks to the sysctl command. The sysctl programs allow to limit system-wide resource use. This can help a lot in system administration, e.g. when a user starts too many processes and therefore makes the system unresponsive for other users. Sysctl basically provides an interface that allows you to examine and change several hundred kernel parameters in Linux or BSD. Changes take effect immediately, and there’s even a way to make them persist after a reboot. By using sysctl judiciously, you can optimize your box without having to recompile the kernel and get the results immediately.

Please note that Not all sysctls are namespaced as of Docker 19.03.0 CE Pre-Release. Docker does not support changing sysctls inside of a container that also modify the host system.

Docker does support setting namespaced kernel parameters at runtime & runc honors this. Have a look:

Method:I

Downloading the static binary archive. Go to https://download.docker.com/linux/static/stable/ (or change stable to nightly or test), choose your hardware platform, and download the .tgz file relating to the version of Docker CE you want to install.

Testing with hello-world

Captain'sBay==>sudo docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
1b930d010525: Pull complete
Digest: sha256:2557e3c07ed1e38f26e389462d03ed943586f744621577a99efb77324b0fe535
Status: Downloaded newer image for hello-world:latest
INFO[2019-04-10T09:26:23.338596029Z] shim containerd-shim started address="/containerd-shim/m
oby/5b23a7045ca683d888c9d1026451af743b7bf4005c6b8dd92b9e95e125e68134/shim.sock" debug=false pid=2953
Hello from Docker!
This message shows that your installation appears to be working correctly.
To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
(amd64)
3. The Docker daemon created a new container from that image which runs the
executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
to your terminal.
To try something more ambitious, you can run an Ubuntu container with:
$ docker run -it ubuntu bash
Share images, automate workflows, and more with a free Docker ID:
https://hub.docker.com/
For more examples and ideas, visit:
https://docs.docker.com/get-started/
## Verifying the new `docker context` command

Creating a 2-Node Docker Swarm Mode Cluster

swarm-node-1:~$ sudo docker swarm init --advertise-addr 10.140.0.6 --listen-addr 10.140.0
.6:2377
Swarm initialized: current node (c78wm1g99q1a1g2sxiuawqyps) is now a manager.
To add a worker to this swarm, run the following command:
docker swarm join --token SWMTKN-1-1bc88158q1v4b4gdof8k0u532bxzdvrgxfztwgj2r443337mja-cmhuu258lu0327
e32l0g4pl47 10.140.0.6:2377
To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

Running Multi-service Docker Compose for Redis

Redis is an open source, in-memory data structure store, used as a database, cache and message broker. Redis Commander is an application that allows users to explore a Redis instance through a browser. Let us look at the below Docker compose file for Redis as well as Redis Commander shown below:

Last week I attended Dockercon 2018 EU which took place at Centre de Convencions Internacional de Barcelona (CCIB) in Barcelona, Spain. With over 3000+ attendees from around the globe, 52 breakout sessions, 11 Community Theatres, 12 workshops, over 100+ total sessions, exciting Hallway Tracks & Hands-on Labs/Trainings, paid trainings, women’s networking event, DockerPals and so on..Dockercon allowed developers, sysadmins, Product Managers & industry evangelists come closer to share their wealth of experience around the container technology. This time I was lucky enough to get chance to Emcee for Docker for Developer Track for the first time. Not only this, I conducted Hallway Track for OpenUSM project & DockerLabs community contribution effort. Around 20-30 participants showed up their interest to learn more around this system management, monitoring & Log Analytics tool.

This Dockercon we had Docker Captains Summit for the first time where the entire day was dedicated to Captains. On Dec #3 ( 10:00 AM till 3:00 PM), we got chance to interact with Docker Staffs, where we put all our queries around Docker Future roadmap. It was amazing to meet all young Captains who joined us this year as well as getting familiar to what they have been contributing to during the initial introductory rounds.

This Dockercon, there has been couple of exciting announcements. 3 of the new features were targeted at Docker Community Edition, while the two were for Docker Enterprise customers. Here’s a rundown of what I think are the most 5 exciting announcements made last week –

#1. Announcement of Cloud Native Application Bundles(CNAB)

Microsoft and Docker have captured a great piece of attention with announcement around CNAB – Cloud Native Application Bundles.

What is CNAB?

Cloud Native Application Bundles (CNAB) are a standard packaging format for multi-component distributed applications. It allows packages to target different runtimes and architectures. It empowers application distributors to package applications for deployment on a wide variety of cloud platforms, cloud providers, and cloud services. It also provides the capabilities necessary for delivering multi-container applications in disconnected environments.

Is it platform-specific tool?

CNAB is not a platform-specific tool. While it uses containers for encapsulating installation logic, it remains un-opinionated about what cloud environment it runs in. CNAB developers can bundle applications targeting environments spanning IaaS (like OpenStack or Azure), container orchestrators (like Kubernetes or Nomad), container runtimes (like local Docker or ACI), and cloud platform services (like object storage or Database as a Service). CNAB can also be used for packaging other distributed applications, such as IoT or edge computing. In nutshell, CNAB are a package format specification that describes a technology for bundling, installing, and managing distributed applications, that are by design, cloud agnostic.

Why do we need CNAB?

The current distributed computing landscape involves a combination of executable units and supporting API-based services. Executable units include Virtual Machines (VMs), Containers (e.g. Docker and OCI) and Functions-as-a-Service (FaaS), as well as higher-level PaaS services. Along with these executable units, many managed cloud services (from load balancers to databases) are provisioned and interconnected via REST (and similar network-accessible) APIs. The overall goal of CNAB is to provide a packaging format that can enable application providers and developers with a way of installing a multi-component application into a distributed computing environment, supporting all of the above types.

Is it open source? Tell me more about CNAB format?

It is an open source, cloud-agnostic specification for packaging and running distributed applications. It is a nascent specification that offers a way to repackage distributed computing apps

The CNAB format is a packaging format for a broad range of distributed applications. It specifies a pairing of a bundle definition(bundle.json) to define the app, and an invocation image to install the app.

The bundle definition is a single file that contains the following information:

Information about the bundle, such as name, bundle version, description, and keywords

Information about locating and running the invocation image (the installer program)

A list of user-overridable parameters that this package recognizes

The list of executable images that this bundle will install

A list of credential paths or environment variables that this bundle requires to execute

What’s Docker future plan to do with CNAB?

This project was incubated by Microsoft and Docker 1 year back. The first implementation of the spec is an experimental utility called Docker App, which Docker officially rolled out this Dockercon and expected to be integrated with Docker Enterprise in near future. Microsoft and Docker plan to donate CNAB to an open source foundation publicly which is expected to happen early next year.

#2. Support for using Docker Compose on Kubernetes.

On the 2nd day of Dockercon, Docker Inc. open sourced Compose on Kubernetes project. Docker Enterprise Edition already had this capability enabled starting Compose File version 3.3 where one can use the same docker-compose.yml file for Swarm deployment as well as one can specify Kubernetes workloads whenever stack is deployed.

What benefit does this bring to Community Developers?

By making it open source, Docker, Inc has really paved a way of infinite possibilities around simplified way of deploying Kubernetes application. Docker Swarm gained popularity because of its simplified approach of application deployment using docker-compose.yml file. Now the community developers can use the same YAML file to deploy their K8s application.

Imagine, you are using Docker Desktop on your Macbook. Docker Desktop provides capability of running both Swarm & Kubernetes. You have context set to GKE cluster which is running on Google Cloud Platform. You just deployed your app using docker-compose.yml on your local Macbook. Now you want to deploy it in the same way but this time on your GKE cluster. Just use docker stack deploy command to deploy it to GKE cluster. Interesting, Isn’t it?

How does Compose on Kubernetes architecture look like?

Compose on Kubernetes is made up of server-side and client-side components. This architecture was chosen so that the entire life cycle of a stack can be managed. The following image is a high-level diagram of the architecture:

If you’re interested to learn further, I would suggest you to visit this link.

This controller uses the standard Kubernetes extension points to introduce the `Stack` to the Kubernetes API. You can use any Kubernetes cluster you like, but if you don’t already have one available then remember that Docker Desktop comes with Kubernetes and the Compose controller built-in, and enabling it is as simple as ticking a box in the settings

Check out the latest doc which shows how to make it work with AKS here.

#3. Introducing Docker Desktop Enterprise

The 3rd Big announcement was an introduction to Docker Desktop Enterprise. With this, Docker Inc. made a new addition to their desktop product portfolio which currently includes the free Docker Desktop Community products for MacOS and Windows. Docker Desktop is the simplest way to get started with container-based development on both Windows 10 and macOS with a set of features now available for the enterprise.

How will Docker Desktop Enterprise be different from Docker Desktop Community Edition?

Good question. Docker Desktop has Docker Engine and Kubernetes built-in and with the addition of swappable version packs you can now synchronize your desktop development environment with the same Docker API and Kubernetes versions that are used in production with Docker Enterprise. You get the assurance that your application will not break due to incompatible API calls, and if you have multiple downstream environments running different versions of the APIs, you can quickly change your desktop configuration with the click of a button.

Not only this, with Docker Desktop Enterprise, you get access to the Application Designer which is a new workflow that provides production-ready application and service templates that let you get coding quickly, with the reassurance that your application meets architectural standards

source ~ Docker, Inc

For those who are interested in Docker Desktop Enterprise – Please note that it is expected to be available for preview in January & General Availability is slated to happen during 1H 2019.

#4. From Zero to Docker in Seconds with “docker assemble” CLI

This time, Docker Team announced a very interesting docker subcommand rightly named as “assemble” to the public. Ann Rahma and Gareth Rushgrove from Docker, Inc. announced assemble, a new command that generates optimized images from non dockerized apps. It will get you from source to an optimized Docker images in seconds.

Here are few of interesting facts around docker assemble utility:

Dockerassemble has capability to build an image without a Dockerfile, all about auto detecting the code framework.

It generates docker images (and lot more) from your code with single command and zero effort! which mean no more dockerfile needed for your app till you have a config (.pom file there).

It can analyze your applications, dependencies, and caches, and give you a sweet Docker image without having to author your own Dockerfiles.

It is built on top of buildKit, will auto detect framework, versions etc. from a config file (.pom file) and automatically add dependencies to the image label, optimize image size and push.

Docker Assemble can also figure out what ports need to be published and what healthchecks are relevant.

The docker–assemble builds app without configuration files, without Dockerfile, just a git repository to deploy

Is it an open source project?

It’s an enterprise feature for now — not in the community version. It is available for a couple languages and frameworks (like Java as demonstrated on Dockercon stage).

How is it different from buildpack?

By reading all through its feature, Docker assemble might look very similar to buildpacks as it overlap with some of the stuff docker-assemble does. But the huge benefit with assemble is that it’s more than just an image (also ports, healthchecks, volume mounts, etc), and it’s integrated into the enterprise toolchain. The docker-assemble is sort of an enterprise-grade buildpack to help with digitalization.

Keep eye on my next blog post to get more detail around the fancy docker assemblecommand.

#5. Docker-app & CNAB together for the first time

On the 2nd day of Dockercon, Docker confirmed that they are the first to implement CNAB for containerized applications and will be expanding it across the Docker platform to support new application development, deployment and lifecycle management. Initially CNAB support will be released as part of our docker-app experimental tool for building, packaging and managing cloud-native applications. With this, Docker now lets you package CNAB bundles as Docker images, so you can distribute and share through Docker registry tools including Docker Hub and Docker Trusted Registry. Additionally, Docker will enable organizations to deploy and manage CNAB-based applications in Docker Enterprise in the upcoming months.

Can I test the preview binaries of docker-app which comes with CNAB support?

Yes, you can find some preview binaries of docker-app with CNAB support here.The latest release of Docker App is one such tool that implements the current CNAB spec. Tt can be used to both build CNAB bundles for Compose (which can then be used with any other CNAB client), and also to install, upgrade and uninstall any other CNAB bundle.

Did you know? There are more than 300,000 Docker Compose files on GitHub.

Docker Compose is a tool for defining and running multi-container Docker applications. It is an amazing developer tool to create the development environment for your application stack. It allows you to define each component of your application following a clear and simple syntax in YAML files. It works in all environments: production, staging, development, testing, as well as CI workflows. Though Compose files are easy to describe a set of related services but there are couple of problems which has emerged in the past. One of the major concern has been around multiple environments to deploy the application with small configuration differences.

Consider a scenario where you have separate development, test, and production environments for your Web application. Under development environment, your team might be spending time in building up Web application(say, WordPress), developing WP Plugins and templates, debugging the issue etc. When you are in development you’ll probably want to check your code changes in real-time. The usual way to do this is mounting a volume with your source code in the container that has the runtime of your application. But for production this works differently. Before you host your web application in production environment, you might want to turn-off the debug mode and host it under the right port so as to test your application usability and accessibility. In production you have a cluster with multiple nodes, and in most of the case volume is local to the node where your container (or service) is running, then you cannot mount the source code without complex stuff that involve code synchronization, signals, etc. In nutshell, this might require multiple Docker compose files for each environment and as your number of service applications increases, it becomes more cumbersome to manage those pieces of Compose files. Hence, we need a tool which can ease the way Compose files can be shareable across different environment seamlessly.

To solve this problem, Docker, Inc recently announced a new tool called “docker-app”(Application Packages) which makes “Compose files more reusable and shareable”. This tool not only makes Compose file shareable but provide us with simplified approach to share multi-service application (not just Docker Image) directly on Dockerhub.

Under this blog post, I will showcase how docker-app tool makes it easier to use Docker Compose for sharing and collaboration and then pushing it directly to DockerHub. Let us get started-

$ docker-app
Build and deploy Docker applications.
Usage:
docker-app [command]
Available Commands:
deploy Deploy or update an application
helm Generate a Helm chart
help Help about any command
init Start building a Docker application
inspect Shows metadata and settings for a given application
ls List applications.
merge Merge the application as a single file multi-document YAML
push Push the application to a registry
render Render the Compose file for the application
save Save the application as an image to the docker daemon(in preparation for push)
split Split a single-file application into multiple files
version Print version information
Flags:
--debug Enable debug mode
-h, --help help for docker-app
Use "docker-app [command] --help" for more information about a command.
[manager1] (local) root@192.168.0.48 ~/app

WordPress Application under dev & Prod environment

If you browse to app/examples/wordpress directory under GitHub Repo, you will see that there is a folder called wordpress.dockerapp that contains three YAML documents:

metadatas

the Compose file

settings for your application

Okay, Fine ! But how you created those files?

The docker-app tool comes with an option “init” which initialize any application with the above 3 YAML files and the directory structure can be initialized with the below command:

docker-app init --single-file wordpress

I have already created a directory structure for my environment and you can find few examples under this directory.

docker-app comes with a few other helpful commands as well, in particular the ability to create Helm Charts from your Docker Applications. This can be useful if you’re adopting Kubernetes, and standardising on Helm to manage the lifecycle of your application components, but want to maintain the simplicity of Compose when writing you applications. This also makes it easy to run the same applications locally just using Docker, if you don’t want to be running a full Kubernetes cluster.

Did you find this blog helpful? Feel free to share your experience. Get in touch with me at twitter @ajeetsraina.

If you want to keep track of latest Docker related information, follow me at https://www.linkedin.com/in/ajeetsraina/.

Estimated Reading Time: 7minutesDocker For Mac 17.12 GA is the first release which includes both the orchestrators – Docker Swarm & Kubernetes under the same Docker platform. As of 1/7/2018 – Experimental Kubernetes has been released under Edge Release(still not available under D4M Stable Release). Experimental Kubernetes is still not available for Docker for Windows & Linux platform. It is slated to be available for Docker for Windows next month(mid of February) and then for Linux by March or April.

Now you might ask why Docker Inc. is making this announcement? What is the fun of having 2 orchestrator under the same roof? To answer this, let me go little back to the past and see how Docker platform looked like:

~ Source – Docker Inc.

Docker platform is like a stack with various layers. The first base layer is called containerd. Containerd is an industry-standard core container runtime with an emphasis on simplicity, robustness and portability. Based on the Docker Engine’s core container runtime, it is available as a daemon for Linux and Windows, which can manage the complete container lifecycle of its host system: image transfer and storage, container execution and supervision, low-level storage and network attachments, etc. Containerd is designed to be embedded into a larger system, rather than being used directly by developers or end-users. It basically includes a daemon exposing gRPC API over a local UNIX socket. The API is a low-level one designed for higher layers to wrap and extend. It also includes a barebone CLI (ctr) designed specifically for development and debugging purpose. It uses runC to run containers according to the OCI specification. The code can be found on GitHub, and here are the contribution guidelines. Let us accept the fact that over the last few years, there has been lots of iteration around this layer but now Docker Inc. has finalised it to a robust, popular and widely accepted container runtime.

On top of containerd, there is an orchestration layer rightly called Docker Swarm. Docker Swarm ties all of your individual machines together which runs container runtime. It allows you to deploy application not on a single machine at a time but into a whole system, thereby making your application distributed.

To take advantage of these layers, as a developer you need tools & environment which can build & package your application that takes advantage of your environment, hence Docker Inc. provides Community Edition like Docker for Mac, Docker for Windows etc. If you are considering to move your application to the production, Docker Enterprise Edition is the right choice.

If the stack looks really impressive, why again the change in architecture?

The reason is – Not everybody uses Swarm.

~ Source – Docker Inc.

Before Swarm & Kubernetes Integration – If you are a developer and you are using Docker, the workflow look something like as shown below. A Developer typically uses Docker for Mac or Docker for Windows.Using a familiar docker build, docker-compose build tool you build your environment and ensure that it gets deployed across a single node cluster OR use docker stack deploy to deploy it across the multiple cluster nodes.

~ Source – Docker Inc.

If your production is in swarm, then you can test it locally on Swarm as it is already inbuilt in Docker platform. But if your production environment runs in Kubernetes, then surely there is lot of work to be done like translating files, compose etc. using 3rd party open source tools and negotiating with their offerings. Though it is possible today but it is not still smooth as Swarm Mode CLI.

With the newer Docker platform, you can seamlessly use both Swarm and Kubernetes flawlessly. Interestingly, you use the same familiar tools like docker stack ls, docker stack deploy, docker ps, `docker stack ps`to display Swarm and Kubernetes containers. Isn’t it cool? You don’t need to learn new tools to play around with Kubernetes cluster.

~ Source – Docker Inc.

The new Docker platform includes both Kubernetes and Docker Swarm side by side and at the same level as shown below. Please note that it is a real kubernetes sitting next to Docker Swarm and NOT A FORK OR WRAPPER.

~ Source – Docker Inc.

Still not convinced why this announcement?

~ Source – Docker Inc.

How does SWARM CLI builds Kubernetes cluster side-by-side?

The docker compose file analyses the input file format and convert it to pods along with creating replicas set as per the instruction set. With the newer Docker for Mac 17.12 release, a new stack command has been added as the first class citizen to Kubernetes CLI.

Ajeets-MacBook-Air:~ ajeetraina$ kubectl get stacks -o wide

NAME AGE

webapp 1h

Important Points –

Future Release of Docker Platform will include both orchestration options available – Kubernetes and Swarm

Swarm CLI will be used for Cluster Management while for orchestration you have a choice of Kubernetes & Swarm

Full Kubernetes API is exposed in the stack, hence support for overall Kubernetes Ecosystem is possible.

Docker Stack Deploy will be able to target both of Swarm or Kubernetes.

Kubernetes is recommended for the production environment

Running both Swarm & Kubernetes is not recommended for the production environment.

AND by now, you must be convinced – “SWARM MODE CLI is NOT GOING ANYWHERE”

Let us test drive the latest Docker for Mac 17.12 beta release and see how Swarm CLI can be used to bring up both Swarm and Kubernetes cluster seamlessly.

Ensure that you have Docker for Mac 17.12 Edge Release running on your Mac system. If you still don’t see 17.12-kube_beta client version, I suggest you to go through my last blog post.

Please note that Kubernetes/kubectl comes by default with Docker for Mac 17.12 Beta release. YOU DON”T NEED TO INSTALL KUBERNETES. By default, a single node cluster is already setup for you by default.

As we have Kubernetes & Swarm Orchestration already present, let us head over to build NGINX services as piece of demonstration on this single node Cluster node.

Writing a Docker Compose File for NGINX containers

Let us write a Docker compose file for nginx image and deploy 3 containers of that image. This is how my docker-compose.yml looks like:

Verifying the NGINX replica sets through the below command:

As shown above, there are 3 replicas of the same NGINX image running as containers.

Verify the cluster using Kubectl CLI displaying the stack information:

Ajeets-MacBook-Air:mynginx ajeetraina$ kubectl get stack -o wide

NAME AGE

webapp 8h

As you see, kubectl and stack deploy displays the same cluster information.

Verifying the cluster using kubectl CLI displaying YAML file:

You can verify that Docker analyses the docker-compose.yaml input file format and convert it to pods along with creating replicas set as per the instruction set which can be verified using the below YAML output format.

We can use the same old stack deploy CLI to verify the cluster information

Managing Docker Stack

Ajeets-MacBook-Air:mynginx ajeetraina$ docker stack services webapp

ID NAME MODE REPLICAS IMAGE PORTS

20e31598-e4c nginx replicated 3/3 nginx *:82->80/tcp,*:444->443/tcp

It’s time to verify if the NGINX webpage comes up well:

Hence, we saw that NGINX service is running both on Kubernetes & Swarm Cluster side by side.

Cleaning up

Run the below Swarm CLI related command to clean up the NGINX service as shown below:

Estimated Reading Time: 5minutesIn my last blog post, I talked about how to get started with NVIDIA docker & interaction with NVIDIA GPU system. I demonstrated NVIDIA Deep Learning GPU Training System, a.k.a DIGITS by running it inside Docker container. ICYMI – DIGITS is essentially a webapp for training deep learning models and is used to rapidly train the highly accurate deep neural network (DNNs) for image classification, segmentation and object detection tasks.The currently supported frameworks are: Caffe, Torch, and Tensorflow. It simplifies common deep learning tasks such as managing data, designing and training neural networks on multi-GPU systems, monitoring performance in real time with advanced visualizations, and selecting the best performing model from the results browser for deployment.

In a typical HPC environment where you run 100 and 100s of NVIDIA GPU equipped cluster of nodes, it becomes important to monitor those systems to gain insight of the performance metrics, memory usage, temperature and utilization. . Tools like Ganglia & Nagios etc. are very popular due to their scalable & distributed monitoring architecture for high-performance computing systems such as clusters and Grids. It leverages widely used technologies such as XML for data representation, XDR for compact, portable data transport, and RRDtool for data storage and visualization. But with the advent of container technology, there is a need of modern monitoring tools and solutions which works well with Docker & Microservices.

It’s all modern world of Prometheus Stack…

Prometheus is 100% open-source service monitoring system and time series database written in Go.It is a full monitoring and trending system that includes built-in and active scraping, storing, querying, graphing, and alerting based on time series data. It has knowledge about what the world should look like (which endpoints should exist, what time series patterns mean trouble, etc.), and actively tries to find faults.

How is it different from Nagios?

Though both serves a purpose of monitoring, Prometheus wins this debate with the below major points –

Nagios is host-based. Each host can have one or more services, which has one check.There is no notion of labels or a query language. But Prometheus comes with its robust query language called “PromQL”. Prometheus provides a functional expression language that lets the user select and aggregate time series data in real time. The result of an expression can either be shown as a graph, viewed as tabular data in Prometheus’s expression browser, or consumed by external systems via the HTTP API.

Nagios is suitable for basic monitoring of small and/or static systems where blackbox probing is sufficient. But if you want to do whitebox monitoring, or have a dynamic or cloud based environment then Prometheus is a good choice.

Nagios is primarily just about alerting based on the exit codes of scripts. These are called “checks”. There is silencing of individual alerts, however no grouping, routing or deduplication.

Let’s talk about Prometheus Pushgateway..

Occasionally you will need to monitor components which cannot be scraped. They might live behind a firewall, or they might be too short-lived to expose data reliably via the pull model. The Prometheus Pushgateway allows you to push time series from these components to an intermediary job which Prometheus can scrape. Combined with Prometheus’s simple text-based exposition format, this makes it easy to instrument even shell scripts without a client library.

The Prometheus Pushgateway allow ephemeral and batch jobs to expose their metrics to Prometheus. Since these kinds of jobs may not exist long enough to be scraped, they can instead push their metrics to a Pushgateway. The Pushgateway then exposes these metrics to Prometheus. It is important to understand that the Pushgateway is explicitly not an aggregator or distributed counter but rather a metrics cache. It does not have statsd-like semantics. The metrics pushed are exactly the same as you would present for scraping in a permanently running program.For machine-level metrics, the textfile collector of the Node exporter is usually more appropriate. The Pushgateway is intended for service-level metrics. It is not an event store.

Under this blog post, I will showcase how NVIDIA Docker, Prometheus & Pushgateway come together to push NVIDIA GPU metrics to Prometheus Stack.

Infrastructure Setup:

Docker Version: 17.06

OS: Ubuntu 16.04 LTS

Environment : Managed Server Instance with GPU

GPU: GeForce GTX 1080 Graphics card

Cloning the GITHUB Repository

Run the below command to clone the below repository to your Ubuntu 16.04 system equipped with GPU card:

NVIDIA provides a python module for monitoring NVIDIA GPUs using the newly released Python bindings for NVML (NVIDIA Management Library). These bindings are under BSD license and allow simplified access to GPU metrics like temperature, memory usage, and utilization.

Next, under the same directory, you will find a python script called “test.py”.

Estimated Reading Time: 5minutesBy default, Docker assigns IPv4 addresses to containers. Does Docker support IPv6 protocol too? If yes, how complicated is to get it enabled? Can I use docker-compose to build micro services which uses IPv6 addresses? What if I work for a company where our services run natively under IPv6 only environment? How shall I build Multi-Node Cluster setup using IPv6? Does Docker 17.06 Swarm Mode support IPv6?

I have been reading numerous queries, GITHUB issues around breaking IPv6 configuration while upgrading Docker version, issues related to IPv6 changes with host configuration etc. and just thought to share few of the findings around IPv6 effort ongoing in Docker upcoming releases.

Does Docker support IPv6 Protocol?

Yes. Support for IPv6 address has been there since Docker Engine 1.5 release.As of Docker 17.06 version (which is the latest stable release as of August 2017) by default, the Docker server configures the container network for IPv4 only. You can enable IPv4/IPv6 dualstack support by adding the below entry under daemon.json file as shown below:

File: /etc/docker/daemon.json

{

“ipv6”: true,

“fixed-cidr-v6”: “2001:db8:1::/64”

}

This is very similar to old way of running the Docker daemon with the --ipv6 flag. Docker will set up the bridge docker0 with the IPv6 link-local address fe80::1.

Why did we add “fixed-cidr-v6”: “2001:db8:1::/64” entry?

By default, containers that are created will only get a link-local IPv6 address. To assign globally routable IPv6 addresses to your containers you have to specify an IPv6 subnet to pick the addresses from. Setting the IPv6 subnet via the --fixed-cidr-v6 parameter when starting Docker daemon will help us achieve globally routable IPv6 address.

The subnet for Docker containers should at least have a size of /80. This way an IPv6 address can end with the container’s MAC address and you prevent NDP neighbor cache invalidation issues in the Docker layer.

With the --fixed-cidr-v6 parameter set Docker will add a new route to the routing table. Further IPv6 routing will be enabled (you may prevent this by starting dockerd with --ip-forward=false).

Let us closely examine the changes which Docker Host undergoes before & after IPv6 Enablement:

Let us enable IPv6 on the Host system. In case you find daemon.json already created under /etc/docker directory, don’t delete the old entries, rather just add these two below entries into the file as shown:

{

“ipv6”: true,

“fixed-cidr-v6”: “2001:db8:1::/64”

}

Restarting the docker daemon to reflect the changes:

sudo systemctl restart docker

A Typical Host Network Configuration – After IPv6 Enablement

Did you see anything new? Yes, the docker0 now gets populated with IPV6 configuration.(shown below)

docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default

Not only this, the docker_gwbridge network interface too received IPV6 changes:

docker_gwbridge: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default

link/ether 02:42:bc:0b:2a:84 brd ff:ff:ff:ff:ff:ff

inet 172.18.0.1/16 scope global docker_gwbridge

valid_lft forever preferred_lft forever

inet6 fe80::42:bcff:fe0b:2a84/64 scope link

valid_lft forever preferred_lft forever

PING TEST – Verifying IPv6 Functionalities For Docker Host containers

Let us try bringing up two containers on the same host and see if they can ping each using IPV6 address:

Setting up Ubuntu Container:

mymanager1==>sudo docker run -itd ajeetraina/ubuntu-iproute bash

Setting up CentOS Container:

mymanager1==>sudo docker run -itd ajeetraina/centos-iproute bash

[Please Note: If you are using default Ubuntu or CentOS Docker Image, you will be surprised to find that ip or ifconfig command doesn’t work. You might need to install iproute package for ip command to work & net-tools package for ifconfig to work. If you want to save time, use ajeetraina/ubuntu-iproute for Ubuntu OR ajeetraina/centos-iproute for CentOS directly.]

Now let us initiate the quick ping test:

In this example the Docker container is assigned a link-local address with the network suffix /64 (here: fe80::42:acff:fe11:3/64) and a globally routable IPv6 address (here: 2001:db8:1:0:0:242:ac11:3/64). The container will create connections to addresses outside of the 2001:db8:1::/64 network via the link-local gateway at fe80::1 on eth0.

So the two containers are able to reach out to each other using IPv6 address.

Does Docker Compose support IPv6 protocol?

The answer is Yes. Let us verify it using docker-compose version 1.15.0 and compose file format 2.1. I faced an issue while I use the latest 3.3 file format. As Docker Swarm Mode doesn’t support IPv6, hence it is not included under 3.3 file format. Till then, let us try to bring up container using IPv6 address using 2.1 file format:

docker-compose version

version 1.15.0, build e12f3b9

docker-py version: 2.4.2

CPython version: 2.7.13

OpenSSL version: OpenSSL 1.0.1t 3 May 2016

Let us first verify the network available in the host machine:

File: docker-compose.yml

version: ‘2.1’

services:

app:

image: busybox

command: ping www.collabnix.com

networks:

app_net:

ipv6_address: 2001:3200:3200::20

networks:

app_net:

enable_ipv6: true

driver: bridge

ipam:

driver: default

config:

-- subnet: 2001:3200:3200::/64

gateway: 2001:3200:3200::1

The above docker-compose file will create a new network called testping_app_net based on IPv6 network under the subnet 2001:3200:3200::/64 and container should get IPv6 address automatically assigned.

Let us bring up services using docker-compose up and see if the services communicates over IPv6 protocol:

Verifying the IPv6 address for each container:

As shown above, this new container gets IPv6 address – 2001:3200:3200::20 and hence they are able to reach other flawlessly.

What’s Next? Under the next blog post, I am going to showcase how does IPv6 works across the multiple host machine and will talk about ongoing effort to bring IPv6 support in terms of Swarm Mode.

Did you find this blog helpful? Feel free to share your experience. Get in touch @ajeetsraina.

Estimated Reading Time: 4minutesDocker 17.06.0-ce-RC5 got announced 5 days back and is available for testing. It brings numerous new features & enablements under this new upcoming release. Few of my favourites includes support for Secrets on Windows, allows specifying a secret location within the container, adds --format option to docker system df command, adds support for placement preference to docker stack deploy, adds monitored resource type metadata for GCP logging driver and adding build & engine info prometheus metrics to list a few. But one of the notable and most awaited feature include support of swarm-mode services with node-local networks such as macvlan, ipvlan, bridge and host.

Under the new upcoming 17.06 release, Docker provides support for local scope networks in Swarm. This includes any local scope network driver. Some examples of these are bridge, host, and macvlan though any local scope network driver, built-in or plug-in, will work with Swarm. Previously only swarm scope networks like overlay were supported. This is a great news for all Docker Networking enthusiasts.

A Brief Intro to MacVLAN:

In case you’re new , the MACVLAN driver provides direct access between containers and the physical network. It also allows containers to receive routable IP addresses that are on the subnet of the physical network.

MACVLAN offers a number of unique features and capabilities. It has positive performance implications by virtue of having a very simple and lightweight architecture. It’s use cases includes very low latency applications and networking design that requires containers be on the same subnet as and using IPs as the external host network.The macvlan driver uses the concept of a parent interface. This interface can be a physical interface such as eth0, a sub-interface for 802.1q VLAN tagging like eth0.10 (.10representing VLAN 10), or even a bonded host adaptor which bundles two Ethernet interfaces into a single logical interface.

To test-drive MacVLAN under Swarm Mode, I will leverage the existing 3 node Swarm Mode clusters on my VMware ESXi system. I have tested it on bare metal system and VirtualBox and it works equally great.

[Updated: 9/27/2017 – I have added docker-stack.yml at the end of this guide to show you how to build services out of docker-compose.yml file. DO NOT FORGET TO CHECK IT OUT]

Installing Docker 17.06 on all the Nodes:

curl -fsSL https://test.docker.com > install-docker.sh

sh install-docker.sh

Verifying the latest Docker version:

Setting up 2 Node Swarm Mode Cluster:

Attention VirtualBox Users:– In case you are using VirtualBox, the MACVLAN driver requires the network and interfaces to be in promiscuous mode.

A local network config is created on each host. The config holds host-specific information, such as the subnet allocated for this host’s containers. --ip-range is used to specify a pool of IP addresses that is a subset of IPs from the subnet. This is one method of IPAM to guarantee unique IP allocations.

Docker 17.05.0 Final release went public exactly 2 week back.This community release was the first release as part of new Moby project. With this release, numerous interesting features like Multi-Stage Build support to the builder, using build-time ARG in FROM, DEB packaging for Ubuntu 17.04(Zesty Zapus) has been included. With this latest release, Docker Team brought numerous new features and improvements in terms of Swarm Mode. Example – synchronous service commands, automatic service rollback on failure, improvement over raft transport package, service logs formatting etc.

Placement Preference under Swarm Mode:

One of the prominent new feature introduced is placement preference under 17.05.0-CE Swarm Mode . Placement preference feature allows you to divide tasks evenly over different categories of nodes. It allows you to balance tasks between multiple datacenters or availability zones. One can use a placement preference to spread out tasks to multiple datacenters and make the service more resilient in the face of a localized outage. You can use additional placement preferences to further divide tasks over groups of nodes. Under this blog, we will setup 5-node Swarm Mode cluster on play-with-docker platform and see how to balance them over multiple racks within each datacenter. (Note – This is not real time scenario but based on assumption that nodes are being placed in 3 different racks).

As per the visualizer, you might end up with uneven distribution of services. Example., Rack-1 holding node-1, node-2 and node-3 looks to have almost equal distribution of services, Rack-2 which holds node3 lack WordPress fronted application.

Here Comes Placement Preference for a rescue…

Under the latest release, Docker team has introduced a new feature called “Placement Preference Scheduling”. Let us spend some time to understand what it actually means. You can set up the service to divide tasks evenly over different categories of nodes. One example of where this can be useful is to balance tasks over a set of datacenters or availability zones.

This uses --placement-pref with a spread strategy (currently the only supported strategy) to spread tasks evenly over the values of the datacenter node label. In this example, we assume that every node has a datacenter node label attached to it. If there are three different values of this label among nodes in the swarm, one third of the tasks will be placed on the nodes associated with each value. This is true even if there are more nodes with one value than another. For example, consider the following set of nodes:

Three nodes with node.labels.datacenter=india

One node with node.labels.datacenter=uk

One node with node.labels.datacenter=us

Considering the last example, since we are spreading over the values of the datacenter label and the service has 5 replicas, at least 1 replica should be available in each datacenter. There are three nodes associated with the value “india”, so each one will get one of the three replicas reserved for this value. There is 1 node with the value “uk”, and hence 1 replica for this value will be receiving it. Finally, “us” has a single node that will again get atleast 1 replica of each service reserved.

To understand more clearly, let us assign node labels to Rack nodes as shown below:

Rack-1 :

Node-1

docker node update --label-add datacenter=india node1

docker node update --label-add datacenter=india node2

docker node update --label-add datacenter=india node3

Rack-2

docker node update --label-add datacenter=uk node4

Rack-3

docker node update --label-add datacenter=us node5

Removing both the services:

docker service rm wordpressdb1 wordpressapp

Let us now pass placement preference parameter to the docker service command:

Visualizer: As shown below, we have used placement preference feature to ensure that the service containers get distributed across the swarm cluster on both the racks.

As shown above, –placement-pref ensures that the task is spread evenly over the values of the datacenter node label. Currently spread strategy is only supported.Both engine labels and node labels are supported by placement preferences.

Please Note: If you want to try this feature with Docker compose , you will need Compose v3.3 which is slated to arrive under 17.06 release.

Did you find this blog helpful? Feel free to share your experience. Get in touch @ajeetsraina.

Estimated Reading Time: 4minutesGo programming language has really helped in shaping Docker as a powerful software and enabling fast development for distributed systems. It has been helping developers and operations team to quickly construct programs and tools for Cloud computing environment. Go offers built-in support for JSON (JavaScript Object Notation) encoding and decoding, including to and from built-in and custom data types.

In last 1 year, Docker Swarm Mode has matured enough to become production-ready. The Orchestration platform is quite stable with numerous features like Logging, Secrets Management, Security Scanning, improvement over Scheduling, networking etc. making it more simple to use and scale-out in just few liner commands. With the introduction of new APIs like swarm, node, volume plugins, services etc., Swarm Mode brings dozens of features to control every aspect of swarm cluster sitting on the master node. But when you start building services in the range of 100s & 1000s and that too distributed across another 100s and 1000s of nodes, there arise a need of quick and handy way of filtering the output, especially when you are interested to capture one specific data out of the whole cluster-wide configuration. Here comes ‘a filtering flag’ as a rescue.

The filtering flag (-f or --filter) format is a key=value pair which is actually a very powerful weapon for developers & system administrators.If you have ever played around with Docker CLI, you must have used docker inspect command to get metadata on a container/ image. Using it with -f provides you more specific information like IP address, network etc. There are numerous guide on how to use filters with standalone host holding the Docker images but I found lack of guides talking about Swarm Mode filters.

Under this blog post, I have prepared a quick list of consolidated filtering commands and outputs in tabular format for Swarm Mode Cluster(shown below).

I have 3 node Swarm Mode cluster running on one of my Google Cloud Engine. Let us first create a simple wordpress application as shown below:

Estimated Reading Time: 6minutesApache JMeter is a popular open source software used as a load testing tool for analyzing and measuring the performance of web application or multitude of services. It is 100% pure Java application, designed to test performance both on static and dynamic resources and web dynamic applications. It simulates a heavy load on one or multitude of servers, network or object to test its strength/capability or to analyze overall performance under different load types. The various applications/server/protocol includes – HTTP, HTTPS (Java, NodeJS, PHP, ASP.NET), SOAP / REST Web services, FTP,Database via JDBC, LDAP, Message-oriented middleware (MOM) via JMS,Mail – SMTP(S), POP3(S) and IMAP(S), native commands or shell scripts, TCP, Java Objects and many more.

JMeter is extremely easy to use and doesn’t take time to get familiar with it.It allows concurrent and simultaneous sampling of different functions by a separate thread group. JMeter is highly extensible which means you can write your own tests. JMeter also supports visualization plugins allow you extend your testing.JMeter supports many testing strategies such as Load Testing, Distributed Testing, and Functional Testing. JMeter can simulate multiple users with concurrent threads, create a heavy load against web application under test.

Architecture of JMeter Distributed Load Testing:

JMeter Distributed Load Testing uses client-server model. It is a kind of testing which use multiple systems to perform stress testing. Distributed testing is applied for testing web sites and server applications when they are working with multiple clients simultaneously.

It includes:

1. Master – the system running JMeter GUI, which controls the test.

2. Slave – the system running JMeter-server, which takes commands from the GUI and send requests to the target system(s).

Good Question ! With every new installation of Apache JMeter, you need to download/install JMeter on every new node participating in Distributed Load Testing. Installing JMeter requires dependencies to be installed first like JAVA, default-jre-headless, iputils etc. The complexity begins when you have multitude OS distributions running on your infrastructure. You can always use automation tools or scripts to enable this solution but again there is an extra cost of maintenance and skills required to troubleshoot with the logs when something goes wrong in the middle of the testing.

With Docker, it is just a matter of 1 Docker Compose file and an one-liner command to get the entire infrastructure ready. Under this blog, I am going to show how a single Docker Compose v3.1 file format can help you setup the entire JMeter Distributed Load Testing tool – all working on Docker 17.03 Swarm Mode cluster.

Under this setup, I will be using instance “master101” as master/client node while the rest of worker nodes as “server/slaves” nodes. All of these instances are running Ubuntu 17.04 with Docker 17.03 installed. You are free to choose any latest Linux distributions which supports Docker executables.

First, let us ensure that you have the latest Docker 17.03 running on your machine:

$curl -sSL https://get.docker.com/ | sh

Ensure that the latest Docker compose is installed which supports v3.0 file format:

One can use docker stack deploy command to quickly setup JMeter Distributed Testing environment. This command requires docker-compose.yml file which holds the declaration of services (apache-jmeter-master and apache-jmeter-server) respectively. Let us clone the repository as shown below to get started –

Run the below commands on the Swarm Master node –

$git clone https://github.com/ajeetraina/jmeter-docker/

$cd jmeter-docker

Under this directory, you will find docker-compose.yml file which is shown below:

In the above docker-compose.yml, there are two service definitions – master and server. Through this docker compose file format, we can push master/client service definition to the master node and server specific service definition which has to be pushed to all the slave nodes. The constraints sections takes care of this functionality. This compose file will automatically create an overlay network called jm-network across the cluster.

Let us start the required services out of the docker-compose file as shown below:

$sudo docker stack deploy -c docker-compose.yml myjmeter

Checking if the services are up and running through docker service ls command:

Let us verify if constraints worked correctly or not as shown below:

As shown above, the constraints worked well pushing the containers to the right node.

It’s time to enter into one of the container running on the master node as shown below:

Using docker exec command, one can enter into the container and browse to /jmeter/apache-jmeter-3.1/bin directory.

This is very important topic which I don’t want to miss out , though we are at the end of this blog post. You might be using Virtual Infrastructure for setting up Apache JMeter Distributed Load tool. Generally, each VM is assigned with static IP and it does make sense as your System Under Test will see load coming from different static IP. The above docker-compose file uses an overlay network and IPs are in the range of 172.16.x.x which gets assigned automatically and taken care by Docker Swarm Networking. In case you are looking out for static IP which should get assigned to each containers automatically, then for sure you need macvlan to be implemented. Follow https://github.com/ajeetraina/jmeter-docker/blob/master/static-README.md to achieve this.

In my next blog post, I will cover further interesting stuffs related to JMeter Distributed Load Testing tool. Drop me a message through tweet (@ajeetsraina) if you face any issue with the above solution.