In this section, we provide a tutorial for running a three-node Kafka cluster and Zookeeper ensemble. By the end of this tutorial, you will have successfully installed and run a simple deployment with Docker.

It is worth noting that we will be configuring Kafka and Zookeeper to store data locally in the Docker containers. For production deployments (or generally whenever you care about not losing data), you should use mounted volumes for persisting data in the event that a container stops running or is restarted. This is important when running a system like Kafka on Docker, as it relies heavily on the filesystem for storing and caching messages. Refer to our documentation on Docker external volumes for an example of how to add mounted volumes to the host machine.

If you’re running on Windows or Mac OS X, you’ll need to use Docker Machine to start the Docker host. Docker runs natively on Linux, so the Docker host will be your local machine if you go that route. If you are running on Mac or Windows, be sure to allocate at least 4 GB of ram to the Docker Machine.

Now that we have all of the Docker dependencies installed, we can create a Docker machine and begin starting up Confluent Platform.

Note

In the following steps we’ll be running each Docker container in detached mode. However, we’ll also demonstrate how access the logs for a running container. If you prefer to run the containers in the foreground, you can do so by replacing the -d flags with --it.

You should see the following (it might take some time for this command to return data. Kafka has to create the __consumers_offset topic behind the scenes when you consume data for the first time and this may take some time):

Before you get started, you will first need to install Docker and Docker Compose. Once you’ve done that, you can follow the steps below to start up the Confluent Platform services.

Clone the CP Docker Images Github Repository.

git clone https://github.com/confluentinc/cp-docker-images

We have provided an example Docker Compose file that will start up Zookeeper and Kafka. Navigate to cp-docker-images/examples/kafka-cluster, where it is located:

Start Zookeeper and Kafka using Docker Compose up command.

docker-compose up

In another terminal window, go to the same directory (kafka-cluster). Before we move on, let’s make sure the services are up and running:

docker-compose ps

You should see the following:

Name Command State Ports
----------------------------------------------------------------------
kafkacluster_kafka-1_1 /etc/confluent/docker/run Up
kafkacluster_kafka-2_1 /etc/confluent/docker/run Up
kafkacluster_kafka-3_1 /etc/confluent/docker/run Up
kafkacluster_zookeeper-1_1 /etc/confluent/docker/run Up
kafkacluster_zookeeper-2_1 /etc/confluent/docker/run Up
kafkacluster_zookeeper-3_1 /etc/confluent/docker/run Up

Check the Zookeeper logs to verify that Zookeeper is healthy. For
example, for service zookeeper-1: