This section provides a basic guide for deploying a Kafka cluster along with all Confluent Platform components in your Docker environment. By the end of this quickstart, you will have a functional Confluent deployment against which you can run any number of applications.

In order to keep things simple, we’ll start with a single node Docker environment. Details on more complex target environments are available later in this documentation (More Tutorials). We will also be configuring Kafka and Zookeeper to store data locally in their Docker containers. You should refer to our documentation on Docker external volumes for examples of how to add mounted volumes to your host machines. Mounted volumes provide a persistent storage layer for deployed containers, which allows images such as cp-kafka and cp-zookeeper to be stopped and restarted without losing their stateful data.

If you’re running on Windows or Mac OS X, you’ll need to use Docker Machine to start the Docker host. Docker runs natively on Linux, so the Docker host will be your local machine if you go that route. If you are running on Mac or Windows, be sure to allocate at least 4 GB of ram to the Docker Machine.

After installing Docker as described in the previous section, we can create a Docker machine and begin starting up Confluent Platform.

Note

The following steps show each Docker container being launched in detached mode. We’ll be sure to show you how access the logs for those detached containers. If you prefer to run the containers in the foreground, you can do so by replacing the -d (“detached”) flag with --it (“interactive”). Containers launched in interactive mode will stream the log messages for the Confluent service directly to the terminal window. For that reason, you’ll need a separate terminal for each Docker image launched in interactive mode.

Create and configure the Docker Machine. This example creates a VirtualBox VM with ~6GB of memory to serve as our Docker host.

Next, configure your terminal window to attach it to your new Docker Machine:

eval$(docker-machine env confluent)

All of the subsequent commands should be run from that terminal window to ensure proper access to the running Docker host. To execute Docker commands from a new terminal window, simply execute the eval$(docker-machineenvconfluent) first.

This command instructs Docker to launch an instance of the confluentinc/cp-zookeeper:3.1.2 container and name it zookeeper. We also specify that we want to use host networking and pass in the required parameter for running Zookeeper: ZOOKEEPER_CLIENT_PORT. For a full list of the available configuration options and more details on passing environment variables into Docker containers, see the configuration reference docs.

We’ll check the Docker logs to confirm that the container has booted up successfully and started the Zookeeper service. The command to do that is:

docker logs zookeeper

With this command, we’re referencing the container name we want to see the logs for. To list all containers (running or failed), you can always run dockerps-a. This is especially useful when running in detached mode.

When you output the logs for Zookeeper, you should see the following message at the end of the log output:

Note that the message shows the Zookeeper service listening at the port we passed in as ZOOKEEPER_CLIENT_PORT above.

If the service is not running, the log messages should provide details to help you identify the problem. Some common errors include:

Network port already in use. In that case, you’ll see a message indicating that the Zookeeper service could not bind to the selcted port. Simply change to an open port or identify (and stop) the Docker container that has a service using that port.

Insufficient resources. In rare occasions, you may see memory allocation or other low-level failures at startup. This will only happen if you dramatically overload the capacity of your Docker host.

You’ll notice that we set the KAFKA_ADVERTISED_LISTENERS variable to localhost:29092. This will make Kafka accessible from outside the container by advertising it’s location on the Docker host. We also passed in the zookeeper port we used when launching that container a moment ago. Because we are using --net=host, the hostname for the zookeeper service can be left at localhost.

Now we can take this very basic deployment for a test drive. We’ll verify that the broker is functioning normally by creating a topic and producing data to it. We’ll use the client tools directly from another Docker container.

First, we’ll create a topic. We’ll name it foo and keep things simple by just giving it one partition and only one replica. Production environments with more broker nodes would obviously use higher values for both partitions and replicas for scalability and resiliency.

Now that we have Kafka and Zookeeper up and running, we can deploy some of the other components included in Confluent Platform. We’ll start by using the Schema Registry to create a new schema and send some Avro data to a Kafka topic. Although you would normally do this from one of your applications, we’ll use a utility provided with Schema Registry to send the data without having to write any code.

As we did before, we can check that it started correctly by viewing the logs.

docker logs schema-registry

For the next step, we’ll publish data to a new topic that will leverage the Schema Registry. For the sake of simplicity, we’ll launch a second Schema Registry container in interactive mode, and then execute our kafka-avro-console-producer utility from there.

Direct the utility at the local Kafka cluster, tell it to write to the topic bar, read each line of input as an Avro message, validate the schema against the Schema Registry at the specified URL, and finally indicate the format of the data.

Once started, the process will wait for you to enter messages, one per line, and will send them immediately when you hit the Enter key. Try entering a few messages:

{"f1":"value1"}{"f1":"value2"}{"f1":"value3"}

Note

If you hit Enter with an empty line, it will be interpreted as a null value and cause an error. You can simply start the console producer again to continue sending messages.

When you’re done, use Ctrl+C or Ctrl+D to stop the producer client. You can then type exit to leave the container altogether. Now that we’ve written avro data to Kafka, we should check that the data was actually produced as expected to consume it. Although the Schema Registry also ships with a built-in console consumer utility, we’ll instead demonstrate how to read it from outside the container on our local machine via the REST Proxy. The REST Proxy depends on the Schema Registry when producing/consuming avro data, so we’ll need to pass in the details for the detached Schema Registry container we launched above.

For the next two steps, we’re going to use CURL commands to talk to the REST Proxy. Our deployment steps thus far have ensured that both the REST Proxy container and the Schema Registry container are accessible directly through network ports on our local host. The REST Proxy service is listening at http://localhost:8082 As above, we’ll launch a new Docker container from which to execute our commands:

Our next curl command will retrieve data from a topic in our cluster (bar in this case). The messages will be decoded, translated to JSON, and included in the response. The schema used for deserialization is retrieved automatically from the Schema Registry service, which we told the REST Proxy how to find by setting the KAFKA_REST_SCHEMA_REGISTRY_URL variable on startup.

The Control Center application provides enterprise-grade capabilities for monitoring and managing your Confluent deployment. Control Center is part of the Confluent Enterprise offering; a trial license will support the image for the first 30 days after your deployment.

First, let’s walk through how to use Confluent Control Center with console producers and consumers to monitor consumption and latency.

We’ll launch the Confluent Control Center image the same as we’ve done for earlier containers, connecting to the ZooKeeper and Kafka containers that are already running. This is also a good opportunity to illustrate mounted volumes, so we’ll first create a directory on the Docker Machine host for Control Center data.

To see the Control Center UI, open the link http://<ip-of-docker-host>:9021 in your browser. The Docker Host IP is displayed with the command docker-machineipconfluent. If your docker daemon is running on a remote machine (such as an AWS EC2 instance), you’ll need to allow TCP access to that instance on port 9021. This is done in AWS by adding a “Custom TCP Rule” to the instance’s security group; the rule should all access to port 9021 from any source IP.

Initially, the Stream Monitoring UI will have no data.

Confluent Control Center Initial View

Next, we’ll run the console producer and consumer with monitoring interceptors configured and see the data in Control Center. First we need to create a topic for testing.

This command will use the built-in Kafka Console Producer to produce 10000 simple messages on a 10 second interval to the c3-test topic. Upon running it, you should see the following:

Produced10000messages.

The message will repeat every 10 seconds, as successive iterations of the shell loop are executed. You can terminate the client with a Ctrl+C.

We’ll use the console consumer with the monitoring interceptor enabled to read the data. We’ll want to run this command in a separate terminal window (prepared with the eval$(docker-machineenvconfluent) as we described earlier).

If everything is working as expected, each of the original messages we produced should be written back out:

1....1000Processedatotalof1000messages

We’ve intentionally setup a slow consumer to consume at a rate
of 1000 messages per second. You’ll soon reach a steady state
where the producer window shows an update every 10 seconds while
the consumer window shows bursts of 1000 messages received
every 1 second. The monitoring activity should appear in the
Control Center UI after 15 to 30 seconds. If you don’t see any
activity, use the scaling selector in the upper left hand corner
of the web page to select a smaller time window (the default is
4 hours, and you’ll want to zoom in to a 10-minute scale). You
will notice there will be moments where the bars are colored red
to reflect the slow consumption of data.

Confluent Control Center provides alerting functionality to
notify you when anomalous events occur in your cluster. This
section assumes the console producer and
consumer we launched to illustrate the stream monitoring features
are still running in the background.

The Alerts / Overview link the lefthand navigation sidebar takes
will display a history of all triggered events. To begin receiving
alerts, we’ll need to create a trigger. Click the “Triggers”
navigation item and then select “+ New trigger”.

Let’s configure a trigger to fire when the difference between our actual
consumption and expected consumption is greater than 1000 messages:

New trigger

Set the trigger name to be “Underconsumption”, which is what will be displayed
on the history page when our trigger fires. We need to select a specific
consumer group (qs-consumer) for this trigger. That’s the name of
the group we specified above in our invocation of
kafka-console-consumer.

Set the trigger metric to be “Consumption difference” where the
condition is “Greater than” 1000 messages. The buffer time (in seconds) is the
wall clock time we will wait before firing the trigger to make sure the trigger
condition is not too transient.

After saving the trigger, Control Center will now prompt us to associate an action that will execute when
our newly created trigger fires. For now, the only action is to send an email.
Select our new trigger and choose maximum send rate for your alert email.

New action

Let’s return to our trigger history page. In a short while, you should see
a new trigger show up in our alert history. This is because we setup our
consumer to consume data at a slower rate than our producer.

In this section, we’ll create a simple data pipeline using Kafka Connect. We’ll start by reading data from a file and writing that data to a new file. We will then extend the pipeline to show how to use Connect to read from a database table. This example is meant to be simple for the sake of this introductory tutorial. If you’d like a more in-depth example, please refer to our tutorial on Using a JDBC Connector with avro data.

First, let’s start up a container with Kafka Connect. Connect stores all its stateful data (configuration, status, and internal offsets for connectors) directly in Kafka topics. We will create these topics now in the Kafka cluster we have running from the steps above.

It is possible to allow connect to auto-create these topics by enabling the autocreation setting. However, we recommend doing it manually, as these topics are important for connect to function and you’ll likely want to control settings such as replication factor and number of partitions.

Next, we’ll create a topic for storing data that we’re going to be sending to Kafka for this tutorial.

For this example, we’ll create a FileSourceConnector, a FileSinkConnector and directories for storing the input and output files. If you are running Docker Machine then you will need to SSH into the VM to run these commands by running docker-machinessh<yourmachinename>.

First, let’s create the directory where we’ll store the input and output data files. Remember, we must do this within the Docker Host.

As you can see in the above command, we tell Connect to refer to the three topics we create in the first step of this Connect tutorial. Let’s check to make sure that the Connect worker is up by running the following command to search the logs:

We will now create our first connector for reading a file from disk. To do this, let’s start by creating a file with some data. Again, if you are running Docker Machine then you will need to SSH into the VM to run these commands by running docker-machinessh<yourmachinename>. (You may also need to run the command as root).

Now create the connector using the Kafka Connect REST API. (Note: Make sure you have curl installed!)

Set the CONNECT_HOST environment variable. If you are running this on Docker Machine, then the hostname will need to be docker-machineip<yourdockermachinename>. If you are running on a cloud provider like AWS, you will either need to have port 28082 open or you can SSH into the VM and run the following command:

Now that the connector is up and running, let’s try reading a sample of 10 records from the quickstart-data topic to check if the connector is uploading data to Kafka, as expected. You’ll want to do this in a separate terminal window, retaining the ssh session to the Docker Host for later commands.

{"schema":{"type":"string","optional":false},"payload":"1"}{"schema":{"type":"string","optional":false},"payload":"2"}{"schema":{"type":"string","optional":false},"payload":"3"}{"schema":{"type":"string","optional":false},"payload":"4"}{"schema":{"type":"string","optional":false},"payload":"5"}{"schema":{"type":"string","optional":false},"payload":"6"}{"schema":{"type":"string","optional":false},"payload":"7"}{"schema":{"type":"string","optional":false},"payload":"8"}{"schema":{"type":"string","optional":false},"payload":"9"}{"schema":{"type":"string","optional":false},"payload":"10"}
Processed a total of 10 messages

Success! We now have a functioning source connector! Now let’s bring balance to the universe by launching a File Sink to read from this topic and write to an output file. You can do so using the following command from the Docker Host session started earlier:

Next we’ll see how to monitor the Kafka Connect connectors in Control Center. Because we specified the monitoring interceptors when we deployed the Connect container, the data flows through all of our connectors will monitored in the same ways as the console producer/consumer tasks we executed above. Additionally, Control Center allows us to visually manage and deploy connectors, as you’ll see now.

Select the Management / Kafka Connect link in the Control Center navigation bar. Select the SOURCES and SINKS tabs at the top of the page to see that both the source and sink are running.

Confluent Control Center showing a Connect source

Confluent Control Center showing a Connect sink

You should start to see stream monitoring data from Kafka Connect in the Control Center UI from the running connectors. Remember that the file contained only 1000 messages, so you’ll only see a short spike of topic data.

Once you’re done, cleaning up is simple. Run the command dockerrm-f$(dockerps-a-q) to delete all the containers we created in the steps above for your target Docker Host. Because we allowed Kafka and Zookeeper to store data on their respective containers, there are no additional volumes to clean up. If you also want to remove the Docker machine you used, you can do so using docker-machinerm<yourmachinename>.

Docker Compose is a powerful tool that enables you to launch multiple docker images in a coordinated fashion. It is ideal for platforms like Confluent. Before you get started, you will need to install both the core Docker Engine and Docker Compose. Once you’ve done that, you can follow the steps below to start up the Confluent Platform services.

Start Zookeeper and Kafka using Docker Compose create and start commands. You’ll run these commands from the directory containing the docker-compose.yml file.

docker-compose create
docker-compose start

Before we move on, let’s make sure the services are up and running:

docker-compose ps

You should see the following:

Name Command State Ports
-----------------------------------------------------------------------
kafkasinglenode_kafka_1 /etc/confluent/docker/run Up
kafkasinglenode_zookeeper_1 /etc/confluent/docker/run Up