How to Install Confluent Kafka Cluster by using Ansible

Overview

The rise of micro-services brings another level of software architecture, which is a event driven architecture. One of the tools out there to support this mechanism is Apache Kafka. Today’s article will speak about how to install Kafka in the server by using Ansible Playbook.

Confluent Kafka Playbook

This playbook will install Confluent Kafka into 3 cluster nodes. Each node will contain one Kafka broker and one Zookeeper instance. They will in sync one another.

For the installation, I will use Kafka from https://www.confluent.io, or for more clarity I will call it as Confluent Kafka. The Confluent itself provides two version, enterprise and open source. This playbook will use the latter version, which is an open source one.

Kafka Installation Diagram

Playbook Structure

The playbook only contains one role, which is confluent-kafka. Basically this roles will install and configure 4 services:

Zookeeper

Apache Kafka

Schema Registry

REST Proxy

The roles itself will automatically install open-jdk-8 as well. You can disable the java installation, if you already installed JDK in your target machine, by setting variable java_install into false inside roles/confluent-kafka/defaults/main.yml.

Registering Hosts for Clustering

Because some setup need to be align with properties configuration, there are some tricks to get all hosts into Ansible variables. For example, in server.properties there are some configuration to list all zookeeper instances. To address this configuration challenges, I set up a group called kafka_broker_nodes, which contains the 3 nodes to be configured.

Please take a note with ansible_host variable that are listed in all nodes. This variable values will be used by the playbook to be manipulated via Jinja filter. Therefore, we can trick the properties configuration for all the services.

Execute

1

2

// make sure you point to the correct host. See hosts files.

ansible-playbook site.yml-ihosts_local

Conclusion

That is it. By using this playbook, we can setup Apache Kafka plus Zookeeper into three nodes in one single execution. One more information, I setup the playbook to run only on Redhat/CentOS 7 and using Ansible version 2.4

All the codes are in my repository. Whenever any of you want to enhance the installation, such as to be able to execute in Debian distribution or create a user instead of root to run all the services, I am more than welcome.

Author: ru rocker

I am a professional software developer with more than 10 years experiences. I am a certified Java Developer (SCJP and SCWCD). However, In the recent months, I have more interest in DevOps and start to become a polyglot developer. Python and Go-lang become my favorite programming languages besides Java.
View all posts by ru rocker