Create a Kafka multi-broker cluster

This section describes the creation of a multi-broker Kafka cluster with brokers located on different hosts. In this scenario:

One server hosts the Zookeeper server and a Kafka broker

The second server hosts a a second Kafka broker

The third server hosts a producer and a consumer

NOTE: Before beginning, ensure that ports 2181 (Zookeeper) and 9092 (Kafka) are open on the first server and port 9092 (Kafka) is open on the second server. Also ensure that remote connections are possible between the three servers (instructions).

Configuring the first server (Zookeeper manager and Kafka broker)

The default configuration may be used as is. However, you must perform the steps below:

Produce some messages by running the command below and then entering some messages, each on a separate line. Enter Ctrl-D to end.

$ /opt/bitnami/kafka/bin/kafka-console-producer.sh --broker-list PUBLIC_IP_ADDRESS_OF_FIRST_KAFKA_BROKER:9092 --producer.config /opt/bitnami/kafka/conf/producer.properties --topic multiBroker
this is a message
this is another message

Consume the messages. The consumer will connect to the cluster and retrieve and display the messages you entered in the previous step.

$ /opt/bitnami/kafka/bin/kafka-console-consumer.sh --zookeeper PUBLIC_IP_ADDRESS_OF_FIRST_KAFKA_BROKER:2181 --topic multiBroker --from-beginning
this is a message
this is another message