Always back up all configuration files before upgrading. This includes /etc/kafka, /etc/kafka-rest, /etc/schema-registry and /etc/camus.

Read through the documentation and draft an upgrade plan that matches your specific requirements and environment before starting the upgrade process. Put differently, don’t start working through this guide on a live cluster. Read the guide entirely, make a plan, then execute the plan.

Pay careful consideration to the order in which components are upgraded. Kafka is backward compatible, which means that clients from Kafka 0.8.x releases (CP 1.0.x) will work with brokers from Kafka 0.9.x and 0.10.x releases (CP 2.0.x and 3.0.x), but not vice-versa. This means you always need to plan upgrades such that all brokers are upgraded before clients. Clients include any application that uses Kafka producer or consumer, command line tools, Camus, Schema Registry, Rest Proxy, Kafka Connect and Kafka Streams.

IMPORTANT: Due to a bug introduced in Kafka 0.9.0.0 (CP 2.0.0), clients that depend on ZooKeeper (old Scala high-level Consumer and MirrorMaker if used with the old consumer) will not work with 0.10.0.x brokers. Therefore, Kafka 0.9.0.0 (CP 2.0.0) clients should be upgraded to 0.9.0.1 (CP 2.0.1) before brokers are upgraded to 0.10.0.x. This step is not necessary for 0.8.x or 0.9.0.1 clients.

IMPORTANT: although not recommended, some deployments have clients co-located with brokers (on the same node). In these cases, both the broker and clients share the same packages. This is problematic because all brokers must be upgraded before clients are upgraded. Pay careful attention to this when upgrading.

Kafka 0.10.0.0 contains breaking changes and deprecations with respect to previous major versions. Refer to the Kafka documentation to understand how they affect applications using Kafka.

Determine if clients are co-located with brokers. If yes, ensure all client processes are not upgraded until all Kafka brokers have been upgraded.

Decide on performing a rolling upgrade or a downtime upgrade. Confluent Platform supports both rolling upgrades (upgrade one broker at a time to avoid cluster downtime) and downtime upgrades (take down the entire cluster, upgrade it, and bring everything back up).

Upgrade all Kafka brokers (more below).

Upgrade Schema Registry, Rest Proxy and Camus (more below).

If it makes sense, build applications that use Kafka producers and consumers against the new 0.10.0.0 libraries and deploy the new versions. See Application Development documentation for more details on using the 0.10.0.0 libraries.

You can upgrade Kafka brokers from CP 2.0.x to CP 3.0.0 by upgrading the installed packages and restarting the respective processes. More on this later.

In a rolling upgrade scenario, upgrade one Kafka broker at a time. In a downtime upgrade scenario, take the entire cluster down, upgrade each Kafka broker, then start the cluster.

Special steps for 2.0.x to 3.0.0 upgrade: In a rolling upgrade scenario, upgrading from CP 2.0.x (Kafka 0.9.0.x) to CP 3.0.0 (Kafka 0.10.0.0) requires special steps because Kafka 0.10.0.0 includes a change to the inter-broker protocol and the on-disk data format. Follow the below steps for a rolling upgrade:

Modify server.properties on all Kafka brokers by adding/changing the following properties (you can edit these while the brokers are still running):

inter.broker.protocol.version=0.9.0.1

log.message.format.version=0.9.0.1 (optional, improves broker performance if most of your clients are still 0.8.x or 0.9.0.x)

Upgrade each Kafka broker, one at a time (see below).

Once all Kafka brokers have been upgraded, modify server.properties again by changing the following property: inter.broker.protocol.version=0.10.0 (0.10.0 instead of 0.9.0.1).

Restart each Kafka broker, one at a time, to apply the configuration change.

Once most clients are using 0.10.0.0, remove the log.message.format.version property to optimize broker performance for 0.10.0.0 clients.

Restart each Kafka broker, one at a time, to apply the configuration change.

Instructions for both deb packages and rpm packages are below. For zip and tar archives, the old archives directory can be simply deleted after the new archive folder has been created and any old configuration files copied over.

deb packages via apt

Backup all configuration files from /etc, including /etc/kafka, /etc/kafka-rest, /etc/schema-registry and /etc/camus.

Stop the services and remove the existing packages and their dependencies. As mentioned above, this can be done on one server at a time for rolling upgrade.

As mentioned above, if it makes sense, build applications that use Kafka producers and consumers against the new 0.10.0.0 libraries and deploy the new versions. See Application Development documentation for more details on using the 0.10.0.0 libraries.

A new required configuration, status.storage.topic was added to Kafka Connect in 0.10.0.0. To upgrade a Kafka Connect cluster, this configuration should be added before updating to the new version. The setting will be ignored by older versions of Kafka Connect.

Backup worker configuration files.

Modify your configuration file to add the status.storage.topic setting. You can safely modify the configuration file while the worker is running. Note that you should create this topic manually. See the Distributed Mode Configuration section of the Connect User Guide for a detailed explanation.