Topic navigation

Blog Articles

Announcing AMQ Streams: Apache Kafka on OpenShift

We are excited to announce a Developer Preview of Red Hat AMQ Streams, a new addition to Red Hat AMQ, focused on running Apache Kafka on OpenShift.

Apache Kafka is a leading real-time, distributed messaging platform for building data pipelines and streaming applications.

Using Kafka, applications can:

Publish and subscribe to streams of records.

Store streams of records.

Process records as they occur.

Kafka makes all of this is possible while being fast, horizontally scalable and fault tolerant. This makes Kafka suitable for a large range of use cases, including website activity tracking, metrics and log aggregation, stream processing, event sourcing, and IoT telemetry. The forthcoming AMQ Streams product will provide Red Hat customers with a supported offering for running Apache Kafka on Red Hat Enterprise Linux and on Red Hat OpenShift Container Platform.

As more and more applications move to Kubernetes and OpenShift, it is increasingly important to be able to run the communication infrastructure on the same platform. OpenShift as a highly scalable platform is a natural fit for messaging technologies such as Kafka. But with AMQ Streams, our target is not to just run Apache Kafka on OpenShift, but rather AMQ Streams makes running and managing Apache Kafka “OpenShift native.”

Develop using Red Hat's most valuable products

Uniting the massive scalability of Kafka on an elastic platform like OpenShift involves resolving a number of technology challenges:

Kafka brokers are inherently stateful, because each has its own identity and data logs that must be preserved in the event of restarts.

Updating and scaling a Kafka cluster requires careful orchestration to ensure that messaging clients are unaffected and no records are lost.

By design, Kafka clients connect to all the brokers in the cluster. This is part of what gives Kafka its horizontal scaling and high availability, but when running on OpenShift, this means the Kafka cluster cannot simply be put behind a load-balanced service like other services. Instead services have to be orchestrated in parallel with cluster scaling.

Running Kafka also requires running a Zookeeper cluster, which has many of the same challenges as running the Kafka cluster.

AMQ Streams simplifies the deployment, configuration, management and use of Apache Kafka on OpenShift using the Operator concept, thereby enabling the inherent benefits of OpenShift, such as elastic scaling. An Operator is an application-specific controller that extends the Kubernetes APIs and combines them with domain-specific knowledge and makes it easy to run and manage complex applications. Developers and administrators used to OpenShift’s declarative approach to resource provisioning can now enjoy those same benefits when working with Kafka, Kafka Connect, and Kafka topics.

AMQ Streams makes it easy to:

Deploy a complete Kafka cluster, at the scale that suits you, with the click of a button or with a single oc create command.

Deploy the Kafka topic right alongside the microservice that uses it.

Scale up the partitions of that topic.

Trivially scale up and and down the Kafka cluster according to load.

AMQ Streams is optimized for running on OpenShift (as opposed to regular Kubernetes). Not only does it benefit from Red Hat’s years of experience and in-depth knowledge gained from developing and running OpenShift, but there is, for example, special support for building Kafka Connect clusters with the user’s own Kafka Connect plugins. Further, OpenShift-specific features and Red Hat product integrations are anticipated, with the overall aim being a seamless experience with full support from the OpenShift fabric up.

Unsurprisingly, because Red hat is the world’s leading provider of open source technologies for the enterprise, AMQ Streams is fully open source and based on the Strimzi project.

The Developer Preview, which is being made available to interested customers this week, provides the foundation for running Kafka on OpenShift. Interested customers and other interested parties are invited try it out, give us their feedback and, if desired, collaborate on the open source Strimzi project to help shape the future direction of AMQ Streams on OpenShift.

If you’re lucky enough to be in San Francisco this week for Red Hat Summit, then you can hear a lot more about AMQ Streams (and the broader Red hat AMQ product) at the following sessions:

Privacy Preference Center

Consent Management

We use cookies on our websites to deliver our online services. Details about how we use cookies and how you may disable them are set out in our Privacy Statement. By using this website you agree to our use of cookies.

Privacy Policy

Required

We use cookies on our websites to deliver our online services. Details about Red Hat's privacy policy, how we use cookies and how you may disable them are set out in our Privacy Page. By using this website you agree to our use of cookies.