Apache Kafka-Based Data Streaming Now Available on AWS Cloud

Apache Kafka-Based Data Streaming Now Available on AWS Cloud

Amazon Web Services Inc. (AWS) announced its managed data streaming service based on Apache Kafka is now generally available.

Amazon Managed Streaming for Apache Kafka (Amazon MSK), helps enterprises who want to use the popular open source distributed streaming platform avoid the details of setting up, scaling and managing Kafka clusters for Big Data processing.

“When you run Apache Kafka on your own, you need to provision servers, configure Apache Kafka manually, replace servers when they fail, orchestrate server patches and upgrades, architect the cluster for high availability, ensure data is durably stored and secured, setup monitoring and alarms, and carefully plan scaling events to support load changes,” the site says.

AWS says its new managed service handles all those details, incorporating Apache Kafka deployment best practices.

After being introduced as an open preview at the company’s re:Invent 2018 conference, the cloud giant has added functionality to Amazon MSK such as:

Encryption in-transit via TLS between clients and brokers, and between brokers

Mutual TLS authentication using ACM private certificate authorities

Support for Kafka version 2.1.0

99.9% availability SLA

HIPAA eligible

Cluster-wide storage scale up

Integration with AWS CloudTrail for MSK API logging

Cluster tagging and tag-based IAM policy application

Defining custom, cluster-wide configurations for topics and brokers

The new managed service works with several other AWS cloud offerings, such as providing metrics in Amazon CloudWatch. AWS spokesperson Jeff Barr said support will soon be added for AWS CloudFormation, for describing and provisioning infrastructure resources in a cloud environment.

“Your MSK clusters always run within an Amazon VPC managed by the MSK service. Your MSK resources are made available to your own VPC, subnet, and security group through elastic network interfaces (ENIs) which will appear in your account,” Barr said in a post that details how to work with the new service.