Topics

Featured in Development

Alex Bradbury gives an overview of the status and development of RISC-V as it relates to modern operating systems, highlighting major research strands, controversies, and opportunities to get involved.

Featured in Architecture & Design

Will Jones talks about how Habito, the leading digital mortgage broker, benefited from using Haskell, some of the wins and trade-offs that have brought it to where it is today and where it's going next. He also talks about why functional programming is beneficial for large projects, and how it helps especially with migrating the data store.

Featured in AI, ML & Data Engineering

Katharine Jarmul discusses research related to fair-and-private ML algorithms and privacy-preserving models, showing that caring about privacy can help ensure a better model overall and support ethics.

Featured in Culture & Methods

This personal experience report shows that political in-house games and bad corporate culture are not only annoying and a waste of time, but also harm a lot of initiatives for improvement. Whenever we become aware of the blame game, we should address it! DevOps wants to deliver high quality. The willingness to make things better - products, processes, collaboration, and more - is vital.

Featured in DevOps

Service mesh architectures enable a control and observability loop. At the moment, service mesh implementations vary in regard to API and technology, and this shows no signs of slowing down. Building on top of volatile APIs can be hazardous. Here we suggest to use a simplified, workflow-friendly API to shield organization platform code from specific service-mesh implementation details.

Microsoft Announces the General Availability of Azure Event Hubs for Apache Kafka

Recently, Microsoft announced the general availability of Azure Event Hubs for Apache Kafka. Microsoft claim that customers will "get the best of both worlds"—the ecosystem and tools of Kafka, along with Azure’s security and global scale. Furthermore, by bringing together two powerful distributed streaming platforms, customers can have access to the breadth of Kafka ecosystem applications without having to manage servers or networks.

At the beginning of June, InfoQ reported on the preview of the integration of Kafka with Azure Event Hubs as the engineering team responsible for Azure Event Hubs made a Kafka endpoint available for users of their service to stream event data into it. Now the integration is ready for general availability, and the team have added a new capability that enables customers to start streaming events from applications using the Kafka protocol directly into Event Hubs by merely changing a connection string. Moreover, customers can keep using their existing Kafka applications, frameworks, and tools to talk to Event Hubs.

With EventHubs for Kafka, customers can start streaming events from applications using the Kafka protocol directly into Event Hubs. Microsoft manages the service, and the customer does not need to run Zookeeper, or control or configure any clusters. Shubha Vijayasarathy, program manager, Azure Service Bus, said in a recent Azure Friday episode:

The biggest benefit of using Azure Event Hubs for Apache Kafka is that you're not changing any of your clients, you're not changing your code, your producers, your applications, your tools, or your frameworks.

Customers can use Kafka Connect or MirrorMaker to talk to Event Hubs without changing a line of code.

Other benefits of the integration of Kafka with Azure Event Hubs are according to the announcement blog:

Event Hubs are deeply-integrated with other Azure services like Azure Databricks, Azure Stream Analytics, and Azure Functions. Therefore, customers can unlock further analytics and processing.

EventHubs for Kafka provides support for Apache Kafka 1.0 and later – the engineering team has mapped the Apache Kafka Protocol to the native AMQP 1.0 protocol. This protocol translation allows other AMQP 1.0 based applications to communicate with Kafka applications. Also, JMS based applications can use Apache Qpid to send data to Kafka based consumers.

Both Azure EventHubs and Kafka are designed to handle large-scale, real-time stream ingestion and are primarily distributed, partitioned, and replicated commit log services. Furthermore, both are using a partitioned consumer model with a client-side cursor concept that provides horizontal scalability for demanding workloads. By adding an endpoint for Kafka, Event Hub is effectively mimicking Kafka. In the Azure Friday episode, Shubha Vijayasarathy explained the conceptual architecture of both:

In the conceptual architecture of event hubs, you have an Event Hub or Kafka Topic; both are very similar. The way Event Hubs or Kafka get data is that they both have both partitions in their architecture, where the data gets distributed. You have a bunch of event producers, and a bunch of event receivers who are interested in this data and the data gets distributed within Event Hubs or Kafka topics in a uniform pattern.