The shelf life of even the best architecture is limited by its ability to evolve alongside the business. Apache Kafka has seen massive adoption by companies not only as a product, but also as a design philosophy. So, is Kafka well-equipped to cultivate evolution? Organizations which design systems … are constrained to produce designs which are copies of the communication structures of these organizations. — Conway's law On one hand, Kafka’s immutable log enables adaptation...

It’s time to realize Apache Kafka’s full potential, spanning past and present

May 10, 2018

Kafka users enjoy a broad sweet spot, one that can naturally grow in the context of use cases and in concert with the organization that runs it. You can simply run it as a message bus. Or, it can drive reactive microservices. In its most sophisticated form, Kafka promises to be the central nervous system (CNS) for a business, turning the database inside out. We can go further. Kafka’s power inspires an ideal system, where...

Building systems in terms of events is one of software’s most potent ideas for influencing product design. Events, if you’re not already familiar, are a means for modeling any kind of activity. Designing with events allows you to use information you already have to ask more meaningful questions and increase the versatility of your software’s functionality. Our industry tragically places a disproportionate amount of excitement about events on the architectural characteristics that they provide, at...

Distributed Masonry is pleased to open source Metamorphic, a Clojure(Script) complex event processing library. Complex event processing (CEP) is a technique for analyzing an input stream for sequences of events that match patterns. Patterns are expressed as a series of rules with properties denoting their relative positional attributes. In this post, we’ll explore why we built Metamorphic, what’s special about it, and what’s coming closely behind it in the next 4 weeks. (Impatient? Skip to...

Patterns of Event Stream Processing is a series of articles surveying concepts that underpin streaming systems. In this post, we’ll study a technique called session windowing that’s used for grouping related data over a period of time. We’re going to look at what they are, what they’re used for, and why they’re hard to build from scratch. We’ll finish with our suggested design pattern for building session windows in a scalable manner. A primer on...

In our last post, The Future of Event Stream Processing, we introduced the idea of a log-based architecture. Logs are a superior approach to database-centric architectures for processing event data for analytics, materialized views, auditing, speculative queries, and more. In this post, we’re going to zero in on one of the most challenging components of implementing log designs: record replay. Record replay rewinds the clock The central idea behind log-oriented architectures are that all events...

Applications based on event streams have more demanding architectural qualities than ever, and traditional approaches to storing, querying and reacting to patterns are tearing at the seams. Business requirements mandate that our systems both record everything that’s ever happened, yet also summarize the entirety of that history with increasingly low latency. Reconciling these attributes and others into a new, unified architecture benefits from a change in perception of the problem. Event streaming is a natural...