RabbitMQ Expert Opinions: RabbitMQ Summit Panel Recap

Earlier this week, I moderated a panel at the first RabbitMQ Summit in London. Organized by Erlang Solutions and 84codes, the event brought together about 150 users and members of the RabbitMQ community. The sessions were fantastic. I learned more about how RabbitMQ works and best practices in those eight hours than the last year.

The organizers will post the session recordings soon. In the meantime, here are my notes from moderating the panel with four RabbitMQ experts:

Use Cases for RabbitMQ

Once upon a time, RabbitMQ was the clear, go-to choice for open source messaging. But with the emergence of Apache Kafka, understanding when to use RabbitMQ has become more nuanced. To help demystify when to use RabbitMQ, I asked the panelist for good and bad use cases for RabbitMQ.

Good use cases for RabbitMQ

Integrating different systems, especially when some are legacy, different languages, etc.

Event streams

When uncertainty is high: you know you need messaging, but not sure what exactly

Building big, microservices architectures

Bad use cases for RabbitMQ

Large, opaque files (e.g. videos, images)

As a data store

Large, long-running queues (UPDATE: clarifying that this refers to queues that keep a permanent backlog and are never emptied. Large and/or long-running aren't inherently problematic on their own...)

What are common pitfalls when using RabbitMQ?

One of RabbitMQ's strengths is its flexibility. But that can also be a challenge and lead to some misuse. I asked the panel what were the most common pitfalls they see among RabbitMQ users.

Common RabbitMQ pitfalls

Turning on all guarantees (e.g. mirroring, persistence, etc.) all at once. There is a cost in performance and stability for these various guarantees...

Stop using Celery: it abstracts too much away

What RabbitMQ features do users overlook?

RabbitMQ has been around for ten years and is hardened to work in a variety of production environments. In many places, it's up and running and "just works". Not surprisingly, there are many features that have come along, but go overlooked. For example, even though RabbitMQ 3.7 came out a year ago, it seemed like many folks weren't aware of the new features. I asked the panelist for their opinion on the most overlooked RabbitMQ features.

Most overlooked RabbitMQ features

Thinking differently about how to scale RabbitMQ

There are different ways to scale RabbitMQ today. Deciding how you should scale RabbitMQ depends on the use case. I asked the panelists to opine a bit on what they would change about the way RabbitMQ scales. On the other hand, I also asked them what they would keep about how RabbitMQ scales today.

What to (potentially) change about how RabbitMQ scales

Separate the management from the messaging processing

Have nodes of different types (e.g. nodes for connections)

Queue location when scaling horizontally

Rebalancing queues*: managing too many high volume queues on the same node

What works well about how RabbitMQ scales

Looking forward to Raft, the algorithms that will be used to reach consensus in a RabbitMQ cluster. This work is in progress and I recommend watching this talk from SprignOne Platform to learn more.

Flexibility to enter the cluster from anywhere

Roadmap highlights

Michael Klishin gave an overview of what the core RabbitMQ engineering is working on. I recommend RabbitMQ users watch that replay when it's available. I also asked the panel which features and capabilities on the roadmap had them most excited.

Most anticipated roadmap features

Quorum queues (see note above about Raft)

Churn statistics

Introducing native protocols

All of the above!

AMQP and the future of RabbitMQ

Gavin Roy kicked off the day with an in-depth look at RabbitMQ's core underlying protocol, AMQP. How AMQP works defines many of the capabilities—and limitations—of RabbitMQ. For example, I learned how short-lived connections are very inefficient for RabbitMQ. This is because of the complex handshake required by AMQP to open that connection.

Given the opinions and expertise of the panel, I asked whether RabbitMQ should move away from AMQP. I also asked how that might be accomplished. I'll note that much of this discussion was not roadmap per se. Rather, the panelists were free to blue sky ideas about RabbitMQ's relationship to an underlying protocol.

The panelists discussed:

Not a fan of AMQP 1.0 and think the spec should be forked (UPDATE: Note that this is not necessarily the opinion of the core RabbitMQ team)

The exchange model is good... - Finding ways to support GRCP and HTTP 2/3

How translating from MQTT to AMQP adds overhead. Note that this is how RabbitMQ works today, but Michael Klishin noted a protocol-agnostic core in his roadmap session...

A first step is to make all protocols more equal

The identity of other protocols is fundamentally different... would have to be generic (to be agnostic). Some potential for a loss in fidelity with this approach, but it's a simplification.

State of the RabbitMQ Community

This was the first RabbitMQ Summit in the technology's eleven-year history. And yet, the popularity of the technology and ample documentation and plug-ins indicate a vibrant community. I asked the panelists what they thought worked well in the RabbitMQ community today. I also asked where they thought the community could do a better job.

RabbitMQ community strengths

RabbitMQ community challenges

Need better onboarding

Need more operator-friendly docs and tutorials

Exposing best practices and helping people find the right patterns

On that last note, there were a ton of best practices shared at the event. We're also hosting a webinar next month on the "10 things every developer using RabbitMQ should know." We'll be organizing those resources to make them more findable for the community on RabbitMQ.com. If you have any other ideas on how to make the RabbitMQ community better, or need clarification on any of these points, please leave a comment or jump on the mailing list.

About the Author

Dormain leads Product Marketing for Pivotal Platform Ecosystem, including GemFire, Pivotal's PCF Services offerings, and ISV offerings for PCF. Previously, she was Director of Product Marketing for Mobile and Pivotal Data Suite. Prior to Pivotal, she was Director of Platform Marketing at Riverbed Technology. Prior to Riverbed, she spent over 5 years as a technology investment analyst, closely following enterprise infrastructure software companies and industry trends. Dormain holds a B. A. in History from the University of California at Los Angeles.