ZeroMQ an introduction

Nicholas Piël| June 23, 2010

ZeroMQ is a messaging library, which allows you to design a complex communication system without much effort. It has been wrestling with how to effectively describe itself in the recent years. In the beginning it was introduced as ‘messaging middleware’ later they moved to ‘TCP on steroids’ and right now it is a ‘new layer on the networking stack’.

I had some trouble understanding ZeroMQ at first and really had to reset my brain. First of all, it is not a complete messaging system such as RabbitMQ or ActiveMQ. I know the guys of Linden Research compared them, but it is apples and oranges. A full fledged messaging system gives you an out of the box experience. Unwrap it, configure it, start it up and you’re good to go once you have figured out all its complexities.

ZeroMQ is not such a system at all; it is a simple messaging library to be used programmatically. It basically gives you a pimped socket interface allowing you to quickly build your own messaging system.

Float like a butterfly, sting like a bee

But why use ZeroMQ and not just use the low level Berkeley socket interface or a high level messaging system? I think the answer is balance. You probably want the flexibility and performance of the low level while still having the ease of implementation of the high level. However, maintaining raw sockets is difficult and cumbersome when you want to implement a scalable system. A high level system often works perfect if you use it for the situation it was designed for, but it can be difficult to change core elements of the system and its ease of use often comes with a cost in performance. This isn’t a problem that is limited to messaging systems only. We can see the previous dilemma also in web frameworks; it could very well be that this is exactly the reason why ‘Micro Frameworks’ gain in popularity.

I believe that ZeroMQ perfectly fits this gap between the high and the low level, so what are its features?

Performance

ZeroMQ is blazing fast. It is orders of magnitude faster than most AMQP messaging systems and it can obtain this high performance because of the following techniques:

It does not have the overhead of an over-engineered protocol such as AMQP

It makes use of intelligent message batching. This allows 0MQ to efficiently utilize a TCP/IP connection by minimizing not only protocol overhead but also system calls.

Simplicity

The API is deceptively simple, and it makes sending messages really simple compared with a raw socket implementation where you have to continuously ‘feed’ the socket buffer. In ZeroMQ you can just fire off an async send call, it will queue the message in a separate thread and do all the work for you. Because of this async nature, your application does not have to waste time waiting until the message has been flushed. The async nature of 0MQ makes it a perfect companion for an event-based framework.

ZeroMQ’s simple wire protocol fits perfectly in the current time setting where we have lots of different transport protocols. With AMQP it always felt a bit weird to use an extra protocol layer on top. 0MQ gives you complete freedom on how you encode your message, as it will just interpret it as a blob. So you can send simple JSON messages, go the binary route with for example BSON, Protocol Buffers or Thrift and all this without feeling guilty.

Scalability

While ZeroMQ sockets look low level they provide lots of features. A single ZeroMQ socket can for example connect to multiple end points and automatically load balance messages over them. Or it can work as some sort of Fan-In, collecting messages from multiple sources through a single socket.

ZeroMQ follows a brokerless design so that there is no single point of failure. Combine this with its simplicity and performance and you get something that you can use to make your application distributed.

Implementing a messaging layer with ZeroMQ

In the next section I will show how to design and implement a messaging layer with ZeroMQ. For the code example I will use Brian Granger’s PyZMQ, which is the excellent Python binding to ZeroMQ.

Implementing a ZeroMQ messaging layer is a three-step approach:

Choose a transport

Set up the infrastructure

Select a messaging pattern

Choosing a transport

The first step is to choosing a transport. ZeroMQ provides 4 different transports:

The TCP transport is often the best choice, it is very performant and robust. However, when there is no need to cross the machine border it can be interesting to look at the IPC or INPROC protocol to lower the latency even more. The MULTICAST transport can be interesting in special cases. But personally, I am a bit careful with applying multicast, as it is difficult to understand how it will behave when scaling up. Think of issues such as figuring out how many multicast groups you can create with this or that hardware and how much stress it is going to put on the different switches in your network. If you want to be sure that your code runs cross platforms it is probably best to go with TCP as the other transports are not guaranteed to be available on the different platforms.

Setting up the infrastructure

When you have decided upon your transport you will have to think about how the different components are connected to each other. It is simply answering the question: “Who connects to whom?” You probably want the most stable part of the network to BIND on a specific port and have the more dynamic parts CONNECT to that. In the image below we have depicted how a server binds to a certain port and how a client connects to it.

It is possible that both ends of the networks are relatively dynamic so that it is difficult to have a single stable connection point. If this is the case, you could make use of the forwarding devices that ZeroMQ provides. These devices can bind to 2 different ports and forward messages from one end to the other. By doing so, the forwarding device can become the stable point in your network where each component can connect to. ZeroMQ provides three kinds of devices:

In the image below we can see such a device being used, in this situation both the client and the server initialize a connection to the forwarder, which binds to two different ports. Using such a device will remove the need of extra application logic, as you will not need to maintain a list of connected peers.

Selecting a message pattern

The previous steps build the infrastructure but did not specify the message flow. The next step is to think carefully about the message pattern each component should follow. The patterns that 0MQ supports are:

Request Reply

The request reply paradigm is very common and can be found in most type of servers. For example: HTTP, POP or IMAP. This pattern has a certain state associated with it as a request has to be followed by a reply. The client uses a socket of type REQ as it will initiate the request by performing a .send() on the socket. The server uses a socket of type REP, and it will start by performing a .recv() to read the incoming request, after which it can send its reply.

ZeroMQ greatly simplifies this pattern by allowing you to have a single socket connect to multiple end points. ZeroMQ will automatically balance requests over the different peers.

The Python code below will create an echo server that listens on port 5000 with a REP socket. It will then loop an alternation of performing .recv() for incoming requests and then .send() a reply to them.

When you have multiple clients connected to this server the ZMQ socket will fair queue between all incoming requests. Now, if you want your client to be able to connect to multiple servers as well, you can take the above code, change port 5000 to 6000 and use it to run an extra server. The following client code will then be able to use both of the servers:

The above sends 10 requests in total but since we are connected to 2 different servers, each server only has to handle 5 requests. Isn’t that great? With only a few lines of code we were able to create a distributed client/server model.

Now, if we want to add an extra server to handle our requests we will have to adjust our code. This can be cumbersome as we need to do this for all our clients to let them know it can now balance the requests over an extra server.

This is exactly where the ZeroMQ devices fit in. Instead of having the clients connect directly to multiple servers it can connect to a single forwarding device. The forwarding device will then reroute all messages to the connected servers.

Publish Subscribe

The Pub/Sub paradigm has gained lots of interest the last few years. You can think of things such as message pushing, XMPP or webhooks. In a pub/sub pattern the components are loosely coupled. This will greatly help you to scale out as there is no need to worry about the subscribers. However, this loose coupling can also lead to unexpected behavior when not fully understood. A nice metaphor for the Pub/Sub paradigm is thinking of it is a radio station. When you publish messages you send something over a certain frequency, only listeners that have subscribed to that frequency will receive the signal. But also, just as with a radio, if you tuned in to the station after the broadcast you will miss the show.

It is good to stress that the various message patterns have no coupling with the infrastructure. It is thus possible to bind to a port and publish to the peers that connect to it. But it is also possible to do it the other way around, connect to multiple peers and broadcast to them. The first example resembles the radio metaphor (everybody can tune in), while the second one more resembles yelling at your peers through a megaphone (a selected group). In both situations your peers can decide not to listen to your messages by not subscribing to them.

The following code shows how you could create a broadcasting server for live soccer events:

Pipelining

The pipeline pattern looks remarkably similar to the Rep/Req pattern, the difference is that instead of requiring a reply being sent to the requester the reply can be pushed down the pipe. This is a paradigm commonly seen when there is a need to process data in parallel. For example, lets say we have some sort of system that does face recognition. We have a job server that pushes the images to one of the workers, which will then process it, once finished it will then push it down the stream again towards some sort of collector.

In the design at the left we can see that a worker will receive its message from an UPSTREAM socket and once they are processed sends them DOWNSTREAM. It routes messages from two different socket types.

The jobserver can just keep pushing tasks DOWNSTREAM through a single socket but with multiple endpoints. ZeroMQ and recently also PyZMQ can send the messages in a zero-copy manner. This is great if you need to push large messages around and you don’t want to waste IO cycles.

Paired sockets

Paired sockets are very similar to regular sockets as the communication is bidirectional, there is no specific state stored within the socket and there can only be one connected peer. Most real life problems can be captured in one of the previously explained patterns and I want to recommend that you look at them first before applying this one as it will simplify your problem.

The figure at the left depicts the infrastructure of a paired socket, the server listens on a certain port and a client connects to it. The red lines indicate the flow of messages, in this pattern both endpoints use a socket of type PAIR and as you can see the messages can flow bidirectional.

The following code shows how to implement such a thing. We will bind to a port on one end:

ZeroMQ and the future

In this post I have given a short introduction to ZeroMQ, I hope that at this point you will now share my ideas about what a great little library it is. But while the library may feel small it has a grand vision of being the new messaging layer. And really, it is not that weird when you come to think of it. Scalability issues are mostly just communication and portability issues, ZeroMQ can solve these problems for you.

Lets say you want to create some new sort of database because Redis, Cassandra, TokyoTyrant, Postgres, MongoDB, DabbleDB, CouchDB, HBase, etc. just don’t serve your needs that well. You create an amazing in memory tree representation for your data and have a blazing fast indexer. Now all you need is some sort of messaging layer such that different clients can talk to your server. Preferably implemented in different programming language and with clustering capabilities. You could of course create such a messaging framework all by yourself, but that is a lot of hard work.

A simple solution is to just implement your database as a ZeroMQ server and pick a message protocol (fe JSON). As you have seen by now, implementing such functionality with ZeroMQ is really easy and on top of this you will get almost instant scalability because of the way ZeroMQ can route messages. It will also make it incredibly easy to implement different clients that will communicate with your server. Basically all you need to do is pick one of the 15 available language bindings, use the same message protocol and you’re done. Currently the following languages have a ZeroMQ binding: Ada, C, C++, Common Lisp, Erlang, Go, Haskell, Java, Lua, .NET, OOC, Perl, PHP, Python and Ruby.

ZeroMQ could very well be the new way in how we connect our components. A good example of someone who understands the possibilities of ZeroMQ is Zed Shaw as can be seen with his recent project Mongrel2. You can use Mongrel2 to bridge the gap between a regular HTTP client and a ZeroMQ component. If you don’t immediately see how awesome this is you probably have never worked with websockets, comet or flash based sockets. Another way to look at the great possibilities of such an implementation is to think of Facebook’s BigPipe where each Pagelet can transparantly be generated by a different component connected with 0MQ.

56 Responses to “ZeroMQ an introduction”

[...] This post was mentioned on Twitter by Igor and zeromq, Pieter Hintjens. Pieter Hintjens said: @nichol4s "ZeroMQ an introduction" http://bit.ly/cUUjF1 – excellent intro to #ZeroMQ and why its the new way to connect our components. [...]

I have been thinking about that, but it would not really be fair. The three are radically different, ZeroMQ will give you the greatest messaging performance, no doubt about that, but it doesn’t support message persistence.

I love Redis but I personally would not use it for realtime Pub/Sub because you will pay in performance and it is not really clear to me how you would route your messages in a more distributed setup. I do think however, that Redis would make an excellent in memory Queue for when you are not interested in realtime performance.

Nice article! We’ve been looking (detailedly) at ZeroMQ as well, and although we really like the ideas, there are a few drawbacks (imho) that are not mentioned in your article:

1. The source code of ZeroMQ is quite difficult to understand! Many concepts, inheritances, relations etc. 2. Many things are not yet finished or are being changed currently. 3. It’s possible that a REQ hangs forever and there is currently no elegant way to solve this (ZeroMQ guys are aware of this, see point 2) 4. Though there are many clients, they are not all fully functional clients. I looked at the Erlang client and it’s so-so. 5. There is quite a long time between releases, which is understandable since only very few people work on the actual code 6. There are many assert() statements in the code, which might make it less suitable for some

All in all, I’m impressed with ZeroMQ and especially the ideas of building a new layer. I really do like it a lot, but get the feeling it’s “not-there-yet”. Hopefully they can fix the most serious problems soon and keep the source code clean without too many confusing concepts.

Very interesting! I’m a newbie to this stuff, so it’s very fascinating.

In your opinion, other commentors included, would 0MQ be a good transport layer for an MPI-type parallel processing library? As I read through this article, it seems like the MPI specs could sit right on top of this type of library.

I know what a pain benchmarking can be so I really appreciate that you guys share the observed results! There is some discussion on the ZeroMQ mailing list concerning these results. I know you already participated in that, but i’ll just mention it here to let other readers know: http://lists.zeromq.org/pipermail/zeromq-dev/2010-June/thread.html

I think the results can be summarized as follows:

* A raw buffer shows a tad better performance for ZeroMQ compared to OpenDDS (170us vs 183us) * You guys argue, and rightfully so, that OpenDDS can handle static typed data where ZeroMQ needs an extra serialization library. When taking this into account you guys show that OpenDDS is actually faster that ZeroMQ + Protocol buffers (205us vs 216us).

However, it looks like ProtoBuf wasn’t optimized for speed, it will be interesting to see if a rerun with this change will make a difference.

I just found this article as I’m looking for a solution to building out a pub/sub network arch. I found the paper with the benchmark really interesting.

I was looking at using DDS but have one hesitating with it. I don’t like the fact that all the implementations highly tie the “message definition” (IDL) to the transport system (DDS). I have seen that each vendors IDL->code generators are a little different so it’s hard to have a single IDL definition that is DDS vendor independent.

Because if this, I was looking at using ProtoBuffers over DDS. That way, the transport is completely independent of the “message definition.” From the paper, it seems that I should be causing myself any extra overhead since this will be a raw buffer as far as DDS is concerned. But then I’m still able to use the “freebies” that DDS provides from the QoS’s. Also, since protobufs have a set way if being encoded, I should still be able to write consume filters.

I wanted to see if anyone can point out any issues to this approach that I’m missing? I’m sure there are some but I can’t think if a major downside. And the test seems to support that I’m not hurting myself by adding the extra layer of protobuf on to DDS.

I believe that ZeroC ICE (http://zeroc.com) includes a component called IceStorm that looks very similar to OpenDDS (including an IDL). It would have been quite interesting to include it in the benchmark.

But if using zmq.h would make the article too long, then one using zmq.hpp would be the next best thing.

Does zeromq have a test suite? Something that’ll test more than speed. Something that’ll try n messages of sizes from 1 to x bytes with random binary content, etc. Plus, tests that are not obvious for newcomers to try, etc.

I really liked your write up. I found your discussion of the 0mq devices (queues, forwarder, etc.) put a whole new perspective on it for me. I’ve been playing Brian Granger’s Python bindings just to get a feel for it. But I don’t see anywhere in his python code anything like the Queue device. Did I miss something? I see where the Queue is implemented in the C++ source code using a Polling object. Does there exist a Python zmq Queue class?

No, there is no Python based Queue by default. But if you are just interested in the forwarder you can just use the C++ one. Also, implementing such a forwarding device with PyZMQ is really easy and can be done in less than 5 lines of code.

You know Nicholas, I actually wrote about this earlier today on my blog. This post has really given me lots of food for thought, I feel that you made many really interesting points. In fact, I wish I had read it before I posted my own post!

I keep coming back to this article. ZeroMQ has latched-on to rather than merely piqued my interest. Each time, though, I run into the same wall: the documentation. For every 1 minute spent reading ZeroMQ documentation, 5 additional minutes worth of questions are raised

The most fundamental questions I was unable to answer for myself were: When would I want to use ZeroMQ and why?

I can sort of see, in my minds eye, where ZeroMQ meets my needs, but that is obscured by past practical experience. It seems to me that for almost all of my immediate use cases, I would wind up using the socket pairs and regretting it

One of my use cases is the offloading of database queries. Sometimes I need a response, but just as often I don’t. And some times I need to offload work to hit the database and have a result forwarded elsewhere (pipelined) for completion.

Mix in the occasional broadcast and it seems like I’m going to start creating a lot of additional overhead for myself in terms of remembering what socket does what…

IMHO ZeroMQ needs to really round off its documentation and provide some more solid/practical use case examples.

* at the time I looked at ZeroMQ it would not explain it’s fundamental concepts (again, complare with Oliver Smith’s post). This leads to people using it for tasks and in ways which don’t play well with ZeroMQ. I gave my students an assignment to solve with ZeroMQ and they were “ouch, why aren’t my messages sent out?”. After asking in the 0mq list we were told that in that specific case, messages won’t be sent out unless they are needed/consumed and that this was “the way async messaging works”. That may be the case however making sure that potential users are made well aware of the fundamental mechanisms/philosphy before they start programming would go a long way to avoid unpleasant surprises.

* ZeroMQ is not error tolerant – if you don’t use it the way you should (and since it doesn’t have abundant documentation) it can be that it won’t return an error code and won’t complain complain on STDERR but just segfault. This can be tough to debug if the “missuse” happened quite some time before in the codepath.

* it’s being hyped a lot and adds to this hype with it’s own claims – so you come to ZeroMQ with high expectations which possibly get disapointed (see other points before), in spite of ZeroMQ possibly being a good product.

That said, it’s very well possible that ZeroMQ solves the posed problems in a brilliant way and it’s a good idea to use it however one needs to be aware of the current state of affairs.

Tomáš, I agree with almost all of your points. I think the main thing to keep in mind is that ZeroMQ is still a very young project and thus lacking in documentation and decent handling of configuration errors. These aspects are closely related.

I believe that the more people that will use and try it the more user friendly it will become because of the community effort. I am not sure if I will call it ‘hype’ that surrounds ZeroMQ, but there sure are a lot of people getting excited about the (funded) strong performance claims of 0MQ.

I think such excitement is a requirement for any young project to grow. Don’t get me wrong, I think it is good that you point to some of the more negative aspects of ZeroMQ and I must admit that I purposely left out that 0MQ still is a young project. But I think the following quote applies here: “If you want to build a ship, don’t drum up people to collect wood, divide the work and give orders. Instead, teach them to long for the endless immensity of the sea.”

Hah! I was just about to reply on your earlier comment and I noted you have been writing some posts on your blog since then as well.

I agree with both of your points, that indeed the documentation is lacking somewhat but that the struggle with the documentation is worth it. Also, there is a wiki on zeromq.org so I expect that a community effort will solve the documentation aspects at some time.

Nicholas, you mentioned that you would not use Redis for realtime Pub/Sub because you’ll pay in performance. Can you expand on that?

I’ve been using Redis so far only for it’s Key-Value store and it’s awesome for that type of usage, but i was considering using it’s pub/sub features as well. However your comment made me stop and think.

* With ZeroMQ you can have Pub/Sub between two nodes without an intermediate server and thus obtain lower latency. * ZeroMQ will allow you to scale up to really large numbers because of the way it can distribute it messages or the way it uses multicast. With Redis you are always limited to the maximum amount of connections on a single machine. * ZeroMQ uses message batching techniques and is thoroughly optimized for throughput performance this is a completely different design goal than Redis has. It would not surprise me if ZeroMQ would be a magnitude faster.

On the other hand, if you expect only a handful of listeners, really want message persistence and don’t care that much about latency I think Redis could be a viable option as it will make your design very simple.

The Python bindings also have some kinks that need ironing out: the worst one to bit me was that there can be exactly one zmq.Context in existence per process, try to instantiate it the 2nd time and python crashes, so it really should be a singleton at the bindings level, UPSTREAM/DOWNSTREAM are confusing (they do exactly the opposite thing one would expect: upstream reads, downstream writes) and deprecated in favour of PUSH/PULL in the underlying library, select blocks everything (signal handling? nope, unless it’s kill -9)…

/me right now in the process of writing a distributed Python task queue thingy

A req does not hang forever if you do a non-blocking receive. With a request socket, you do a Send and then do a Receive. The Receive will hang waiting for the response, but you can instead do a non-blocking receive. In c#:

I recently discovered ZeroMQ and I was attracted by your article in the “ZeroMQ and the future” section that talks about creating a new database. Well, that’s what I want to do, to create a new database and I’m looking at how to get clients connected to it. After reading all your article, I still don’t understand how a classical approach with: 1. connect client 2. run query 3. get a lot of data as fast as possible will work with ZeroMQ.

Especially sending a lot of data does not seem to be done easily with ZeroMQ. I guess you need to create PAIR sockets on the fly for that to work, so one would need to open lots of ports.

Nicholas: You rock! This introduction to 0mq saved me a ton of time for a project on a very tight deadline. Best of luck with your work, and congrats on securing another dedicated blog subscriber (yours truly!) ^_^ ~Eric

The only one concern that makes me unable to use ZeroMQ is IPC support for windows.

I have a application that ZeroMQ can perfectly simplify our code, but we need cross platform IPC support for that. Now we use boost::interprocess but it sucks (unreliable). I am looking for ZeroMQ’s windows IPC support for a long time. If ZeroMQ support IPC on windows, I will switch to ZeroMQ immediately.

I have read Request Reply paradigm description and would like to clarify a couple concerns. Nicholas, to automatically balance requests you suggest connecting one more socket on client side. Requests balancing on client side? Client should be aware about all servers? Who will support the server list on client side? How to manage clients if you have access to server side only? Are you serious about requests balancing on client side? Balance the requests over an extra server (on diagram named as Queue) – bottleneck, isn’t it?