Introduction

ZeroMQ (also spelled ØMQ, 0MQ or ZMQ) is a very lightweight message queuing open source software. It doesn't have a stand-alone server; messages are sent directly from application to application. It is very simple to learn and implement. It is composed of one single library called libzmq.dll written in c++ that can be linked to any application. To use it in the .Net environment we need a wrapper for this library which is called clrzmq.dll written in C#.

ZeroMQ can be run on Windows, OS X, and Linux. Several languages can be used to implement applications using ZeroMQ including C, C++, C#, Java, Python… This gives the ability to communicate with different applications on different platforms.

The heart of ZeroMq

The main part of ZeroMQ is the socket. It's not the traditional socket, but it's a socket that provides a layer of abstraction on top of the traditional socket API, which frees us from the complexity and the repeated tasks that we do in our applications. ZeroMQ supports several types of sockets (the type of the socket is defined as an attribute value in the socket itself). The different combinations of socket types in the sending and receiving ends give us different communication patterns that we will explore some of them in this article.

Asynchronous Communication

The communications done by ZeroMQ are done in an asynchronous way. That means our application will not be blocked during setting up or closing the socket connection, re-connection and message delivery. These operations are managed by ZeroMQ tself in background threads and in parallel to the regular processing done by our application. It queues messages (either at sender or receiver side) automatically when needed. It does this intelligently, pushing messages as close as possible to the receiver before queuing them.

Transport Protocols

ZeroMQ suppports 4 types of transport protocols. Each transport is defined by an address string which consists of two parts: transport://endpoint. The transport part specifies the underlying transport protocol to use, and the endpoint part is defined according to the used protocol as follows:

TCP (tcp://hostname:port): communication over the network

INROC (inproc://name): communication within the same process (between threads)

IPC (ipc:///tmp/filename): inter-process communication within the same host

PGM (pgm://interface;address:port and epgm://interface;address:port): multicast communication over the network

Message Formats

The default message types that we can send or receive are string and array of bytes. ZeroMQ does not impose any format on messages that are sent between sockets. We are free to encode our messages; we can use XML, JSON, MessagePack… In this article we will use only strings for simplicity.

ZeroMQ library (libzmq.dll) will be generated in the bin\win32 sub-directory.

clrzmq.dll:

It's a .NET wrapper of libzmq.dll library written in C#. You can get the source code from https://github.com/zeromq/clrzmq (get version 3.0.0.0 beta). To build it, follow these steps:

Microsoft Visual C# 2010 Express or later

Unpack the .zip source archive.

In Visual C# open the solution src\clrzmq.sln.

Select 'Release' in the 'Solution Configurations' on the tool bar

Check the 'XML documentation file' check box in the 'Build' tab in ZeroMQ project in order to generate the Xml documentation file.

Build the solution.

clrzmq.dll library will be generated in the src\ZeroMQ\bin\Release sub-directory

libzmq.dll (version 3.2.1-beta 2 ) library will be in the \lib(\x86)|(\x64). This library has been downloaded by the project via nuGet packages.

I have built this solution and got the two dlls (x86: Clrzmq.dll and libzmq.dll) and used it in this article solution.

Note: You can get the binaries of the beta release of Clrzmq.dll either from NuGet (you must select “Include Prerelease” in the Manage NuGet Packages Window) or from GitHub. In this package the libzmq.dll (both x86 and x64) are bundled directly in the assembly and are selectively extracted/loaded on application startup (Thanks to John, the maintainer of clrzmq, for his remark).

ZeroMQ Bundle Project

It's the article's solution that I created containing several small console applications allowing us to test different communication patterns in different situations very easily. It contains the two libraries libzmq.dll and clrzmq.dll that I have got from building the clrzmq solution as I mentioned above. Each of these applications has a set of command line parameters (thanks to Giacomo Stelluti Scala for his CommandLine parser open source library), to visualize these parameters you can type the application name followed by the switch /?. I have also included some batch files for each communication pattern containing the necessary commands for running the pattern. These batch files facilitate running the same pattern at any time rapidly.

First of all, we create a context; from this context we can create the famous ZeroMQ socket. The type of socket is defined when we create it. With this socket we can do one of two things:

Bind to an endpoint and wait for connections from other sockets.

Connect to an end point

The selection between bind or connect depends on the communication pattern that we use (explained later in this article).

At last we can send or receive messages. As we can see, we used very few lines to establish a communication. These steps are used everywhere in the communication patterns.

Communication patterns

A communication pattern is a set of connected sockets that specify a message flow. We will use some shapes and symbols for illustrating the connections between the sockets. The following diagram shows the basic connection between sockets:

A rectangle is an application that contains one or more sockets. Each socket can bind or connect to an endpoint. A socket that binds to an endpoint is a socket waiting for connections from other sockets.

You will notice that I have added a delay (can be set to any value in milliseconds) before sending or receiving messages. The purpose of this delay is either to:

Slow sending messages.

Simulate a busy state.

Give some time for socket connections to be completed before sending messages in order not to lose them.

Request/Reply Pattern (REQ/REP)

This pattern has the following characteristics:

The server uses a socket of type REP and the client uses a socket of type REQ.

The client sends a request and receives a reply while the server receives a request and sends a reply.

It allows one client to be connected to one or more servers. In this case the requests are round-robined among all the servers (Reps), one request is send to one server and the next request is sent to the next sever and so on.

State based pattern: The client has to receive a reply for its request before sending another one, and for the server has to send a reply before receiving another request.

Let us explore this pattern using two cases of client-server connections:

1. One Client - One Server.

In this case we have one client (Req) connected to one server (Rep). The following diagram illustrates it:

The first command will start a new colored (/T:fg command) DOS command window and run the application Rep.exe. The Rep application will bind to endpoint tcp://127.0.0.1:5000 and wait for incoming requests. When a request arrives, it will send a reply composed of the incoming request (#msg# macro) concatenated with the word ‘Reply’. The delay before sending the reply is zero milliseconds (-d switch).

The second command will start a new colored (/T:fg command) DOS command window and run the application Req.exe. The Req application will connect to endpoint tcp://127.0.0.1:5000. Then it sends 5 messages (a word ‘Request’ concatenated with the message number (#nb# macro)). It will wait for 1000 milliseconds before sending each request (-d switch).

After running the above commands we get the following result:

2. One client – Two Servers

Here we have one client (Req) connected to two servers (Rep). The following diagram represents this case:

Double click on ReqRep_Patttern_2.bat file under bin directory. This batch file contains the following commands:

The first two commands will run two instances of Rep application; each instance will wait for connections on different port number (5000 and 5001). The last command will run the Req application which will connect to the two running Reps.

After running the above commands we get the following result:

We notice that the requests are round-robined between the two servers (Reps), one request is send to one server and the next request is sent to the other sever. It’s the outgoing routing strategy of the REQ sockets.

Publish/Subscribe Pattern (PUB/SUB)

This pattern has the following characteristics:

The publisher uses socket of type PUB and the subscriber uses socket of type SUB.

One publisher can have one or more subscribers.

On subscriber can be connected to one or more publishers.

Publishers send messages and subscribers receive them.

Subscriber must subscribe either to all publisher messages by using SubscribeAll method or to a specific message by using Subscribe method and specify (as parameter) the prefix of the message the subscriber is interested in.

Subscriber can unsubscribe from either all publisher messages by using UnsubscribeAll method or from a specific message by using Unsubscribe method and specifying the message prefix as the method parameter.

The message filtering happens at:

Publisher side (ZeroMQ 3.x for protocols tcp:// and ipc://). The publisher filters messages before sending them to subscribers.

Subscriber side (ZeroMQ 3.x for protocol epgm:// and ZeroMQ 2.x). The subscriber drops the unwanted messages received from publishers.

If a publisher has no connected subscribers, then messages will be dropped.

A subscriber which is connected to more than one publisher will receive messages evenly (fair-queuing).

Let us explore this pattern using three cases of publisher-subscriber connections:

1. One Publisher - Two Subscribers (all messages subscription)

In this case we have one publisher (PUB) having two connected subscribers (SUB). The following diagram represents this case:

The first two commands will run two instances of subscriber application (Sub.exe). Each subscriber will connect to endpoint tcp://127.0.0.1:5000 and subscribe to all publisher messages (we did not define a subscription prefix which is the default value). The third command will run the publisher (Pub.exe) which will bind to endpoint tcp://127.0.0.1:5000 and wait for connections from subscribers. Then it sends 5 messages (alternates between the word ‘Orange’ and ‘Apple’), each of these words is concatenated with the message number (#nb# macro). The delay between messages is 1000 milliseconds (-d switch).

After running the above batch file we get the following result:

2. One Publisher - Two subscriber (specific message subscription)

In this case we have two subscribers (Sub) connected to one publisher (Pub). The first subscriber will subscribe to receive messages that start with the word “Orange” or “Apple” and the second one will subscribe to receive messages that start with the word “Kiwi”. The following diagram represents this case:

Double click on PubSub_Pattern_2.bat file under bin directory. This batch file contains the following commands:

Notice that the subscriber is connected to two different endpoints which represent the two publishers. The subscriber is subscribed to all messages.

After running the above batch file we get the following result:

The subscriber in this example receives messages evenly from among each connection (publisher), one message from one connection and the next one from the next connection and so on, which is the incoming routing strategy of the SUB socket.

Pipeline pattern (PUSH/PULL)

This pattern is commonly used when there is a need to do parallel data processing. The pipeline pattern scenario is as follows:

Normally we have a task distributor that pushes messages (tasks) to workers in a round-robin fashion (different task for each worker).

When the worker receives the message it will process it and then it will push it to a sort of task collector that receives the messages (tasks).

The messages received by the collector are fair-queued among all connected workers.

This pattern has the following characteristics:

The task distributor uses socket of type PUSH. It binds to its endpoint and waits to receive connections from workers.

A worker has tow sockets, one socket is of type PULL connected to the task distributor socket and the other socket is of type PUSH connected to the collector socket.

The task collector has a socket of type PULL. It binds to its endpoint and waits to receive connections from workers.

The first command will run the Task Distributor which will bind to endpoint tcp://127.0.0.1:5000 and wait for connections. The second command will run the Task Collector which will bind to endpoint tcp://127.0.0.1:5001 and wait for connections. The third and forth commands will run two instances of workers. Each worker will connect to the Task Distributor and collector.

After running the above batch file we get the following result:

Notice that:

the task distributor has distributed tasks between the connected workers, different task for each worker (round-robin), which is the outgoing routing strategy of the PUSH socket.

the task collector has received the processed tasks from workers evenly (fair-queue), which is the incoming routing strategy of the PULL socket.

With this pattern we can simply add other workers without changing any configuration in the task distributor and collector since the workers are connected (not bounded) to their endpoints. We can say that the distributor and the collector are the stable parts of the pattern and the workers are the dynamic parts.

Conclusion

ZeroMQ is a very lightweight open source library. With a few simple lines of code we can build a communication pattern used on the same or over multiple machines having the same or different platforms. The different parts of a pattern can be implemented by the same or different languages.

Comments and Discussions

Hi! Great intro into 0MQ for .net. Thanks! But IPC client side does not work. The server part starts with no issue; the client side gives on Connect "Unknown Protocol" error. Perhaps 0MQ .net wrapper is not yet developed for IPC?

I did allow read write permission to the folder "temp" as well as working directory.
Also there is no firewall on the machine where this is being currently executed.
Any help would be highly appreciated. thanks in advance.

It is a great product. It improved our network reliably immensely -- disconnects and reconnects became non-issues.

Regards,
Phillip

------------------------------------------------------------------------
Phillip Piper www.bigfoot.com/~phillip_piper phillip.piper@gmail.com
A man's life does not consist in the abundance of his possessions

- very fast message throughput
- concurrency: alternative to locking when developing multi threaded applications
- Request Queuing. deal with one request at a time on a single socket, and ZMQ automatically queues up concurrent requests.
- minimal configuration - much simpler than WCF

You need to do the de/serialization yourself which is something extra to bear in mind..