Streaming feature of MessagePack

Streaming serialization/deserialization is one of the key features of MessagePack.
It enables you to store and load multiple objects to/from a stream (like files or network connections). It is easy to use and fast.

I’ll introduce the streaming features with Ruby codes.

Storing objects to a file

Now I want to store structured objects to a log file.
I can serialize objects with MessagePack and store them to a file. And then, I can take from the file objects out one after another.

What’s MessagePack-RPC?

MessagePack-RPC is a inter-process messaging system that is similar to Facebook’s thrift, Apache avro or Google’s Protocol Buffers. It uses MessagePack format for object serialization and passes serialized object across network connection.

MessagePack is cross-language serialization format and MessagePack-RPC enables you to exchange messages between programs implemented in heterogeneous languages.

Today, I want to introduce the C++ implementation of MessagePack-RPC. Its most important characteristics are (1) High-performance, (2) concept of “Future” and (3) Thread-safe connection pool.

(1) High-Performance

One of the most important factors of the performance is parallel processing because even cheap machines have multiple cores today. MessagePack-RPC for C++ has an intelligent parallel processing mechanism that other libraries (Thrift, Avro, Protocol Buffers, etc.) don’t have.

It processes all messages in parallel even if the messages are reached on single connection.

This characteristic is particularly effective in the cluster application that servers communicate mutually.

The following graph shows you the effectiveness of the parallel processing.

(2) Future and Asynchronous call

MessagePack-RPC for C++ provides flexible asynchronous processing model supported by concept of Future. Future describes an object that acts as a proxy for a result that is initially not known (Wikipedia).

Why MessagePack?

Two important points. For speed, and tight integration with each language.

(1) Speed

For speed, MessagePack has zero copy serialization + streaming deserialization. Please look at Introduction to MessagePack. We also compared with other libraries in the view point of speed and size. Please look at here.

(2) Tight Integration with the language

Actually “MessagePack” itself is the name of the format, like JSON. And many libraries handling it were written by many languages. Currently, C, C++, Java, Ruby, Python, Perl, PHP, Haskell, Lua, D implementations are available.

Some implementations provide the easy packaging. For example, Ruby version is provided as Ruby gem.

$ gem install msgpack

Same for other languages (Maven2 for Java, PyPI for Python, CPAN for Perl, Hackage for Haskell).

The library API is basically the same across languages but provides the convenient way for the language. Here’s the deselialization with for loop in Python version. Really cool!

Difference between MessagePack-RPC and Thrift

Release management per language bindings, not by whole project. Formerly, I submitted to the bug fix patch to the PHP binding. But Thrift release cycle is decided by all language bindings. Then, they couldn’t release as soon as possible.