It's like JSON.but fast and small.

MessagePack is an efficient binary serialization format. It lets you exchange data among multiple languages like JSON. But it's faster and smaller. Small integers are encoded into a single byte, and typical short strings require only one extra byte in addition to the strings themselves.

Next:

Redis scripting has support for MessagePack because it is a fast and compact serialization format with a simple to implement specification. I liked it so much that I implemented a MessagePack C extension for Lua just to include it into Redis.

Salvatore Sanfilippo, creator of Redis

Fluentd uses MessagePack for all internal data representation. It's crazy fast because of zero-copy optimization of msgpack-ruby. Now MessagePack is an essential component of Fluentd to achieve high performance and flexibility at the same time.

Sadayuki Furuhashi, creator of Fluentd

Treasure Data built a multi-tenant database optimized for analytical queries using MessagePack. The schemaless database is growing by billions of records every month. We also use MessagePack as a glue between components. Actually we just wanted a fast replacement of JSON, and MessagePack is simply useful.

Kazuki Ohta, CTO

MessagePack has been simply invaluable to us. We use MessagePack + Memcache to cache many of our feeds on Pinterest. These feeds are compressed and very quick to unpack thanks to MessagePack while Memcache gives us fast atomic pushes.

Feature: Use your own serialization/deserialization routines for custom class and struct types.

msgpack-d provides the functions registerPackHandler / registerUnpackHandler to allow you
to use custom routines during the serialization or deserialization of user-defined class and struct types.
This feature is especially useful when serializing a derived class object when that object is statically
typed as a base class object.

Copyright

License

MessagePack for Python

What's this

MessagePack is an efficient binary serialization format.
It lets you exchange data among multiple languages like JSON.
But it's faster and smaller.
This package provides CPython bindings for reading and writing MessagePack data.

As an alternative to iteration, Unpacker objects provide unpack,
skip, read_array_header and read_map_header methods. The former two
read an entire message from the stream, respectively de-serialising and returning
the result, or ignoring it. The latter two methods return the number of elements
in the upcoming container, so that each element in an array, or key-value pair
in a map, can be unpacked or skipped individually.

Each of these methods may optionally write the packed data it reads to a
callback function:

Notes

string and binary type

Early versions of msgpack didn't distinguish string and binary types (like Python 1).
The type for representing both string and binary types was named raw.

For backward compatibility reasons, msgpack-python will still default all
strings to byte strings, unless you specify the use_bin_type=True option in
the packer. If you do so, it will use a non-standard type called bin to
serialize byte arrays, and raw becomes to mean str. If you want to
distinguish bin and raw in the unpacker, specify raw=False.

Only in packing. Atoms are packed as binaries. Default value is pack.
Otherwise, any term including atoms throws badarg.

{known_atoms, [atom()]}

Both in packing and unpacking. In packing, if an atom is in this list
a binary is encoded as a binary. In unpacking, msgpacked binaries are
decoded as atoms with erlang:binary_to_existing_atom/2 with encoding
utf8. Default value is an empty list.

Even if allow_atom is none, known atoms are packed.

{unpack_str, as_binary|as_list}

A switch to choose decoded term style of str type when unpacking.
Only available at new spec. Default is as_list.

At both. The default behaviour in case of facing ext data at decoding
is to ignore them as its length is known.

Now msgpack-erlang supports ext type. Now you can serialize everything
with your original (de)serializer. That will enable us to handle
erlang- native types like pid(), ref() contained in tuple(). See
test/msgpack_ext_example_tests.erl for example code.

> publishLocal # Install to local .ivy2 repository
> publish # Publishing a snapshot version to the Sonatype repository
> release # Run the release procedure (set a new version, run tests, upload artifacts, then deploy to Sonatype)

For publishing to Maven central, msgpack-scala uses sbt-sonatype plugin. Set Sonatype account information (user name and password) in the global sbt settings. To protect your password, never include this file in your project.

MessagePack for CLI

What is it?

This is MessagePack serialization/deserialization for CLI (Common Language Infrastructure) implementations such as .NET Framework, Silverlight, Mono (including Moonlight.)
This library can be used from ALL CLS compliant languages such as C#, F#, Visual Basic, Iron Python, Iron Ruby, PowerShell, C++/CLI or so.

How to build

You must install .NET Framework 3.5, 4.x, .NET Core, and Xamarin dev tools to build all builds successfully.
If you do not want to install options, edit <TargetFrameworks> element in *.csproj files to exclude platforms you want to exclude.

Install latest .NET Core SDK.

Run with Visual Studio Developer Command Prompt:

msbuild MsgPack.sln /t:Restore
msbuild MsgPack.sln

Or (for Unity 3D drops):

msbuild MsgPack.compats.sln /t:Restore
msbuild MsgPack.compats.sln

Or (for Windows Runtime/Phone drops and Silverlight 5 drops):

msbuild MsgPack.Windows.sln /t:Restore
msbuild MsgPack.Windows.sln

Or (for Xamarin unit testing, you must have Xamarin Business or upper license and Mac machine on the LAN to build on Windows):

msbuild MsgPack.Xamarin.sln /t:Restore
msbuild MsgPack.Xamarin.sln

Or open one of above solution files in your IDE and run build command in it.

For Mono

Install latest Mono and .NET Core SDK.

Now, you can build MsgPack.sln and MsgPack.Xamarin.sln with above instructions and msbuild in latest Mono. Note that xbuild does not work because it does not support latest csproj format.

Own Unity 3D Build

First of all, there are binary drops on github release page, you should use it to save your time.
Because we will not guarantee source code organization compatibilities, we might add/remove non-public types or members, which should break source code build.
If you want to import sources, you must include just only described on MsgPack.Unity3D.csproj.
If you want to use ".NET 2.0 Subset" settings, you must use just only described on MsgPack.Unity3D.CorLibOnly.csproj file, and define CORLIB_ONLY compiler constants.

msgpack for C/C++

Version 2.1.5

It's like JSON but small and fast.

Overview

MessagePack is an efficient binary serialization
format, which lets you exchange data among multiple languages like JSON,
except that it's faster and smaller. Small integers are encoded into a
single byte while typical short strings require only one extra byte in
addition to the strings themselves.

as3-msgpack was designed to work with the interfaces IDataInput and IDataOutput, thus the API might be easily connected with the native classes that handle binary data (such as ByteArray, Socket, FileStream and URLStream).
Moreover, as3-msgpack is capable of decoding data from binary streams.
Get started: http://loteixeira.github.io/lib/2013/08/19/as3-msgpack/

MessagePack for R6RS Scheme

API references

Pack message to message pack format bytevector and put it into the
bv destructively. Given bv must have enough length to hold the message.

Optional argument offset indicates where to start with, default is 0.

Function (pack message)

The same as pack! but this one creates a new bytevector.

Function (pack-size message)

Calculate the converted message size.

Function (unpack bv)Function (unpack bv offset)

Unpack the given message format bytevector to Scheme object.

Optional argument offset indicates where to start with, default is 0.

Function (get-unpack in)

Unpack the given binary input port to Scheme object.

Conversion rules

As you already know, Scheme doesn't have static types so the conversion of
Scheme objects to message pack data might cause unexpected results. To avoid
it, I will describe how conversion works.

Scheme to message packInteger conversion

The library automatically decides proper size. More specifically, if it
can fit to message pack's fixnum then library uses it, so are uint8-64.
If the number is too big, then an error is raised. Users must know it tries
to use uint as much as possible. If the given number was negative then
sint will be used.

Floating point conversion

Unfortunately R6RS doesn't have difference between float and double. So
when flonum is given then it always converts to double number.

Collection conversion

Message pack has collections which are map and array. And these are associated
with alist (association list) and vector respectively. When you want to convert
alist to message pack data, then you need to make sure the cdr part will be
the data and if you put (("key" "value))_ then it will be converted to nested
map.

The collection size calculation is done automatically. It tries to use the
smallest size.

Message pack to Scheme

The other way around is easy, it can simply restore the byte data to Scheme
object. Following describes the conversion rules;

u-msgpack-python

u-msgpack-python is a lightweight MessagePack serializer and deserializer module written in pure Python, compatible with both Python 2 and 3, as well CPython and PyPy implementations of Python. u-msgpack-python is fully compliant with the latest MessagePack specification.

NOTE: The standard method for encoding integers in msgpack is to use the most compact representation possible, and to encode negative integers as signed ints and non-negative numbers as unsigned ints.

For compatibility with other implementations, I'm following this convention. On the unpacking side, every integer type becomes an Int64 in Julia, unless it doesn't fit (ie. values greater than 2^63 are unpacked as Uint64).

I might change this at some point, and/or provide a way to control the unpacked types.

The Extension Type

The MsgPack spec defines the extension type to be a tuple of (typecode, bytearray) where typecode is an application-specific identifier for the data in bytearray. MsgPack.jl provides support for the extension type through the Ext immutable.

MsgPack reserves typecodes in the range [-128, -1] for future types specified by the MsgPack spec. MsgPack.jl enforces this when creating an Ext but if you are packing an implementation defined extension type (currently there are none) you can pass impltype=true.

clojure-msgpack

clojure-msgpack is a lightweight and simple library for converting
between native Clojure data structures and MessagePack byte formats.
clojure-msgpack only depends on Clojure itself; it has no third-party
dependencies.

clojure-msgpack provides a streaming API for situations where it is more
convenient or efficient to work with byte streams instead of fixed byte arrays
(e.g. size of object is not known ahead of time).

Tutorial

Known Problems

Our recommendation is MLton, MLKit, Poly/ML and SML#(>=2.0.0) as all tests passed on them.
SML/NJ and Moscow ML are fine if you don't use real values.

SML/NJ

Packing real values fail or produces imprecise results in some cases.

Moscow ML

Packing real values is not supported, since some components of the SML Basis Library are not provided.

HaMLet

Packing real values is not supported, since some components of the SML Basis Library are not provided.
Some functions are very slow, although they work properly. (We tested HaMLet compiled with MLton.)

Alice ML

Packing real values is not supported, since some components of the SML Basis Library are not provided.
Also, some unit tests fail.

SML#

Most functions do not work properly because of bugs of SML# prior to version 2.0.0.

See Also

There already exists another MessagePack implemenatation for SML,
called MsgPack-SML, which is targeted for MLton.

CMP is thread aware; while contexts cannot be shared between threads, each
thread may use its own context freely.

CMP is tested using the MessagePack test suite as well as a large set of custom
test cases. Its small test program is compiled with clang using -Wall -Werror -Wextra ... along with several other flags, and generates no compilation
errors in either clang or GCC.

CMP's source is written as readably as possible, using explicit, descriptive
variable names and a consistent, clear style.

CMP's source is written to be as secure as possible. Its testing suite checks
for invalid values, and data is always treated as suspect before it passes
validation.

CMP's API is designed to be clear, convenient and unsurprising. Strings are
null-terminated, binary data is not, error codes are clear, and so on.

CMP provides optional backwards compatibility for use with other MessagePack
implementations that only implement version 4 of the spec.

Building

There is no build system for CMP. The programmer can drop cmp.c and cmp.h
in their source tree and modify as necessary. No special compiler settings are
required to build it, and it generates no compilation errors in either clang or
gcc.

Versioning

CMP's versions are single integers. I don't use semantic versioning because
I don't guarantee that any version is completely compatible with any other. In
general, semantic versioning provides a false sense of security. You should be
evaluating compatibility yourself, not relying on some stranger's versioning
convention.

Stability

I only guarantee stability for versions released on
the releases page. While rare, both master and develop
branches may have errors or mismatched versions.

Backwards Compatibility

Version 4 of the MessagePack spec has no BIN type, and provides no STR8
marker. In order to remain backwards compatible with version 4 of MessagePack,
do the following:

Field names can be set in much the same way as the encoding/json package. For example:

typePersonstruct {
Namestring`msg:"name"`Addressstring`msg:"address"`Ageint`msg:"age"`Hiddenstring`msg:"-"`// this field is ignored
unexported bool// this field is also ignored
}

By default, the code generator will satisfy msgp.Sizer, msgp.Encodable, msgp.Decodable,
msgp.Marshaler, and msgp.Unmarshaler. Carefully-designed applications can use these methods to do
marshalling/unmarshalling with zero heap allocations.

While msgp.Marshaler and msgp.Unmarshaler are quite similar to the standard library's
json.Marshaler and json.Unmarshaler, msgp.Encodable and msgp.Decodable are useful for
stream serialization. (*msgp.Writer and *msgp.Reader are essentially protocol-aware versions
of *bufio.Writer and *bufio.Reader, respectively.)

As long as the declarations of MyInt and Data are in the same file as Struct, the parser will determine that the type information for MyInt and Data can be passed into the definition of Struct before its methods are generated.

Mostly stable, in that no breaking changes have been made to the /msgp library in more than a year. Newer versions
of the code may generate different code than older versions for performance reasons. I (@philhofer) am aware of a
number of stability-critical commercial applications that use this code with good results. But, caveat emptor.

You can read more about how msgp maps MessagePack types onto Go types in the wiki.

Here some of the known limitations/restrictions:

Identifiers from outside the processed source file are assumed (optimistically) to satisfy the generator's interfaces. If this isn't the case, your code will fail to compile.

Like most serializers, chan and func fields are ignored, as well as non-exported fields.

Encoding of interface{} is limited to built-ins or types that have explicit encoding methods.

Maps must have string keys. This is intentional (as it preserves JSON interop.) Although non-string map keys are not forbidden by the MessagePack standard, many serializers impose this restriction. (It also means any well-formed struct can be de-serialized into a map[string]interface{}.) The only exception to this rule is that the deserializers will allow you to read map keys encoded as bin types, due to the fact that some legacy encodings permitted this. (However, those values will still be cast to Go strings, and they will be converted to str types when re-encoded. It is the responsibility of the user to ensure that map keys are UTF-8 safe in this case.) The same rules hold true for JSON translation.

If the output compiles, then there's a pretty good chance things are fine. (Plus, we generate tests for you.) Please, please, please file an issue if you think the generator is writing broken code.

As one might expect, the generated methods that deal with []byte are faster for small objects, but the io.Reader/Writer methods are generally more memory-efficient (and, at some point, faster) for large (> 2KB) objects.

Multicast UDP example

Example servers join to group 224.0.0.5 and listen on port 8000. Their only
method echo returns its parameter.

Client joins group to 224.0.0.5, sends multicast request to group on port 8000
and waits for 5 seconds for responses. If some responses are received,
protocol callbacks with tuple of results and individual parts are checked for
errors. If no responses are received, protocol errbacks with TimeoutError.

Because there is no common way to determine number of peers in group,
MsgpackMulticastDatagramProtocol always wait for responses until waitTimeout
expires.

Since J has no native Dictionary / Hashmap type, one has been implemented for the purposes of MsgPack serialization.

Construction:

`HM =: '' conew 'HashMap'`

This will instantiate a new HashMap object.

`set__HM 'key';'value'`

This will add a key value pair to the dicitonary. Note the length of the boxed array argument must be two. i.e. if the value is an array itself, then it must be boxed together before appending to the key value.

`get__HM 'key'`

This will return the value for the given key, if one exists.

To pack a HashMap:

`packObj s: HM`

Here HM is the HashMap reference name. It must be symbolized first, before packing. Furthermore, to add a HashMap as a value of another HashMap:

`set__HM 'hashmapkey';s:HM2`

The inner HashMap reference (HM2) must be symbolized before adding to the dictionary. If you are adding a list of HashMaps to the parent HashMap:

`set__HM 'key'; <(s:HM2;s:HM3;s:HM4)`

Note the HashMap array is boxed so that the argument for set is of length two. Since the HashMap HM stores the reference to the child HashMaps as symbols, they must be desymbolized if retrieved. e.g.

`ChildHM =: getHashmapFromValue_HashMap_ get__HM 'mychildHashMapkey'`

Here, getHashmapFromValue_HashMap_ ensures that the retrieved object is a reference to a hashmap, as is wanted.

When unpacking data, assuming the root object is a dictionary / hashmap:

`HM =: 5 s: unpackObj 'some serialized data'`

5 s: must be called to desymbolize the reference to the HashMap. Furthermore, all child HashMaps of HM must also be desymbolized too.

msgpack-nim

msgpack-nim currently provides only the basic functionality.
Please see what's listed in Todo section. Compared to other language bindings, it's well-tested by
1000 auto-generated test cases by Haskell QuickCheck, which always runs
on every commit to Github repository. Please try make quickcheck on your local machine
to see what happens (It will take a bit while. Be patient). Have a nice packing!

The core of MPack contains a buffered reader and writer, and a tree-style parser that decodes into a tree of dynamically typed nodes. Helper functions can be enabled to read values of expected type, to work with files, to allocate strings automatically, to check UTF-8 encoding, and more. The MPack featureset can be configured at compile-time to set which features, components and debug checks are compiled, and what dependencies are available.

The MPack code is small enough to be embedded directly into your codebase. The easiest way to use it is to download the amalgamation package and insert the source files directly into your project. Copy mpack.h and mpack.c into to your codebase, and copy mpack-config.h.sample as mpack-config.h. You can use the defaults or edit it if you'd like to customize the MPack featureset.

Note that no additional error handling is needed in the above code. If the file is missing or corrupt, if map keys are missing or if nodes are not in the expected types, special "nil" nodes and false/zero values are returned and the tree is placed in an error state. An error check is only needed before using the data.

The above example decodes into allocated pages of nodes. A fixed node pool can be provided to the parser instead in memory-constrained environments. For maximum performance and minimal memory usage, the Expect API can be used to parse data of a predefined schema.

In the above example, we encode to a growable memory buffer. The writer can instead write to a pre-allocated or stack-allocated buffer, avoiding the need for memory allocation. The writer can also be provided with a flush function (such as a file or socket write function) to call when the buffer is full or when writing is done.

If any error occurs, the writer is placed in an error state. The writer will flag an error if too much data is written, if the wrong number of elements are written, if the data could not be flushed, etc. No additional error handling is needed in the above code; any subsequent writes are ignored when the writer is in an error state, so you don't need to check every write for errors.

Note in particular that in debug mode, the mpack_finish_map() call above ensures that two key/value pairs were actually written as claimed, something that other MessagePack C/C++ libraries may not do.

Comparison With Other Parsers

MPack is rich in features while maintaining very high performance and a small code footprint. Here's a short feature table comparing it to other C parsers:

A larger feature comparison table is available here which includes descriptions of the various entries in the table.

This benchmarking suite compares the performance of MPack to other implementations of schemaless serialization formats. MPack outperforms all JSON and MessagePack libraries, and in some tests MPack is several times faster than RapidJSON for equivalent data.

Why Not Just Use JSON?

Conceptually, MessagePack stores data similarly to JSON: they are both composed of simple values such as numbers and strings, stored hierarchically in maps and arrays. So why not just use JSON instead? The main reason is that JSON is designed to be human-readable, so it is not as efficient as a binary serialization format:

Compound types such as strings, maps and arrays are delimited, so appropriate storage cannot be allocated upfront. The whole object must be parsed to determine its size.

Strings are not stored in their native encoding. Special characters such as quotes and backslashes must be escaped when written and converted back when read.

Numbers are particularly inefficient (especially when parsing back floats), making JSON inappropriate as a base format for structured data that contains lots of numbers.

Binary data is not supported by JSON at all. Small binary blobs such as icons and thumbnails need to be Base64 encoded or passed out-of-band.

The above issues greatly increase the complexity of the decoder. Full-featured JSON decoders are quite large, and minimal decoders tend to leave out such features as string unescaping and float parsing, instead leaving these up to the user or platform. This can lead to hard-to-find platform-specific and locale-specific bugs, as well as a greater potential for security vulnerabilites. This also significantly decreases performance, making JSON unattractive for use in applications such as mobile games.

While the space inefficiencies of JSON can be partially mitigated through minification and compression, the performance inefficiencies cannot. More importantly, if you are minifying and compressing the data, then why use a human-readable format in the first place?

Running the Unit Tests

The MPack build process does not build MPack into a library; it is used to build and run the unit tests. You do not need to build MPack or the unit testing suite to use MPack.

On Linux, the test suite uses SCons and requires Valgrind, and can be run in the repository or in the amalgamation package. Run scons to build and run the test suite in full debug configuration.

On Windows, there is a Visual Studio solution, and on OS X, there is an Xcode project for building and running the test suite.

You can also build and run the test suite in all supported configurations, which is what the continuous integration server will build and run. If you are on 64-bit, you will need support for cross-compiling to 32-bit, and running 32-bit binaries with 64-bit Valgrind. On Ubuntu, you'll need libc6-dbg:i386. On Arch you'll need gcc-multilib or lib32-clang, and valgrind-multilib. Use scons all=1 -j16 (or some appropriate thread count) to build and run all tests.

Features

Convenient API

RMP is designed to be lightweight and straightforward. There are low-level API, which gives you
full control on data encoding/decoding process and makes no heap allocations. On the other hand
there are high-level API, which provides you convenient interface using Rust standard library and
compiler reflection, allowing to encode/decode structures using derive attribute.

Zero-copy value decoding

RMP allows to decode bytes from a buffer in a zero-copy manner easily and blazingly fast, while Rust
static checks guarantees that the data will be valid as long as the buffer lives.

Clear error handling

RMP's error system guarantees that you never receive an error enum with unreachable variant.

Robust and tested

This project is developed using TDD and CI, so any found bugs will be fixed without breaking
existing functionality.

Requirements

Rust 1.16

Versioning

This project adheres to Semantic Versioning. However until 1.0.0 comes there
will be the following rules:

Any API/ABI breaking changes will be notified in the changelog explicitly and results in minor
version bumping.

msgpack4nim

I am fully aware of another msgpack implementation written in nim. But I want something easier to use. Another motivation come from the nim language itself. The current version of nim compiler offer many improvements, including 'generics ' specialization. I found out nim compiler is smart enough to make serialization/deserialization to/from msgpack easy and convenient.

object and tuple

object and tuple by default converted to msgpack array, however
you can tell the compiler to convert it to map by supplying --define:msgpack_obj_to_map

nim c --define:msgpack_obj_to_map yourfile.nim

or --define:msgpack_obj_to_stream to convert object/tuple fields value into stream of msgpack objects

nim c --define:msgpack_obj_to_stream yourfile.nim

What this means? It means by default, each object/tuple will be converted to one msgpack array contains
field(s) value only without their field(s) name.

If you specify that the object/tuple will be converted to msgpack map, then each object/tuple will be
converted to one msgpack map contains key-value pairs. The key will be field name, and the value will be field value.

If you specify that the object/tuple will be converted to msgpack stream, then each object/tuple will be converted
into one or more msgpack's type for each object's field and then the resulted stream will be concatenated
to the msgpack stream buffer.

Which one should I use?

Usually, other msgpack libraries out there convert object/tuple/record/struct or whatever structured data supported by
the language into msgpack array, but always make sure to consult the documentation first.
If both of the serializer and deserializer agreed to one convention, then usually there will be no problem.
No matter which library/language you use, you can exchange msgpack data among them.

ref-types:

ref something :

if ref value is nil, it will be packed into msgpack nil, and when unpacked, you will get nil too

if ref value not nil, it will be dereferenced e.g. pack(val[]) or unpack(val[])

ref subject to some restriction. see restriction below

ptr will be treated like ref during pack

unpacking ptr will invoke alloc, so you must dealloc it

circular reference:
altough detecting circular reference is not too difficult(using set of pointers), the current implementation does not provide circular reference detection. If you pack something contains circular reference, you know something bad will happened

Restriction:
For objects their type is not serialized. This means essentially that it does not work if the object has some other runtime type than its compiletime type:

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

mruby-simplemsgpack

Installation

mruby-simplemsgpack searches for msgpack-c on your system, if it can find it it links against it, there is also a bundled version of msgpack-c included if you don't have it installed in your system.
You need at least msgpack-c 1.

When using MessagePack.unpack with a block and passing it a incomplete packed Message
it returns the number of bytes it was able to unpack, if it was able to unpack the howl Message it returns self.
This is helpful if the given data contains an incomplete
last object and we want to continue unpacking after we have more data.

For nil, true, false, Fixnum, Float, String, Array and Hash a registered
ext type is ignored. They are always packed according to the MessagePack
specification.

Proc, blocks or lambas

If you want to pack and unpack mruby blocks take a look at the mruby-proc-irep-ext gem, it can be registered like the other extension types

Overriding to_msgpack

It's not supported to override to_msgpack, MessagePack.pack ignores it, same when that object is included in a Hash or Array.
This gem treats objects like ruby does, if you want to change the way your custom Class gets handled you can add to_hash, to_ary, to_int or to_str methods so it will be packed like a Hash, Array, Fixnum or String (in that order) then.

The first argument of addExtPacker and addExtUnpacker should be an integer within the range of 0 and 127 (0x0 and 0x7F). myClassPacker is a function that accepts an instance of MyClass, and should return a buffer representing that instance. myClassUnpacker is the opposite: it accepts a buffer and should return an instance of MyClass.

If you pass an array of functions to addExtPacker or addExtUnpacker, the value to be encoded/decoded will pass through each one in order. This allows you to do things like this:

codec.addExtPacker(0x00, Date, [Number, msgpack.encode]);

You can also pass the codec option to msgpack.Decoder(options), msgpack.Encoder(options), msgpack.createEncodeStream(options), and msgpack.createDecodeStream(options).

If you wish to modify the default built-in codec, you can access it at msgpack.codec.preset.

Custom Codec Options

msgpack.createCodec() function accepts some options.

It does NOT have the preset extension types defined when no options given.

var codec =msgpack.createCodec();

preset: It has the preset extension types described above.

var codec =msgpack.createCodec({preset:true});

safe: It runs a validation of the value before writing it into buffer. This is the default behavior for some old browsers which do not support ArrayBuffer object.

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

Introduction

msgpack-tools contains simple command-line utilities for converting from MessagePack to JSON and vice-versa. They support options for lax parsing, lossy conversions, pretty-printing, and base64 encoding.

msgpack2json -- Convert MessagePack to JSON

json2msgpack -- Convert JSON to MessagePack

They can be used for dumping MessagePack from a file or web API to a human-readable format, or for converting hand-written or generated JSON to MessagePack. The lax parsing mode supports comments and trailing commas in JSON, making it possible to hand-write your app or game data in JSON and convert it at build-time to MessagePack.

For other platforms, msgpack-tools must be built from source. Download the msgpack-tools tarball from the latest release page (not the "source code" archive generated by GitHub, but the actual release package.)

msgpack-tools uses CMake. A configure wrapper is provided that calls CMake, so you can simply run the usual:

./configure && make && sudo make install

If you are building from the repository, you will need md2man to generate the man pages.

Differences between MessagePack and JSON

MessagePack is intended to be very close to JSON in supported features, so they can usually be transparently converted from one to the other. There are some differences, however, which can complicate conversions.

These are the differences in what objects are representable in each format:

JSON keys must be strings. MessagePack keys can be any type, including maps and arrays.

JSON supports "bignums", i.e. integers of any size. MessagePack integers must fit within a 64-bit signed or unsigned integer.

JSON real numbers are specified in decimal scientific notation and can have arbitrary precision. MessagePack real numbers are in IEEE 754 standard 32-bit or 64-bit binary.

MessagePack supports binary and extension type objects. JSON does not support binary data. Binary data is often encoded into a base64 string to be embedded into a JSON document.

A JSON document can be encoded in UTF-8, UTF-16 or UTF-32, and the entire document must be in the same encoding. MessagePack strings are required to be UTF-8, although this is not enforced by many encoding/decoding libraries.

By default, msgpack2json and json2msgpack convert in strict mode. If an object in the source format is not representable in the destination format, the converter aborts with an error. A lax mode is available which performs a "lossy" conversion, and base64 conversion modes are available to support binary data in JSON.

In the examples above, the method pack automatically packs a value depending on its type.
But not all PHP types can be uniquely translated to MessagePack types. For example,
MessagePack format defines map and array types, which are represented by a single array
type in PHP. By default, the packer will pack a PHP array as a MessagePack array if it
has sequential numeric keys, starting from 0 and as a MessagePack map otherwise:

Automatically detecting an MP type of PHP arrays/strings adds some overhead which can be noticed
when you pack large (16- and 32-bit) arrays or strings. However, if you know the variable type
in advance (for example, you only work with UTF-8 strings or/and associative arrays), you can
eliminate this overhead by forcing the packer to use the appropriate type, which will save it
from running the auto detection routine:

If the packed data is received in chunks (e.g. when reading from a stream), use the tryUnpack
method, which will try to unpack data and return an array of unpacked data instead of throwing an InsufficientDataException:

The binary MessagePack format has unsigned 64-bit as its largest integer data type,
but PHP does not support such integers. By default, while unpacking uint64 value
the library will throw an IntegerOverflowException.

You can change this default behavior to unpack uint64 integer to a string:

In addition to the basic types,
the library provides the functionality to serialize and deserialize arbitrary types.
To do this, you need to create a transformer, that converts your type to a type, which can be handled by MessagePack.

If you don't want to unpack every single thing included in the message-pack byte array, you can also specify an amount to unpack, if you want to keep the remaining bytes, you can put true in the returnRemainingBytes argument, the remaining bytes will stored in the end of the NSData array.

MPack

This library is a lightweight implementation of the MessagePack binary serialization format. MessagePack is a 1-to-1 binary representation of JSON, and the official specification can be found here: https://github.com/msgpack....

Notes

Its easiest to understand how this library works if you think in terms of json. The type MPackMap represents a dictionary, and the type MPackArray represents an array.

Create MPack instances with the static method MPack.From(object);. You can pass any simple type (such as string, integer, etc), or any Array composed of a simple type. MPack also has implicit conversions from most of the basic types built in.

Transform an MPack object back into a CLR type with the static method MPack.To<T>(); or MPack.To(type);. MPack also has explicit converions going back to most basic types, you can do string str = (string)mpack; for instance.

MPack now supports native asynchrounous reading and cancellation tokens. It will not block a thread to wait on a stream.

NuGet

MPack is available as a NuGet package!

PM> Install-Package MPack

Usage

Create a object model that can be represented as MsgPack. Here we are creating a dictionary, but really it can be anything:

Turn MPack objects back into types that we understand with the generic To<>() method. Since we know the types of everything here we can just call To<bool>() to reconstruct our bool, but if you don't know you can access the instance enum MPack.ValueType to know what kind of value it is:

### Flags
<p>Currently there are three flags which you may use to initialize a MsgPack object:</p>
* <code>MsgPackFlags.READ_STRING_AS_BYTE_ARRAY</code>: message pack string data is read as byte array instead of string;
* <code>MsgPackFlags.ACCEPT_LITTLE_ENDIAN</code>: MsgPack objects will work with little endian buffers (message pack specification defines big endian as default).
* <code>MsgPackFlags.SPEC2013_COMPATIBILITY</code>: MsgPack will run in backwards compatibility mode.
```actionscript
var msg:MsgPack;
// use logical operator OR to set the flags.
msgpack = new MsgPack(MsgPackFlags.READ_STRING_AS_BYTE_ARRAY | MsgPackFlags.ACCEPT_LITTLE_ENDIAN);

Advanced Usage

Extensions

You can create your own Extension Workers by extending the ExtensionWorker Class and then assigning it to the MsgPack Factory.

The following example assigns a custom worker which extends the ExtensionWorker Class.

```actionscript
var msgpack:MsgPack = new MsgPack();

// Assign the new worker to the factory.
msgpack.factory.assign(new CustomWorker());

<p>For more information regarding Extensions refer to the MessagePack specification.</p>
### Priorities
<p>Worker priority behaves similar to how the Adobe Event Dispatcher priorities work. In MessagePack, deciding which worker will be use for serializing/deserializing depends on two(2) factors.</p>
1. The order in which the worker was assigned to the factory.
2. The priority of the worker. Higher values take precedence.
All workers have a default priority of 0.
<p>In the following example <code>workerB</code> will never be used because it's assign after <code>workerA</code></p>
```actionscript
var msgpack:MsgPack = new MsgPack();
var workerA:StringWorker = new StringWorker();
var workerB:DifferentStringWorker = new DifferentStringWorker();
msgpack.factory.assign(workerA);
msgpack.factory.assign(workerB);

However if we adjust the priority of workerB, then workerA will never be used.

## Credits
This application uses Open Source components. You can find the source code of their open source projects along with license information below. We acknowledge and are grateful to these developers for their contributions to open source.
Project: as3-msgpack https://github.com/loteixeira/as3-msgpack
Copyright (C) 2013 Lucas Teixeira
License (Apache V2.0) http://www.apache.org/licenses/LICENSE-2.0

RcppMsgPack

Convert to and from msgpack objects in R using the official msgpack-c API through Rcpp.

A flowchart describing the conversion of R objects into msgpack objects and back.

Msgpack EXT types are converted to raw vectors with EXT attributes containing the extension type. The extension type must be an integer from 0 to 127.

Maps are converted to data.frames with additional class "map". Map objects in R contain key and value list columns and can be simplified to named lists or named vectors. The helper function msgpack_map creates map objects that can be serialized into msgpack.

DESCRIPTION

The present module proposes an implemetation of the MessagePack specification as described on http://msgpack.org/. The implementation is now in Pure Perl which could come as a performance penalty opposed to some other packer implemented in C.

WHY THAT MODULE

There are already some part of MessagePack implemented in Perl6, with for instance MessagePack available here: https://github.com/uasi/messagepack-pm6, however that module only implements the unpacking part of the specification. Futhermore, that module uses the unpack functionality which is tagged as experimental as of today

FUNCTIONS

function pack

That function takes a data structure as parameter, and returns a Blob with the packed version of the data structure.

function unpack

That function takes a MessagePack packed message as parameter, and returns the deserialized data structure.

Contributing

MessagePack

MessagePack is an efficient binary serialization format. It lets you exchange data among multiple languages like JSON. But it's faster and smaller. Small integers are encoded into a single byte, and typical short strings require only one extra byte in addition to the strings themselves.

CWPack

CWPack is a lightweight and yet complete implementation of the
MessagePack serialization format
version 5.

Excellent Performance

Together with MPack, CWPack is the fastest open-source messagepack implementation. Both totally outperform
CMP and msgpack-c

Design

CWPack does no memory allocations and no file handling. All that is done
outside of CWPack.

CWPack is working against memory buffers. User defined handlers are called when buffers are
filled up (packing) or needs refill (unpack).

Containers (arrays, maps) are read/written in parts, first the item containing the size and
then the contained items one by one. Exception to this is the cw_skip_items function which
skip whole containers.

Backward compatibility

CWPack may be run in compatibility mode. It affects only packing; EXT is considered illegal, BIN are transformed to STR and generation of STR8 is supressed.

Error handling

When an error is detected in a context, the context is stopped and all future calls to that context are immediatly returned without any actions.

CWPack does not check for illegal values (e.g. in STR for illegal unicode characters).

Build

CWPack consists of a single src file and two header files. It is written
in strict ansi C and the files are together ~ 1.2K lines. No separate build is neccesary, just include the
files in your own build.

CWPack has no dependencies to other libraries.

Test

Included in the test folder are a module test and a performance test and shell scripts to run them.

MessagePack for C# (.NET, .NET Core, Unity, Xamarin)

The extremely fast MessagePack serializer for C#. It is 10x faster than MsgPack-Cli and outperforms other C# serializers. MessagePack for C# also ships with built-in support for LZ4 compression - an extremely fast compression algorithm. Performance is important, particularly in applications like game development, distributed computing, microservice architecture, and caching.

You can add custom type support and has some official/third-party extension package. for ImmutableCollections(ImmutableList<>, etc), for ReactiveProperty and for Unity(Vector3, Quaternion, etc...), for F#(Record, FsList, Discriminated Unions, etc...). Please see extensions section.

MessagePack.Nil is built-in null/void/unit representation type of MessagePack for C#.

Object Serialization

MessagePack for C# can serialze your own public Class or Struct. Serialization target must marks [MessagePackObject] and [Key]. Key type can choose int or string. If key type is int, serialized format is used array. If key type is string, serialized format is used map. If you define [MessagePackObject(keyAsPropertyName: true)], does not require KeyAttribute.

All patterns serialization target are public instance member(field or property). If you want to avoid serialization target, you can add [IgnoreMember] to target member.

target class must be public, does not allows private, internal class.

Which should uses int key or string key? I recommend use int key because faster and compact than string key. But string key has key name information, it is useful for debugging.

MessagePackSerializer requests target must put attribute is for robustness. If class is grown, you need to be conscious of versioning. MessagePackSerializer uses default value if key does not exists. If uses int key, should be start from 0 and should be sequential. If unnecessary properties come out, please make a missing number. Reuse is bad. Also, if Int Key's jump number is too large, it affects binary size.

I don't need type, I want to use like BinaryFormatter! You can use as typeless resolver and helpers. Please see Typeless section.

Resolver is key customize point of MessagePack for C#. Details, please see extension point.

DataContract compatibility

You can use [DataContract] instead of [MessagePackObject]. If type is marked DataContract, you can use [DataMember] instead of [Key] and [IgnoreDataMember] instead of [IgnoreMember].

[DataMember(Order = int)] is same as [Key(int)], [DataMember(Name = string)] is same as [Key(string)]. If use [DataMember], same as [Key(nameof(propertyname)].

Using DataContract makes it a shared class library and you do not have to refer to MessagePack for C#. However, it is not included in analysis by Analyzer or code generation by mpc.exe. Also, functions like UnionAttribute, MessagePackFormatterAttribute, SerializationConstructorAttribute etc can not be used. For this reason, I recommend that you use the MessagePack for C# attribute basically.

Union

MessagePack for C# supports serialize interface. It is like XmlInclude or ProtoInclude. MessagePack for C# there called Union. UnionAttribute can only attach to interface or abstract class. It requires discriminated integer key and sub-type.

MessagePackSerializer.Typeless is shortcut of Serialize/Deserialize<object>(TypelessContractlessStandardResolver.Instance). If you want to configure default typeless resolver, you can set by MessagePackSerializer.Typeless.RegisterDefaultResolver.

TypelessFormatter can use standalone and combinate with existing resolvers.

If type name was changed, can not deserialize. If you need to typename fallback, you can use `TypelessFormatter.BindToType`.
```csharp
MessagePack.Formatters.TypelessFormatter.BindToType = typeName =>
{
if (typeName.StartsWith("SomeNamespace"))
{
typeName = typeName.Replace("SomeNamespace", "AnotherNamespace");
}
return Type.GetType(typeName, false);
};

For string key encode, pre-generated member name bytes and use fixed sized binary copy in IL, see: UnsafeMemory.cs

Before creating this library, I implemented a fast fast serializer with ZeroFormatter#Performance. And this is a further evolved implementation. MessagePack for C# is always fast, optimized for all types(primitive, small struct, large object, any collections).

Deserialize Performance per options

Performance varies depending on options. This is a micro benchamark with BenchmarkDotNet. Target object has 9 members(MyProperty1 ~ MyProperty9), value are zero.

Method

Mean

Error

Scaled

Gen 0

Allocated

IntKey

72.67 ns

NA

1.00

0.0132

56 B

StringKey

217.95 ns

NA

3.00

0.0131

56 B

Typeless_IntKey

176.71 ns

NA

2.43

0.0131

56 B

Typeless_StringKey

378.64 ns

NA

5.21

0.0129

56 B

MsgPackCliMap

1,355.26 ns

NA

18.65

0.1431

608 B

MsgPackCliArray

455.28 ns

NA

6.26

0.0415

176 B

ProtobufNet

265.85 ns

NA

3.66

0.0319

136 B

Hyperion

366.47 ns

NA

5.04

0.0949

400 B

JsonNetString

2,783.39 ns

NA

38.30

0.6790

2864 B

JsonNetStreamReader

3,297.90 ns

NA

45.38

1.4267

6000 B

JilString

553.65 ns

NA

7.62

0.0362

152 B

JilStreamReader

1,408.46 ns

NA

19.38

0.8450

3552 B

IntKey, StringKey, Typeless_IntKey, Typeless_StringKey are MessagePack for C# options. All MessagePack for C# options achive zero memory allocation on deserialization process. JsonNetString/JilString is deserialized from string. JsonNetStreamReader/JilStreamReader is deserialized from UTF8 byte[] with StreamReader. Deserialization is normally read from Stream. Thus, it will be restored from byte[](or Stream) instead of string.

String key is often useful, contractless, simple replacement of JSON, interoperability with other languages, and more certain versioning. MessagePack for C# is also optimized for String Key. First of all, it do not decode UTF8 byte[] to String for matching with the member name, it will look up the byte[] as it is(avoid decode cost and extra allocation).

And It will try to match each long type (per 8 character, if it is not enough, pad with 0) using automata and inline it when IL code generating.

This also avoids calculating the hash code of byte[], and the comparison can be made several times on a long unit.

If the number of nodes is large, search with a embedded binary search.

Extra note, this is serialize benchmark result.

Method

Mean

Error

Scaled

Gen 0

Allocated

IntKey

84.11 ns

NA

1.00

0.0094

40 B

StringKey

126.75 ns

NA

1.51

0.0341

144 B

Typeless_IntKey

183.31 ns

NA

2.18

0.0265

112 B

Typeless_StringKey

193.95 ns

NA

2.31

0.0513

216 B

MsgPackCliMap

967.68 ns

NA

11.51

0.1297

552 B

MsgPackCliArray

284.20 ns

NA

3.38

0.1006

424 B

ProtobufNet

176.43 ns

NA

2.10

0.0665

280 B

Hyperion

280.14 ns

NA

3.33

0.1674

704 B

ZeroFormatter

149.95 ns

NA

1.78

0.1009

424 B

JsonNetString

1,432.55 ns

NA

17.03

0.4616

1944 B

JsonNetStreamWriter

1,775.72 ns

NA

21.11

1.5526

6522 B

JilString

547.51 ns

NA

6.51

0.3481

1464 B

JilStreamWriter

778.78 ns

NA

9.26

1.4448

6066 B

Of course, IntKey is fastest but StringKey also good.

LZ4 Compression

MessagePack is a fast and compact format but it is not compression. LZ4 is extremely fast compression algorithm, with MessagePack for C# can achive extremely fast perfrormance and extremely compact binary size!

MessagePack for C# has built-in LZ4 support. You can use LZ4MessagePackSerializer instead of MessagePackSerializer. Builtin support is special, I've created serialize-compression pipeline and special tuned for the pipeline so share the working memory, don't allocate, don't resize until finished.

built-in LZ4 support uses primitive LZ4 API. The LZ4 API is more efficient if you know the size of original source length. Therefore, size is written on the top.

Compression speed is not always fast. Depending on the target binary, it may be short or longer. However, even at worst, it is about twice, but it is still often faster than other uncompressed serializers.

If target binary size under 64 bytes, LZ4MessagePackSerializer does not compress to optimize small size serialization.

Compare with protobuf, JSON, ZeroFormatter

protbuf-net is major, most used binary-format library on .NET. I love protobuf-net and respect that great work. But if uses protobuf-net for general-purpose serialization format, you may encounts annoying issue.

protobuf(-net) can not handle null and empty collection correctly. Because protobuf has no null representation( this is the protobuf-net authors answer).

MessagePack specification can completely serialize C# type system. This is the reason to recommend MessagePack over protobuf.

Protocol Buffers has good IDL and gRPC, that is a much good point than MessagePack. If you want to use IDL, I recommend Google.Protobuf than MessagePack.

JSON is good general-purpose format. It is perfect, simple and enough spec. Utf8Json which created me that adopts same architecture as MessagePack for C# and avoid encoding/decoing cost so work like binary. If you want to know about binary vs text, see Utf8Json/which serializer should be used section.

ZeroFormatter is similar as FlatBuffers but specialized to C#. It is special. Deserialization is infinitely fast but instead the binary size is large. And ZeroFormatter's caching algorithm requires additional memory.

Again, ZeroFormatter is special. When situation matches with ZeroFormatter, it demonstrates power of format. But for many common uses, MessagePack for C# would be better.

Extensions

MessagePack for C# has extension point and you can add external type's serialization support. There are official extension support.

MessagePack for C# operates at the byte[] level, so byte[] API is faster than Stream API. If byte [] can be used for I/O, I recommend using the byte [] API.

Deserialize<T>(Stream) has bool readStrict overload. It means read byte[] from stream strictly size. The default is false, it reads all stream data, it is faster than readStrict but if the data is contiguous, you can use readStrict = true.

High-Level API uses memory pool internaly to avoid unnecessary memory allocation. If result size is under 64K, allocates GC memory only for the return bytes.

LZ4MessagePackSerializer has same api with MessagePackSerializer and DefaultResolver is shared. LZ4MessagePackSerializer has additional SerializeToBlock method.

All api works on byte[] level, no use Stream, no use Writer/Reader so improve performance. Many builtin formatters exists under MessagePack.Formatters. You can get sub type serializer by formatterResolver.GetFormatter<T>. Here is sample of write own formatter.

DateTime is serialized to MessagePack Timestamp format, it serialize/deserialize UTC and loses Kind info. If you useNativeDateTimeResolver serialized native DateTime binary format and it can keep Kind info but cannot communicate other platforms.

If you want to make your extension package, you need to make formatter and resolver. IMessagePackFormatter accepts IFormatterResolver on every request of serialize/deserialize. You can get child-type serialize on resolver.GetFormatterWithVerify<T>.

MessagePackFormatterAttribute

MessagePackFormatterAttribute is lightweight extension point of class, struct, interface, enum and property/field. This is like JSON.NET's JsonConverterAttribute. For example, serialize private field, serialize x10 formatter.

Reserved Extension Types

MessagePack for C# already used some messagepack ext type codes, be careful to use same ext code.

Code

Type

Use by

-1

DateTime

msgpack-spec reserved for timestamp

30

Vector2[]

for Unity, UnsafeBlitFormatter

31

Vector3[]

for Unity, UnsafeBlitFormatter

32

Vector4[]

for Unity, UnsafeBlitFormatter

33

Quaternion[]

for Unity, UnsafeBlitFormatter

34

Color[]

for Unity, UnsafeBlitFormatter

35

Bounds[]

for Unity, UnsafeBlitFormatter

36

Rect[]

for Unity, UnsafeBlitFormatter

37

Int[]

for Unity, UnsafeBlitFormatter

38

Float[]

for Unity, UnsafeBlitFormatter

39

Double[]

for Unity, UnsafeBlitFormatter

99

All

LZ4MessagePackSerializer

100

object

TypelessFormatter

for Unity

You can install by package and includes source code. If build target as PC, you can use as is but if build target uses IL2CPP, you can not use Dynamic***Resolver so use pre-code generation. Please see pre-code generation section.

MessagePack.UnityShims NuGet package is for .NET ServerSide serialization support to communicate with Unity. It includes shim of Vector3 etc and Safe/Unsafe serialization extension.

If you want to share class between Unity and Server, you can use SharedProject or Reference as Link or new MSBuild(VS2017)'s wildcard reference etc. Anyway you need to source-code level share. This is sample project structure of use SharedProject.

SharedProject(source code sharing)

Source codes of server-client shared

ServerProject(.NET 4.6/.NET Core/.NET Standard)

[SharedProject]

[MessagePack]

[MessagePack.UnityShims]

ClientDllProject(.NET 3.5)

[SharedProject]

[MessagePack](not dll, use MessagePack.unitypackage's sourcecodes)

Unity

[Builded ClientDll]

Other ways, use plain POCO by DataContract/DataMember can use.

Pre Code Generation(Unity/Xamarin Supports)

MessagePack for C# generates object formatter dynamically by ILGenerator. It is fast and transparently generated at run time. But it needs generate cost at first time and it does not work on AOT environment(Xamarin, Unity IL2CPP, etc.).

Note: If Unity's build target as PC, does not need code generation. It works well.

If you want to avoid generate cost or run on Xamarin or Unity, you need pre-code generation. mpc.exe(MessagePackCompiler) is code generator of MessagePack for C#. mpc can download from releases page, mpc.zip. mpc is using Roslyn so analyze source code.

In default, mpc.exe generates resolver to MessagePack.Resolvers.GeneratedResolver and formatters generates to MessagePack.Formatters.***. And application launch, you need to set Resolver at first.

// CompositeResolver is singleton helper for use custom resolver.// Ofcourse you can also make custom resolver.MessagePack.Resolvers.CompositeResolver.RegisterAndSetAsDefault(
// use generated resolver first, and combine many other generated/custom resolvers
MessagePack.Resolvers.GeneratedResolver.Instance,
// finally, use builtin/primitive resolver(don't use StandardResolver, it includes dynamic generation)
MessagePack.Resolvers.BuiltinResolver.Instance,
AttributeFormatterResolver.Instance,
MessagePack.Resolvers.PrimitiveObjectResolver.Instance
);

Note: mpc.exe is basically run on only Windows. But you can run on Mono, that supports Mac and Linux.

RPC

MessagePack advocated MessagePack RPC, but formulation is stopped and it is not widely used. I've created gRPC based MessagePack HTTP/2 RPC streaming framework called MagicOnion. gRPC usually communicates with Protocol Buffers using IDL. But MagicOnion uses MessagePack for C# and does not needs IDL. If communicates C# to C#, schemaless(C# classes as schema) is better than IDL.

How to Build

Open MessagePack.sln on Visual Studio 2017.

Unity Project is using symbolic link. At first, run make_unity_symlink.bat so linked under Unity project. You can open src\MessagePack.UnityClient on Unity Editor.

Author Info

Yoshifumi Kawai(a.k.a. neuecc) is a software developer in Japan.
He is the Director/CTO at Grani, Inc.
Grani is a mobile game developer company in Japan and well known for using C#.
He is awarding Microsoft MVP for Visual C# since 2011.
He is known as the creator of UniRx(Reactive Extensions for Unity)

msgpack-ll

This is a low-level @nogc, nothrow, @safe, pure and betterC compatible
MessagePack serializer and deserializer. The
library was designed to avoid any external dependencies and handle the low-level protocol
details only. As a result the library doesn't have to do any error handling or
buffer management. This library does never dynamically allocate memory.

A simple example showing the complete API

import msgpack_ll;
// Buffer allocation is not handled by the libraryubyte[128] buffer;
// The MsgpackType enum contains all low-level MessagePack typesenum type = MsgpackType.uint8;
// The DataSize!(MsgpackType) function returns the size of serialized data// for a certain type.// The formatter and parser use ref ubyte[DataSize!type] types. This// forces the compiler to do array length checks at compile time and avoid// any runtime bounds checking.// Format the number 42 as a uint8 type. This will require// DataSize!(MsgpackType.uint8) == 2 bytes storage.
formatType!(type)(42, buffer[0..DataSize!type]);
// To deserialize we have to somehow get the data type at runtime// Then verify the type is as expected.assert(getType(buffer[0]) == type);
// Now deserialize. Here we have to specify the MsgpackType// as a compile time value.const result = parseType!type(buffer[0..DataSize!type]);
assert(result ==42);

The generated code is obviously slighly more complex. The interesting part here
is that type checking is directly done using the raw type value and not the
enum values returned by getType. Even manually written ASM probably can't do
much better here.

msgpack-response

Extension of the current ExpressJS API; Introducing the Response.msgPack(jsObject) method on the standard ExpressJS Response object.

Getting Started

With auto-detection and transformation enabled, the middleware detects automatically the HTTP header Accept: application/x-msgpack and piggybacks the Response.json() method of the ExpressJS API, to encode the JSON response as Message Pack. This method is usefull, when you have existing applications that need use the middleware, without changing the codebase very much.

Note: Remember the add the header Accept: application/x-msgpack in the request.

Also it can have auto detection and transformation disabled. The middleware extends the Response object of the ExpressJS framework, by adding the msgPack() method to it. Then to return an encoded response, you just use the Response.msgPack() method that accepts the Javascript object as parameter. For example,

msgpack for R

msgpack is a binary data format with compact encoding and data
structure support similar to JSON. It can be a drop-in replacementfor
JSON in most applications. It is designed to be fast to parse and
compact to transmit and store.

Documentation

TODO

Sponsors

License

Copyright (c) 2017 Patrik Simek

The MIT License

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

The pack function takes a Racket object and a binary output port as arguments
and writes the serialised data to the port. The unpack function takes a
binary input port and returns one de-serialised object, consuming the necessary
amount of bytes from the port in the process. For more details please refer to
the documentation.

In the above example code we set the output and input ports to be byte strings
so we could work with the packed and unpacked data directly inside the Racket
instance.

Status

The library is fully functional, covered by test cases, and the API should be
reasonably mature, but I am not yet willing to completely rule out changes. See
also below for parts of the library that could not be tested at the moment due
to technical reasons.

Caveats

The following cases cannot be tested for the time being:

The bin32 type, storing a byte string that is 2^32 bytes long
requires 4GiB, my machine simply runs out of memory.

The same goes for the str32 type

The same goes for the array32 type

The same goes for the map32 type

The same goes for the ext32 type

Strings are only tested using ASCII characters, if anyone can generate
UTF-8 strings with a given length in bytes please help out.

License

msgpack2R

Convert to and from msgpack objects in R using the official msgpack-c API through Rcpp.

A flowchart describing the conversion of R objects into msgpack objects and back.

Msgpack EXT types are converted to raw vectors with EXT attributes containing the extension type. The extension type must be an integer from 0 to 127.

Maps are converted to data.frames with additional class "map". Map objects in R contain key and value list columns and can be simplified to named lists or named vectors. The helper function msgpack_map creates map objects that can be serialized into msgpack.

msgpack-objc

An Objective-C wrapper for msgpack-c. Focuses on ease of use and speed.
If you need configurability, there are other, more advanced libraries, for example MPMessagePack.

This library will always try to use sane defaults. If any nil value is encountered in the MessagePack-data, the object will
be omitted instead of returning an [NSNull null]. This means that there can be no nil objects in dictionaries, and object-less
keys will be lost in translation.

Extension support

The library supports MessagePack-timestamps,
and will return an NSDate-object whenever one is encountered. When serializing, any NSDate-objects will also be
serialized as native MessagePack-timestamps.

You can add native serialization for your own classes by subclassing MessagePackExtension.

mood.msgpack

Python MessagePack implementation

MessagePack is an efficient binary serialization
format. It lets you exchange data among multiple languages like JSON. But it's
faster and smaller. Small integers are encoded into a single byte, and typical
short strings require only one extra byte in addition to the strings themselves.

Note: This implementation is not geared towards inter language exchange but
towards serialization/de-serialization of Python object structures (it was
designed as a pickle
substitute). It does not expose MessagePack's extensions mechanism but uses it
internally to pack/unpack non-standard types.
That said, if you only deal with standard objects/types (None, True,
False, integers, floating point numbers, bytes, strings, tuples and
dictionaries) you are fine to use this module to produce or consume data that is
targeted at or originates from other programming languages.
The following documentation is largely adapted from Python's pickle module
documentation.

Packing Class Instances

The interface is currently defined as follows: the __reduce__ method takes
no argument and shall return either a unique string or preferably a tuple.

If a string is returned, the string should be interpreted as the name of a
global variable. This behaviour is typically useful for singletons (and is the
case for built-in functions).

When a tuple is returned, it must be between two and five items long.
Optional items can either be omitted, or None can be provided as their
value. The semantics of each item are in order:

A callable object that will be invoked to create the initial version of the
object (it must be registered in order to be unpacked).

A tuple of arguments for the callable object. An empty tuple must be given
if the callable does not accept any argument.

Optionally, the object's state, which will be passed to the object's
__setstate__ method. If the object has no such method then, the value
must be a dictionary and it will be added to the object's __dict__
attribute.

Optionally, a sequence/iterator yielding successive items. These items
will be appended to the object using object.extend(items). This is
primarily used for list subclasses, but may be used by other classes as long
as they have an extend method with the appropriate signature. If the
object has no such method then, an in-place concatenation will be attempted
(equivalent to object += items).

Optionally, a dict/mapping or a sequence/iterator yielding successive
key-value pairs. These pairs will be stored in the object using
object.update(pairs). This is primarily used for dictionary subclasses,
but may be used by other classes as long as they have an update method
with the appropriate signature. If the object has no such method then, an
attempt will be made to store these pairs using object[key] = value.

Field names can be set in much the same way as with the encoding/json package. For example:

typePersonstruct {
Namestring`msg:"name"`Addressstring`msg:"address"`Ageint`msg:"age"`Hiddenstring`msg:"-"`// this field is ignored
unexported bool// this field is also ignored
}

By default, the code generator will satisfy msgp.Sizer, msgp.Encoder, msgp.Decoder, msgp.Marshaler,
and msgp.Unmarshaler. Carefully-designed applications can use these methods to do marshalling/unmarshalling
with zero heap allocations.

While msgp.Marshaler and msgp.Unmarshaler are quite similar to the standard library'sjson.Marshaler
and json.Unmarshaler, msgp.Encoder and msgp.Decoder are useful for stream serialization.
(*msgp.Writer and *msgp.Reader are essentially protocol-aware versions of *bufio.Writer and *bufio.Reader.)

As long as the declarations of MyInt and Data are in the same file as Struct, the parser will determine that the type information for MyInt and Data can be passed into the definition of Struct before its methods are generated.

Extensions

MessagePack supports defining your own types through "extensions," which are just a tuple of the data "type" (int8) and the raw binary.
You can see a worked example in the wiki.

Status

The code generator here and runtime library are both stable. Newer versions of the code may generate different code than older versions for performance reasons.

You can read more about how msgp maps MessagePack types onto Go types in the wiki.

Here some of the known limitations/restrictions:

Identifiers from outside the processed source file are assumed to satisfy the generator's interfaces. If this isn't the case, your code will fail to compile.

The chan and func fields and types are ignored as well as non-exported fields.

Encoding of interface{} is limited to built-ins or types that have explicit encoding methods.

Maps must have string keys. This is intentional (as it preserves JSON interoperability). Although non-string map keys are not forbidden by the MessagePack standard, many serializers impose this restriction. (It also means any well-formed struct can be de-serialized into a map[string]interface{}.) The only exception to this rule is that the deserializers will allow you to read map keys encoded as bin types, since some legacy encodings permitted this. (However, those values will still be cast to Go strings, and they will be converted to str types when re-encoded. It is the responsibility of the user to ensure that map keys are UTF-8 safe in this case.) The same rules hold true for JSON translation.

If the output compiles, then there's a pretty good chance things are fine. (Plus, we generate tests for you.) Please, please, please file an issue if you think the generator is writing broken code.

As one might expect, the generated methods that deal with []byte are faster for small objects, but the io.Reader/Writer methods are generally more memory-efficient (and, at some point, faster) for large (> 2KB) objects.

Credits

This repository is a fork of github.com/tinylib/msgp. The original authors did a great job, but
this repository takes the project in a new and better direction.

Differences between this tool and tinylib/msgp:

Here we have regular expression matching for type names in directives.

Here we do not use package unsafe for conversions from byte slices to strings: []byte is converted efficiently
to string simply with the built-in string().

This codebase is thoroughly refactored to be more Go-idiomatic, efficient, and inviting to contributors.

Additionally, in this library we plan to add an omitempty feature, like what encoding/json has.

swift-msgpack-serialization

Another swift serialization library for msgpack with support for the Codable environement.

This framework is designed to be (optionaly) compatibel to the msgpack-java library.
See Compatibility.md for more detailed information about this.

Please make sure, that the swift model classes and java model classes you use are compatibel to each other, e.g. enums are named equaly (including case).

This framework also supplies the possibiliy to encode keys of other types than String (generalContainer function of MsgpackEncoder).

swift-msgpack-serialization works with MessagePack.swift. This means, that you can encode your classes, structs and enums, that conform to Codable to MessagePackValue from MessagePack.swift, do something with this MessagePackValue and then encode it to msgpack using MessagePack.swift. You may of course also decode to MessagePackValue first and then decode a particular MessagePackValue to your own type.

Instalation

Currently, only carthage is supported.

Compatibilty restrictions

Make sure you never call .encode(to:) directly! Instead use the encode method on a keyed, unkeyed or singleValue container or cast the encoder passed to you to MsgpackEncoder and call encodeIntermediate(_) on this encoder. If you do not use this, this framework won't be able to do it's work properly.

Converting MsgPack to Matlab

Note that since structs don't support arbitrary field names, they can't be used for representing maps. We use containers.Map instead.

Tests

runtests()

License

MATLAB (R) is copyright of the Mathworks

Copyright (c) 2014 Bastian Bechtold
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:

Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.

Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the
distribution.

Neither the name of the copyright holder nor the names of its
contributors may be used to endorse or promote products derived
from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.