Introduction

The advent of .NET Framework is the most significant event in the programming world over the last 10 years and I think it is still underestimated. It is the most solid and comprehensive framework ever. However, nothing and nobody is perfect. This article is about .NET Framework communication and compact framework communication (serialization) in particular.

Communication System Model

What is a communication framework? Generally a communication framework is an object delivery system. No more and no less. If this system implements RPC/RMI, another layer is built on top of that. The object delivery system in turn consists of just two sub layers:

Transport layer (physically delivers the stream of bytes to the destination)

Converter from the byte stream to the object (a serializer) And nothing else.

The rest can be hidden inside this ideal communication framework. The last statement is true only for the systems that describe the objects in the same terms (same OS, same .NET Framework).

Minimal (Optimal) Communication System

Minimal communication system has just one method Send(Object). Does the programmer have to care about how the object is converted to the byte stream, how it is managed by the threads, how it is queued, fragmented, etc.? No, it is the communication framework's job.

WCF

When I first looked at WCF, I had a feeling that I was missing something. It could not be true: automatic generic serialization had gone! Instead it has a semi manual very restrictive serialization. Judge for yourself:

Does everybody like that? Apparently not. The Web is full of blogs suggesting numerous "smart" solutions overcoming the restrictive nature of WCF. The limited WCF serialization is "successfully" worked around by using BinaryFormatter and then putting the byte array as a parameter. Is it the price paid for the interoperability? Microsoft apparently sacrificed versatility for interoperability simply by limiting the functionality. The logical extreme of such an approach would be a return to the good old raw socket communication. Hence most of the WCF programs run on Windows, would it be more sensible having the generic serializer as a default and the rest as an option?

What Makes the Communication Framework OS Specific

The only thing that makes the system OS specific is the formatter (the serializer). It is the only component in the system that converts the stream to the object. The stream is always the same. As the sculpture is made of stone (any stone) so the object is made of stream. In fact, WCF has all these bits and pieces but the way the system is composed is far from perfect.

Interoperability

As I already mentioned, the Serializer makes the framework OS specific. In turn the serializer exists in the framework [say .NET]. What if the OS does not support the frameworks and does not have even the notion of the serializer? In this particular case, manual (or semi manual) processing of the byte stream is appropriate but that should be an option, not a default.

If You Like Everything Automatic, Why Not to Use Remoting?

Standard remoting is based on just synchronous communication model. It is unidirectional. The attempts to use call backs for the duplex communication do not work with the client behind proxy or NAT because Remoting opens a second connection for the events. Since the firewall is a standard network component, we may forget about bidirectional communication using remoting.

Total Recall

.NET standard communication suffers from the same disease that Windows OS suffered 10 years ago. Core Windows API is the same as it was 10 years ago, but the programming of Windows Form application 10 years back was a serious ordeal. Why? Because the programmer had to provide all the parameters manually. In order to provide all the parameters, he needed to know all Windows internals and how to use them. In .NET environment, all the defaults are set by the framework. We drag the controls from the toolbox and the rest is done automatically. If we need extra features, we can select them from the property grid or type them in the code.

In real life, we send the object (the letter or the parcel) to the recipient. Do we really care how the letter is delivered, what is the postman's name or the name of his pet dog? No, we don't. Usually we drop the letter in the postbox and the rest is done by default. If we want some extra features, like better security or delivery confirmation, we can get that as an extra. If the letter delivery looked like .NET communication (WCF in particular), the procedure of the letter delivery would have certainly been like that: we select from the catalog the number plate of the car that carries the letter, the name of the driver, the flight number, the type of the plane, envelope color, brand …. Most probably that would stop us from sending the letter at all.

Alternatives

While high priests of IT industry keep saying that everything is better than ever, the communication frameworks (mostly .NET socket based) keep multiplying. The Code Project website hosts at least 10 of them. The design of a primitive communication class with a limited functionality (often it is enough) is not too hard, but the major showstopper was a serializer, specifically for the Compact Framework.

Compact Framework Remoting

Unfortunately Remoting and Binary formatter is not implemented (and apparently will not be) for the compact framework. The only choice is using limited XMLSerializer (which is not generic) and the implementation of the system where you can simply drop the object on one side and get it on the other is not technically possible. Apart from the serializer, other components like channels or threads are readily available in CF.

Compact Framework Serialization

This work is based on a brilliant Angelo Scotto's CompactFormatter for the full and Compact Framework. The original CompactFormatter was written for .NET 1.1. However the years went by and the opportunities for improvement emerged. Basically the reasons for the redesign are:

Compact Framework 2.0 has serializable attribute. This attribute was introduced only for compatibility with a full framework. Other than that, the practical value of this attribute is questionable because the classes built before this attribute was introduced obviously do not have it. And solely relying only on this attribute will make quite serializable classes non serializable. Original CompactFormatter has a similar attribute but it is not compatible with the full framework. Checking for this attribute (Angelo's serializable attribute) was removed. It made the classes created for the full framework and compact framework compatible. Though it has some drawback – totally non serializable objects will be attempted to serialize.

Serialization of the DataSet and DataTable in CompactFormatter is not full. DataSet serialization designed by SimmoTech is far more advanced. Simotech serialization and the code can be found in The Code Project article. This code was incorporated into CompactFormatter as a surrogate.

For performance reasons, the original CompactFormatter puts the indexes of the classes in the stream instead of the class name. That is good if the serializer exists consistently during the communication session. Practically that is not always the case. The serializer may be instantiated dynamically and CompactFormatter certainly fails.

The way the classes are instantiated when deserialized: CompactFormatter uses incomplete assembly name or fully qualified assembly name. Either method fails if the communication platforms are different. For instance, fully qualified assembly name is meaningless when the object is serialized on full framework and deserialized on the compact. Only the primitive types and the types defined within the serializer will work.

Automatic surrogate generation tool can be used now for the surrogates.

Extreme Performance and Serialization Studio

Original CompactFormatter uses surrogates, overrides and custom serialization for the better performance. This certainly works; however writing the surrogate is always tricky and tedious business even for the experienced programmer. To make this task easier, custom serialization and surrogates were removed.

Instead now there is just one custom serialization – an override.

Override (in Angelo's Scotto terms) is the original mechanism that allows for injecting custom serialization modules for the specific class. Also CompactFormatter has been redesigned for including automatically generated binary serializers. Serialization Studio is the tool that generates binary super fast serializers (10-60 times faster than BinaryFormatter). The tool is free and can be downloaded from here. This tool is specifically tuned to generate the surrogate for CompactFormatterPlus.

If the fast serializer is generated for this class, UnknownObject has to be serialized at runtime because its type is not known at compilation time. If this happens, the generic serializer takes over the control and continues the serialization. In turn, if the CompactFormatterPlus detects the field (class) that can be serialized by the fast serializer, the control returns recursively back to the fast serializer. Actually the programmer does not have to know such intricacies of the serialization. The fast serializer (if any) only has to be added to CompactFormatterPlus. The runtime of the fast serializer consists of FSWriter and FSReader. It has some extra methods against the original Binary reader and writer. It looks a bit ugly because instead of inheriting everything from Binary reader (writer), it has its own implementation of all overloads. The reason for that is: CF compiler apparently has a bug. Overloaded Write (byte) evaluates to [presumably] Write(double). [8 bytes instead of 1]

Using the Code

On the PC side, the classes that have to be serialized should be included in the solution as the source code or the library. In the case of library, it has to be referenced in the PC project. Your class must have the same namespace name and the class name as the namespace and the class name on PDA. All the above are true for PDA. The solutions are separate. PC files should not be used or referenced on the PDA side and vice versa. Common are just the names.

Sometimes it is possible to use CF files (EXE and DLLs) on PC and PC files on PDA. But that is a bad practice. It may work or fail with a weird and irrelevant message. Typical scenario: During the debugging session, if you have PC generated assembly in your mobile solution, this file will drag all referenced assemblies to your PDA from PC and you will run out of memory.

CompactFormatterPlus Serializable Types

All primitive types

Arrays of primitive types

DateTime

ArrayList

Hashtable

List<T>

Dictionary<T Key, T1 Value>

DataSet

DataTable

Complex objects composed of the above types

Credits

The user by the name Fabien Castell has improved the typed dataset serialization and the instantiation of the objects if they are defined in different assemblies. The code can be downloaded from the original website.

Comments and Discussions

I tried using compact formatter to serialise and deserialise DataSet object in Windows CE environment (using .Net CF 3.5 framework). The DataSet is expected to contain considerable amount of data and hence it is serialised and deserialised in several scenarios.

I could manage to serialise the DataSet without any problems, however while deserialising it, half of the time it threw OutOfMemory exception on the mobile handheld device. Can you please help me to understand and resolve this small glitch which is making this entire solution bit unreliable in nature.

Instantiate a hash table... ad numerous objects to it... serialize it to the file system... works like a champ. Deserialize it back into a hash table... works like a champ. Now here's where I get the bug: delete a couple of records from the middle of the hash table and serialize... I am getting an endless recursion bug. But, if I do the delete by creating a new hashtable and copying over only the records from the old ones I didn't want to delete, it serializes beautifully. Might be because the hashtable gets re-indexed by that. So that's a working workaround, but the bossman is leery of putting this code in production because of that bug, and I really have no opther good option for caching functionality in CF 3.5. Has anyone else seen this? Thanks, : ) Steve

I'm developing for the Xbox360 and windows phone platforms that do not have the BinaryFormatter. Does this library function on the 360?If a file is created with the stock BinaryFormatter, can it be read with this library?My structures contain arrays of structure arrays. Is this library able to serialize/deserialize nested structures and arrays?

And finally, as I'm not doing anything with Access Data Objects, I guess I can just stick to the CompactFormatter?

I have a web service in FULL framework that contains a method that receives an array of bytes. That array contains a Hashtable.

I have other smartdevice project that calls that service and passes a Hashtable as an array of bytes.The problem is that in the web service code the contains method of the hashtable return false. It only happens when the client is a smartdevice. If the client is in full framework it doesn't happen and the contains method returns true.

How can I solve this. Maybe some bug in the serialize/deserialize method?

The simplest way to use this serializer with WCF is to serialize your object and send it using WCF as an array (as a parameter).Perhaps there is a better way of doing that, however it has never been investigated.WCF is not the smartest thing to waste the time on it.

If you have more specific questions, please contact us support@dotnetremoting.com

Very, very good article.I can´t understand how Microsoft did not implement binary formatters in the Compact Framework to be used with Windows Communication Foundation. Every developer in the world that works with Windows CE know that this is a MUST HAVE feature, but in Microsoft, this feature is ignored in each version of the compact framework that is released.

When I try to serialize an XmlDocument object from .NET Compact Framework 2.0, it often raises a StackOverflowException.

I think this is due to the great number of fields in this element, each one containing a lot of fields, leading to a stack overflow (which is 128KB on .NETCF 2.0, 64KB on .NETCF 1.x). The other reason is the implementation of CompactFormatterPlus, which is based on recursion within an object to serialize each field. This makes the code easier to implement and to understand but this can cause such a problem.

The only workaround I found is to serialize an XML string rather than the XmlDocument object itself: I get it with the OuterXml property of the XmlDocument object.When I deserialize the stream, I can reconstruct the object with the LoadXml method of XmlDocument class.