Please Note

Introduction

The officially sanctioned way of making distributed function calls between C++ programs is to use CORBA, but for many applications, this is overkill. The CORBA specifications allow distributed function calls to be made between code written in any number of languages, and to make it all work, specialized tools need to be integrated into the build process, in order to translate object definitions written in CORBA's IDL to whichever native language is being used (C++, Java, etc.).

However, if we assume that the server and client are both written in the same language, let us assume C++, since it is possible to do away with these complexities. In particular, instead of elaborate definitions of interfaces and marshalling specifications, we can simply defer to C++.

Instead of separate IDL files with object interfaces, we specify the interfaces directly in C++ source code, using the preprocessor, and to marshal arguments across process boundaries, we use the native C++ serialization framework provided in the latest release of the Boost library.

The Boost.Serialization library is used to serialize parameters and return values. It handles standard types and containers automatically, and is easily extended to user defined classes. It also allows us to serialize pointers, with proper handling of polymorphic pointers and multiple pointers to single objects.

Basic Usage

There are three basic steps to using this framework:

Use the RCF_xxx macros to define interfaces.

Use the RcfServer class to expose objects that implement the interface.

Use the RcfClient<> classes to invoke methods on the objects exposed by the server.

type is the identifier for the interface, type_id is a string giving a runtime description of the interface. The RCF_METHOD_xx macros define the member functions, and are named according to the number of arguments and whether the return value is void or not. So, for a function func accepting two strings and returning an integer, we write:

RCF_METHOD_R2(int, func, std::string, std::string);

and if the function has a void return type, we would instead write:

RCF_METHOD_V2(void, func, std::string, std::string);

Dispatch IDs for each function are generated automatically; the first member function is numbered 0, the next one 1, and so on. So, the order in which the functions appear in the definition is important, unlike in CORBA, where dispatch IDs are based on the function name. The dispatch IDs are generated using templates and not any preprocessor __LINE__ trickery, so the interface does not change if blank lines are inserted. The maximum number of member functions that can appear between RCF_BEGIN() and RCF_END() is at the moment limited to 25, but this limit is arbitrary.

The purpose of the RCF_xxx macros is to define the class RcfClient<type>. This class serves as a client stub, from the user's point of view, but also has facilities that allow the framework to use it as a server stub. These macros can be used in any namespace, not just the global namespace.

Once we have defined an interface using the RCF_xxx macros, we can start a server and bind the interface to concrete objects:

{
// create the server and tell it which port to listen on
RCF::RcfServer server(port);
// Interface is the identifer of the interface we're exporting,// Object is a type that implements that interface// one object for each client
server.bind<Interface, Object>();
// ... or one object shared by all clientsObject object;
server.bind<Interface>(object);
// tell the server to start listening for connections
server.start();
// ...// the server will shut down automatically as it goes out of scope
}

The objects are statically bound to the corresponding interface; there is no need for the object to derive from an interface class as is the case for traditional dynamic polymorphism. Instead, the compiler resolves the interface at compile time, which is not only more efficient, but also allows more flexible semantics.

The server can handle multiple simultaneous clients, even in single threaded mode, and can be stopped at any time. The lifetime of objects exposed by the server is determined by the number of current connections to the given object; once there are no more live connections to the object, a timeout is set, and when it expires, the object is deleted.

To make a client call, we instantiate the corresponding RcfClient<> template and pass the server IP and port number to the constructor. When the first remote method is called, the client then attempts to connect to the server, queries for the given object, invokes the requested member function of the remote object, and then returns the remote return value.

Should any exceptions arise on the server side while invoking the requested object, an exception of type RCF::RemoteException will be propagated back to the client and thrown. Should any exceptions arise anywhere else on the server side, e.g., while serializing arguments, then the server will forcibly close the connection, and the client will throw an exception.

RCF will automatically handle a range of parameter types, including C++ primitive types (int, double, etc.), std::string, STL containers, and pointers and references to any of the previously mentioned types. Polymorphic pointers and references, and multiple pointers to single objects are correctly handled as well. Smart pointers are also supported (boost::shared_ptr, std::auto_ptr), and are the safest way of passing polymorphic parameters.

In CORBA, one can tag a parameter as in, out, or inout, depending on which direction(s) one wants the parameter to be marshaled. In RCF, the marshaling directions are deduced from the parameter type, according to the following conventions:

Value: in
Pointer: in
Const reference: in
Nonconst reference: inout
Nonconst reference to pointer: out

To use user-defined types as parameters or return values, some additional serialization code is needed. What that code is depends on which serialization protocols are being used; by default Boost.Serialization is used, and an example of passing a user-defined type would look like the following:

Details

The server and client classes use BSD-style sockets to implement the networking, over TCP, and the whole framework has been compiled and tested on Linux, Solaris (x86 and SPARC) and Win32, using Visual C++ 7.1, Codewarrior 9.0, Borland C++ 5.5, and GCC 3.2. Building RCF requires v. 1.32.0 or later of the Boost library, although the only parts of Boost that need to be built are Boost.Serialization, and, for multithreaded builds, Boost.Threads. Multithreaded builds are enabled by defining RCF_USE_BOOST_THREADS before including any RCF headers.

To use RCF in your own application, you'll need to include the src/RCF.cpp file among the sources of the application, and link to the necessary libraries from Boost, along with OS-specific socket libraries (on Windows that would be ws2_32.lib, on Linux libnsl, etc.).

I've included a demo project for Visual Studio .NET 2003, which includes everything needed to compile, link, and run a server/client pair, with the exception of the Boost library, which needs to be downloaded and unzipped, but no building is needed.

Performance, as measured in requests/second, is highly dependent on the serialization protocol, and also on the compiler being used. Before turning to Boost.Serialization, I used a serialization framework of my own, with which I could clock around 3000 minimal requests/sec. using Visual C++ 7.1, and 3300 requests/sec. with Codewarrior 9.0, on a loopback connection on a 1400Mhz, 384Mb PC running Windows XP. GCC 3.2, on the other hand, was far slower. Using Boost.Serialization, however, I've been nowhere near these numbers; on average, it's around five times slower.

Conclusion

RMI is a well known concept in Java circles, what I've done here is to do something similar in C++, without all the complications of CORBA. If you like it, please tell me, if you don't, well, please tell someone else.... Jokes aside, any and all feedback is appreciated, all I ask is that if you grade the article, and do so with a low grade, then please leave an explanatory comment!

History

8 Feb 2005 - First release.

10 Mar 2005

Now includes a custom serialization framework, so you no longer have to use Boost's. Both serialization frameworks are supported though, use the project-wide RCF_NO_BOOST_SERIALIZATION and RCF_NO_SF_SERIALIZATION defines to control which ones are used. Default behaviour is to compile both.

Default client timeout changed to 10s.

Server can be configured to only accept clients from certain IP numbers.

Server can be configured to listen only on a specific network interface, such as 127.0.0.1.

Client stubs automatically reset their connections when exceptions are thrown (eg for timeouts).

Finer-grained exception classes.

11 July 2005

Stripped CVS folders from distribution.

Added user-definable callback functions to be called when RcfServer has started.

16 Aug 2005

Added facilities for server-bound objects to query the IP address of the client that is currently invoking them. To see how it works, open the file RCF/test/Test_ClientInfo.cpp in the download. Just place a call to RCF::getCurrentSessionInfo().getClientInfo().getAddress(), and you'll receive a string containing the IP address of the client that is invoking the method.

23 Sep 2005

Initialization and deinitialization of the framework can now be done explicitly, be defining the project-wide preprocessor symbol RCF_NO_AUTO_INIT_DEINIT, and then calling RCF::init() and RCF::deinit() at appropriate times. This is mainly useful for DLL builds, so that the DLL can be loaded without automatically initializing Winsock.

19 Oct 2005

Compatible with Boost 1.33.0.

Added enum serialization to the built-in serialization engine, through the SF_SERIALIZE_ENUM macro. For an example of its use, see test/Test_Serialization.cpp.

Added a license.

30 Jan 2006

Miscellaneous bugfixes.

The built-in maximum message size limit has been changed to 50 Kb. Look in src/RCF/Connection.cpp, line 374, if you need to change this.

I'll only be making sporadic maintenance releases of this version of RCF from now on. You can find the next generation of RCF here.

License

Share

About the Author

Software developer, ex-resident of Sweden and now living in Canberra, Australia, working on distributed C++ applications. Jarl enjoys programming, but prefers skiing and playing table tennis. He derives immense satisfaction from referring to himself in third person.

Comments and Discussions

AH! Now that is a good reason... Security is always an issue when it comes to this
sort of things.

Well for my part, I am certain we are the only ones who talk to that server.
it will be running on private LANs with no external access whatsoever. I guess
I will just remove that limitation until your next release.

Hello,
I am happy to see that RCF is growing in feature and usability.

It is possible to see RCF integrated in boost in the next?
I hope so.

I am testing some routines and I think that RCF will be the core of message and command dispatching on a robotic platform that I am working on ..

So, the question.
I need to send unsigned char (bytes) over a channel, but I see that unsigned char is not natively serialized in RCF/boost ...
As the informations are basically byte arrays, I think that there is already a method to deal with such structures. But at the moment the reading of the tests is quite diffficukt to me.
Have you got some tip on hoe to solve that?

Glad to hear you're finding RCF useful! Sounds like an interesting project. As for the unsigned char's, they're already supported, at least in RCF, although I'm pretty sure they're supported in boost.serialization also. If you want to send an array of unsigned char's, the easiest way is to use std::vector, eg

Ok, now I figure out!
The secret is to wrap up standard types into STL containers.
At now RCF is well suited for commanding a remote device (in a slave manner). I am trying to figure out hoe to implement a publish/subscribe policy with RCF ...

Publish/subscribe over stream-oriented transports like TCP will be really easy in the next version of RCF, I already have it implemented and ready to go. So if you can wait until some time before the end of the year when I release it, you'll have the work done for you...

You can still implement it yourself, externally to RCF, but it won't be as efficient, mainly because the publisher will have to spawn its own connection publisher->subscriber, instead of just turning around the connection, subscriber->publisher, that already exists. So the subscriber will have to pass its own ip and port to the publisher, which really should not be necessary.

This helped.
The builtin size limit is what caused the program to fail.
In both cases.
However, I can't understand why the exception thrown in Connection.cpp is not catched ...
Instead a timeout was shown ...

I finally have a few numbers to report that compare
Boost.serialization and Ebenezer Enterprises
approaches. I built two parallel executables using
gcc 4.0.2 that send/serialize a std::list that has
500 ints. In non-optimized executables I found
that our approach took around .23 times as long as
the Boost version. Adding -O3 to the building,
resulted in our approach taking .37 times as long as
the Boost version. (Do you remember if the
performance tests you did used optimization?) Also,
with the optimized build, the Boost executable was 5.8
times larger than the Ebenezer Enterprises executable.
Here is a table that summarizes things.

It would be interesting to see how you've set the tests up, do you think you could email me the source code?

I'm assuming you're running RCF with boost.serialization. Did you try RCF without boost.serializaton, ie using the serialization engine that is built into RCF?

Also, it might be a good idea to compile your test programs with Visual C++ 7.1, or later, and see what happens to your figures. My own experience with gcc, at least versions 3.x, is that the optimizer is terrible; on heavily templated code, like that of boost.serialization or even RCF for that matter, gcc's executables are both an order of magnitude larger and slower than those produced by compilers like Visual C++, or Metrowerks Codewarrior.

Anyway, thanks for posting your results! I'll be happy to give you any feedback on your test harness.

David Abrahams posted some test results that I sent
him on the Boost developers' list. I wanted to
clarify something here in case anyone reads this and
that and finds the results to be conflicting. The
results I posted here were not apples to apples in
my opinion. I forgot that the Ebenezer Send
functions flush the buffer at the end of the
functions. Based on code analysis and runtime
checks this Boost statement

oArch & lst;

doesn't flush anything. So the results posted on
the Boost list are without the Ebenezer flush and I
think are more accurate. I agree with you about
wanting to test on other compilers but gcc is all I
have at the moment.

Greetings! Your library is very good, but I want to make some changes. I want to abstract out of transport methods for I could write my own transport methods. For example I want to use Naped Pipes. Could you tell me, what classes should I recode on my own without breaking down the library infrastructure ??

The class Connection, defined in include/RCF/Connection.hpp, is probably a good place to start making changes. If you want to understand how it functions, try stepping through a few remote calls in the debugger, on both server and client sides.

I'm not too far (a month or two) from releasing a major update of RCF, with among other things pluggable transports, with TCP and UDP transports to begin with. If you can wait for that, you'll have a considerably easier time writing your own transports, I think.

Sorry for taking so long, I just realized I never answered your questions. In any case, of those interfaces, only the first interface, I_ObjectFactory, is in use, and it's the interface that clients use to create objects on the server. You can remove the other interfaces if you want. As for the members of RCF::ClientStub, they are:

Token token; // identifies a specific object on the server
int fd; // file descriptor for a socket
bool oneway; // flag to indicate if client should proceed without waiting for a response from the server
bool service; // temporary hack to indicate that a client wants to access a object of a given name, instead of using a token to look up an an object
std::string endpointName; // not in use
std::string objectName; // usually this is the runtime name of the interface that is being used

First of all, this is good topic. I agree that RPC, CORBA or RMI are all too complicated, but .Net has provided a reasonally simple remoting solution for developers. If we are writing new code on .Net, I don't see why we should build a custom RMI.

With regard to reusing old unmanaged code, it would be a different story. It is worth exploring how we can easily distribute legacy unmanaged objects. There are a number of options. First, COM appears to be #1 choice recommended by many developers. Second, some like to write their own remoting based on sockets. I think your try falls into this category. But they both require significant rework on the old code. Third, .Net has provided another alternative, Interop, which means in our case mixing unmanaged objects with managed remoting. One problem is that the managed remoting cannot marshall unmanaged objects. You may have to write a managed wrapper class for each unmanaged class. This is going to be ugly and eventually similar to the COM approach.

If we are writing new code on .Net, I don't see why we should build a custom RMI.

Well, for one, you might want to make and receive remote calls to and from non-Windows platforms. With .NET remoting you're locking yourself in, right from the start. That's a pretty serious disadvantage; what are you gaining to offset that?

I find it hard to take .NET remoting seriously, as long as Microsoft is pushing attribute-based specifications of remote calls. That attitude may be ok for toy projects, but when things start to get more complex, you really need a proper interface declaration, a la CORBA, or like what I've tried to do with RCF.

Its basic knowledge in software development that one writes interfaces before implementations. I guess you have to be Micoosoft to be able to flaunt basic principles like that...

Jun Du wrote:

First, COM appears to be #1 choice recommended by many developers

Do you have some facts for this, or is it your own opinion? There's a whole world of programming outside Win32, and I can guarantee you that COM is not in high demand there I personally think you have to be a Microsoft junkie to think highly of distributed COM; CORBA is far more suited to the task of distributing objects, and on top of that can be used on just about any platform.

Jun Du wrote:

Second, some like to write their own remoting based on sockets. I think your try falls into this category. But they both require significant rework on the old code

The advantage of building your own remoting is that you can include only the things that you think are important, and leave the rest out, or at least make them optional. Neither COM nor .NET remoting allow that. I'm not really understanding what you mean by "significant rework on the old code"; adding and using RCF in a project would not require any changes at all to existing code to make it work. Once again, thats not true for either COM or .NET remoting.

Jun Du wrote:

Third, .Net has provided another alternative, Interop,

Using .NET Interop between unmanaged Win32 programs seems very awkward to me. You're adding the whole .NET overhead, to both ends, just to do remote invocations.

Well, for one, you might want to make and receive remote calls to and from non-Windows platforms. With .NET remoting you're locking yourself in, right from the start

Of course there are more issues here than just the availability of .NET on the target platform; endian-ness and alignment issues can cause problems too.

Jarl Lindrud wrote:

There's a whole world of programming outside Win32, and I can guarantee you that COM is not in high demand there

Very true, but in the win32 world DCOM is still the commonest solution.

Jarl Lindrud wrote:

Using .NET Interop between unmanaged Win32 programs seems very awkward to me

This is very true. Even in mixed managed/unmanaged apps it is painful. However, there is one key difference between Jarl's RMI and more heavyweight solutions such as DCOM & .NET remoting (& CORBA too): The later all have a mechanism for instantiating objects remotely, and for passing objects transparently back and forth across an interface - they are object-centric rather than interface-centric. There are times when this is good, and times when this means you end up with an enormous amount of unwanted baggage.

fwiw we evaluated Jarl's RMI lib, DCOM & .NET remoting for some new projects & settled on .NET remoting for those projects which had other reasons for having partly-unmanaged code and rolled our own low-level remoting for other lightweight apps.

We didn't use this library for two reasons:
1. Difficulty of integrating boost.thread with our own threading and serialisation libraries, and
2. Not quite enough documentation to really get to grips with customising the behaviour of the library.

I did however draw inspiration from a number of ideas in the source code even if I did end up with something rather different.