Please Note

Introduction

The officially sanctioned way of making distributed function calls between C++ programs is to use CORBA, but for many applications, this is overkill. The CORBA specifications allow distributed function calls to be made between code written in any number of languages, and to make it all work, specialized tools need to be integrated into the build process, in order to translate object definitions written in CORBA's IDL to whichever native language is being used (C++, Java, etc.).

However, if we assume that the server and client are both written in the same language, let us assume C++, since it is possible to do away with these complexities. In particular, instead of elaborate definitions of interfaces and marshalling specifications, we can simply defer to C++.

Instead of separate IDL files with object interfaces, we specify the interfaces directly in C++ source code, using the preprocessor, and to marshal arguments across process boundaries, we use the native C++ serialization framework provided in the latest release of the Boost library.

The Boost.Serialization library is used to serialize parameters and return values. It handles standard types and containers automatically, and is easily extended to user defined classes. It also allows us to serialize pointers, with proper handling of polymorphic pointers and multiple pointers to single objects.

Basic Usage

There are three basic steps to using this framework:

Use the RCF_xxx macros to define interfaces.

Use the RcfServer class to expose objects that implement the interface.

Use the RcfClient<> classes to invoke methods on the objects exposed by the server.

type is the identifier for the interface, type_id is a string giving a runtime description of the interface. The RCF_METHOD_xx macros define the member functions, and are named according to the number of arguments and whether the return value is void or not. So, for a function func accepting two strings and returning an integer, we write:

RCF_METHOD_R2(int, func, std::string, std::string);

and if the function has a void return type, we would instead write:

RCF_METHOD_V2(void, func, std::string, std::string);

Dispatch IDs for each function are generated automatically; the first member function is numbered 0, the next one 1, and so on. So, the order in which the functions appear in the definition is important, unlike in CORBA, where dispatch IDs are based on the function name. The dispatch IDs are generated using templates and not any preprocessor __LINE__ trickery, so the interface does not change if blank lines are inserted. The maximum number of member functions that can appear between RCF_BEGIN() and RCF_END() is at the moment limited to 25, but this limit is arbitrary.

The purpose of the RCF_xxx macros is to define the class RcfClient<type>. This class serves as a client stub, from the user's point of view, but also has facilities that allow the framework to use it as a server stub. These macros can be used in any namespace, not just the global namespace.

Once we have defined an interface using the RCF_xxx macros, we can start a server and bind the interface to concrete objects:

{
// create the server and tell it which port to listen on
RCF::RcfServer server(port);
// Interface is the identifer of the interface we're exporting,
// Object is a type that implements that interface
// one object for each client
server.bind<Interface, Object>();
// ... or one object shared by all clients
Object object;
server.bind<Interface>(object);
// tell the server to start listening for connections
server.start();
// ...
// the server will shut down automatically as it goes out of scope
}

The objects are statically bound to the corresponding interface; there is no need for the object to derive from an interface class as is the case for traditional dynamic polymorphism. Instead, the compiler resolves the interface at compile time, which is not only more efficient, but also allows more flexible semantics.

The server can handle multiple simultaneous clients, even in single threaded mode, and can be stopped at any time. The lifetime of objects exposed by the server is determined by the number of current connections to the given object; once there are no more live connections to the object, a timeout is set, and when it expires, the object is deleted.

To make a client call, we instantiate the corresponding RcfClient<> template and pass the server IP and port number to the constructor. When the first remote method is called, the client then attempts to connect to the server, queries for the given object, invokes the requested member function of the remote object, and then returns the remote return value.

Should any exceptions arise on the server side while invoking the requested object, an exception of type RCF::RemoteException will be propagated back to the client and thrown. Should any exceptions arise anywhere else on the server side, e.g., while serializing arguments, then the server will forcibly close the connection, and the client will throw an exception.

RCF will automatically handle a range of parameter types, including C++ primitive types (int, double, etc.), std::string, STL containers, and pointers and references to any of the previously mentioned types. Polymorphic pointers and references, and multiple pointers to single objects are correctly handled as well. Smart pointers are also supported (boost::shared_ptr, std::auto_ptr), and are the safest way of passing polymorphic parameters.

In CORBA, one can tag a parameter as in, out, or inout, depending on which direction(s) one wants the parameter to be marshaled. In RCF, the marshaling directions are deduced from the parameter type, according to the following conventions:

Value: in
Pointer: in
Const reference: in
Nonconst reference: inout
Nonconst reference to pointer: out

To use user-defined types as parameters or return values, some additional serialization code is needed. What that code is depends on which serialization protocols are being used; by default Boost.Serialization is used, and an example of passing a user-defined type would look like the following:

Details

The server and client classes use BSD-style sockets to implement the networking, over TCP, and the whole framework has been compiled and tested on Linux, Solaris (x86 and SPARC) and Win32, using Visual C++ 7.1, Codewarrior 9.0, Borland C++ 5.5, and GCC 3.2. Building RCF requires v. 1.32.0 or later of the Boost library, although the only parts of Boost that need to be built are Boost.Serialization, and, for multithreaded builds, Boost.Threads. Multithreaded builds are enabled by defining RCF_USE_BOOST_THREADS before including any RCF headers.

To use RCF in your own application, you'll need to include the src/RCF.cpp file among the sources of the application, and link to the necessary libraries from Boost, along with OS-specific socket libraries (on Windows that would be ws2_32.lib, on Linux libnsl, etc.).

I've included a demo project for Visual Studio .NET 2003, which includes everything needed to compile, link, and run a server/client pair, with the exception of the Boost library, which needs to be downloaded and unzipped, but no building is needed.

Performance, as measured in requests/second, is highly dependent on the serialization protocol, and also on the compiler being used. Before turning to Boost.Serialization, I used a serialization framework of my own, with which I could clock around 3000 minimal requests/sec. using Visual C++ 7.1, and 3300 requests/sec. with Codewarrior 9.0, on a loopback connection on a 1400Mhz, 384Mb PC running Windows XP. GCC 3.2, on the other hand, was far slower. Using Boost.Serialization, however, I've been nowhere near these numbers; on average, it's around five times slower.

Conclusion

RMI is a well known concept in Java circles, what I've done here is to do something similar in C++, without all the complications of CORBA. If you like it, please tell me, if you don't, well, please tell someone else.... Jokes aside, any and all feedback is appreciated, all I ask is that if you grade the article, and do so with a low grade, then please leave an explanatory comment!

History

8 Feb 2005 - First release.

10 Mar 2005

Now includes a custom serialization framework, so you no longer have to use Boost's. Both serialization frameworks are supported though, use the project-wide RCF_NO_BOOST_SERIALIZATION and RCF_NO_SF_SERIALIZATION defines to control which ones are used. Default behaviour is to compile both.

Default client timeout changed to 10s.

Server can be configured to only accept clients from certain IP numbers.

Server can be configured to listen only on a specific network interface, such as 127.0.0.1.

Client stubs automatically reset their connections when exceptions are thrown (eg for timeouts).

Finer-grained exception classes.

11 July 2005

Stripped CVS folders from distribution.

Added user-definable callback functions to be called when RcfServer has started.

16 Aug 2005

Added facilities for server-bound objects to query the IP address of the client that is currently invoking them. To see how it works, open the file RCF/test/Test_ClientInfo.cpp in the download. Just place a call to RCF::getCurrentSessionInfo().getClientInfo().getAddress(), and you'll receive a string containing the IP address of the client that is invoking the method.

23 Sep 2005

Initialization and deinitialization of the framework can now be done explicitly, be defining the project-wide preprocessor symbol RCF_NO_AUTO_INIT_DEINIT, and then calling RCF::init() and RCF::deinit() at appropriate times. This is mainly useful for DLL builds, so that the DLL can be loaded without automatically initializing Winsock.

19 Oct 2005

Compatible with Boost 1.33.0.

Added enum serialization to the built-in serialization engine, through the SF_SERIALIZE_ENUM macro. For an example of its use, see test/Test_Serialization.cpp.

Added a license.

30 Jan 2006

Miscellaneous bugfixes.

The built-in maximum message size limit has been changed to 50 Kb. Look in src/RCF/Connection.cpp, line 374, if you need to change this.

I'll only be making sporadic maintenance releases of this version of RCF from now on. You can find the next generation of RCF here.

Share

About the Author

Software developer, ex-resident of Sweden and now living in Canberra, Australia, working on distributed C++ applications. Jarl enjoys programming, but prefers skiing and playing table tennis. He derives immense satisfaction from referring to himself in third person.

I am wondering if compatible implementations exist in Perl or PHP or Python or any other popular scripting language?

I am looking for something that would give me the interoperability of SOAP, without the performance penalty.

To give you a bit more details - I am looking for a framework to use for an open API project, but I do not want to have to re-implement the API for every single target language. SOAP gives us this flexibility. However in certain cases high volumes of messages need to be exchanged, and in those cases SOAP becomes a major performance bottleneck.

I am sorry if this has already been answered, but I did search the forum as best as I could.

Thanks!

vess

P.S. If I am looking at the wrong tool, and there are better suggestions, please let me know as well.

My task is to port a 3d render engine from 32-bits to 64. There are a lot of things are (at least) difficult to port (GUI, QuickTime calls, plug-ins, shaders etc). Thus I plan to port "core" to 64 and leave other parts in 32. The only way to do this is to have 2 processes (one 32-bits, another one 64) and organize their interactions. So my questions are:

1) Can I use RCF for OSX (Carbon) platform?
2) Can I use RCF for Windows platform?
3) What is a fastest IPC (no probs if it's low-level/primitive, speed is critical for me)?

RCF works well as a 32/64 bit bridge, I've seen it used for that purpose myself. As for your questions,

1) Probably, with some minor tweaking. I don't have access to an OSX platform to test on, but I know users who have made small changes to RCF, and been able to run it on OSX.

2) Yes.

3) In RCF 0.9c you would be best off using TCP. UDP is faster, but doesn't offer any form of connection-oriented interaction, which is often a deal-breaker.

RCF will, probably in the next release, support named pipe transports on Windows, which will be a bit faster than TCP. TCP on a localhost connection is quite fast though, most applications would not have a problem with it.

Hi Jarl,
First,only tell you that I think this is one of the best and more usable code ever written!!
Now, I only have one question,how can I develop an asynchronous call?could you give me an example??

Thanks hulka. By the way, there are much newer versions of RCF, have a look here:

http://www.codeproject.com/threads/Rcf_Ipc_For_Cpp.asp

Asynchronous calls are not yet built in to RCF, but I am working on that right now... The next release of RCF will have native asynchronous capability, but until then, if you need a thread to do something while a call is pending, you can either use client progress callbacks, or start your own background threads, and queue up remote calls on that while the main thread does something else.

In RCF 0.9c, have a look in Test_ClientProgress.cpp to see how client progress callbacks are used.

In this case, compiler generates error! In Compile process, code “RCF_METHOED_V1(void, func1, IntArray)” is perhaps converted to a code to instantiate “SF::DynamicArray” with no argument. but in SF/SerializeDynamicArray.hpp, no default creator is defined. So error is generated(I guess from complier error message).

To send an array of integers, you should use std::vector as the type of the parameter. Static arrays like int[] are not supported as interface parameters. SF::DynamicArray is an internal implementation class, and is not intended for use as an RCF interface parameter, either.

Sorry for the late response... If you get the latest version of RCF, RCF 0.9c, available here , you can use the RCF::ByteBuffer class as a parameter in your interfaces, and that should be a lot faster than using std::vector with older versions of RCF.

I would need to solve this problem:
If some remote function does not respond to the client RCF::RecvTimeoutException is thrown.
But how to kill a remote function on the server which caused this exception.
Could you help me ?
Thanks,
Petr

It sounds like you want to explicitly kill a server thread, which is not really a good idea. In general there's only 1 thread in the server, answering client requests one at a time, so if you kill it, the server will no longer respond to any client requests.

If you're bent on doing it, though, the operating system API has functions for killing threads, any of those would do.

I was looking for a lightweight RPC library for C++ and found your page. This looks like a very nice interface and attracted me immediately.

However, it seems to rely on boost 1.33.0. I have two questions:

1. Are you planning to make it independent of boost? How much effort would that be? (I could probably help a bit if it's not too much of work)

2. Or are you planning to make it compatible with boost 1.33.1 (or whatever the latest version is)? I tried to build it with 1.33.1 and it complains about missing files (test/minimal.hpp, read_write_mutext.hpp, etc)

Sorry, I didn't realize there were any problems compiling with boost 1.33.1, I'll put together an update and post it here. The code is meant to work with any of the newer versions of boost.

As for making RCF independent of Boost, that has so far not been a high priority for me I've used Boost pretty extensively thoughout RCF, and eliminating it would be a lot of work, with questionable results, IMHO.

Finally, I should point out that I'm no longer actively maintaining this version of RCF; there is a newer version available here:

I just wanted to put in a good word for your project. I am using the C++ RMI framework for a project and found the overall setup very nice. We are using the framework to glue a Matlab application to binary data base. Writing/modification of the Matlab client ".mex" was the most challenging part. We did have to up the maximum message size from the original 1024*50 to support our needs.

Could this code be used to "new up" proxy objects on the client side that mapped to concrete objects on the server. Example - a video server, where there is a server class CTwoDObject with a bunch of methods. I want to be able to create proxy object representations on the client side and have the server create concrete objects, then be able to invoke methods on these objects and have your code handle the messaging between the client-server.

Right now, I have this working currently by having each method Send a command to pass a message to the server (each object is identified with a unique handle). It is rather cumbersome as each method has a SendCommand and/or ReceiveReply. Also, the server has to have something similar.

Have a look at the ObjectFactoryService class, it lets clients create objects on the server and then call methods on them. There's a short example in the article, if you have any questions about it just ask.

I've made quite a few changes to RCF over the last 6 months, but they're on a different branch than this version, you can find the new version here[^]. I'll answer your questions based on that version:

chrhol wrote:

As a user I want to be able to use a transport layer of my choice, and not be bound to the default tcp implementation. Can this be done?

The transport layer is now separated from the rest of RCF (it wasn't before), so you can write your own; I've written server and client transports for TCP and UDP, you can check them out to see how it works.

chrhol wrote:

Can I bind application specific data to a client connection on the server side?

Not right now, but it should be easy for me to add, I'll add it to the list of things to implement for the next release. What you can do at the moment is to let clients create objects on the server, using ObjectFactoryService, and have the client-specific data in those objects. They're not tied to the physical connection though.

chrhol wrote:

Can I make asynchrounous calls from the client?

Besides the usual two-way request/response way of making a call, you can also make one-way calls; the client sends a message to the server, but then doesn't hang around for a reply, and in fact the server never sends one either. Proper asynchronous calls are currently not implemented, ie where you're informed at a later point in time that a call has completed.

chrhol wrote:

Can I feed the server and client with raw input and it'll decode and invoke the content?

Not implemented, but that probably wouldn't be hard to setup. In what scenario would you be using it?

For tunnelling your RCF protocol over an already established connection where I want to mix protocols from different systems. If I oversimplify, I would like to do something like this:

OK I see, that would be useful. I'll put it on the list of things to do. Might be a little while before I get around to dealing with it though, I'm a bit occupied at the moment getting RCF to run on Unix, but after that

It's been a while since I built RCF on solaris, but I used bjam, from Boost, to build it, instead of a makefile. I can use the same jamfile to build things on Windows/Linux/Solaris, with just about any compiler, and I find it a lot simpler than using makefiles.

It shouldn't make any difference, though, which build tool you're using. Which version of which compiler are you using? What errors are you getting? Compiler errors or linker errors?

First of all it's enough to just compile RCF.cpp and SF.cpp, they will pull in the rest of the source files themselves. So the rules for SF.o and RCF.o only need to compile SF.cpp and RCF.cpp, respectively.

The error that you're getting is because you're trying to compile MyService.hpp. You need to remove the MyService.o target, it will be included as a .hpp file into the server and client anyway.

Other than that, you should be fine. The jamfile that I use to build RCF test executables is in the /test directory, called "Jamfile". It has lots of targets, but you only need to build one at a time. I can type "bjam -sTOOLS=gcc -sBUILD=debug" Test_Minimal" to build a debug executable with the gcc compiler, for instance.

You'd need to build bjam first. It's part of boost, and there are instructions there on how to do it. Probably you only need to go to the ...boost\tools\build\jam_src and run the build.sh script.

Hello!
I ve used SF not the boost lib to serialize. No have found that the executable moans about the size of the objects to be returned. It says:
..\rcf\src\rcf\connection.cpp(376): int __thiscall RCF::Connection::receive(void)::Thread-id=2436:
Timestamp(ms)=52327: THROW : RecvException: too much data: fd=1944, length=15592,

How can I overcome this error?
Many thanks in advance
Reinhard Stoeckl

There's currently a hardcoded 10K limit on message sizes, which is what you're running into. The message being sent to the server is ok, but the message coming back is apparently too big. You'll need to change the limit to something more appropriate; look in src/RCF/Connection.cpp, around line 375.

And yes, the next release will have a proper way of setting the maximum message size...

|| I used a serialization framework of my own,
|| with which I could clock around 3000 minimal requests/sec.
|| using Visual C++ 7.1, and 3300 requests/sec.
Would you be willing to share you serialization framework? Perhaps attach it to your zip file. This would be a viable alternative to the boost approach which will pull perfomance down tremendously.

A RCF::Token object is assigned to each client. When the connections limit exceeds, the number if free tokens turns into zero and further connection attempts always fail.
The question is: how is it possible to put the free token back into the pool of free token explicitly ?? or just to break down the connection in other words ?? The most wanted way, as I see, is to leave the scope of the RcfClient object or maybe it is a call to some method like client.disconnect(). Please, enlight me, in such a bottleneck.

The best thing is to set up your appllication so that you bind interfaces directly to an instance:

RcfServer server(ip, port);X x;server.bind<I_X>(x);

Then all the clients will be accessing the same object, they won't be needing individual tokens, and the token limit will no longer play a role.

If you still need separate server objects for each client, then you could increase the number of tokens, if you want. Currently its hardcoded to 100, but you can change the value of NumberOfTokens in RcfServer.hpp to something bigger.

The tokens are returned to the pool when the corresponding server object is deleted. That only happens when a server object has been inactive for 60 seconds though, so after a RcfClient goes out of scope, it will still take a minute before the server decides to delete the object and recycle the token.

Hope that helps If possible, go with the first option, with all the clients sharing an object, and then the issue won't turn up at all...

I have got one more question about the RCF connections. How can I track the connection event - I need to have an opportunity to accept or to refuse pending connection. I have found out, that your framework can filter the ip-addresses of connecting clients, but I need a little bit more sophisticated server logics. For example: I have a data storage nested in server. The server has a "reader" interface and a "writer" interface. The idea is - the server must distinguish the client's access - "writer" interface must not be allowed for "everyone", except hard-fixed "priviledged" clients. And another sub-question - how can I implement the "many readers, one writer" concept - like the boost::read_write_mutex, using the RCF facilities ??
If it is possible to handle the both problems, I will be very happy .

In test/Test_ClientInfo.cpp, there's an example of using the RCF::getCurrentSession() function, which can be called from within any server thread and gives information on the session that is currently being processed. Calling RCF::getCurrentSessionInfo().getClientInfo().getAddress()will give you a string representation of the client's IP address, and then from that you can decide whether you want to grant the client read or write access.

Yeah, it's been on my mind... I've been busy the past months rewriting and extending RCF, and when its released I'll see what the people at Boost think of it, and if they think it could eventually be accepted.

Can't say I'm looking forward to having to change the names of all my classes and functions, though, to agree with the Boost guidelines...

No, it's still supported, its just a little awkwardly configured. The original error you got was because the client wasn't configured to use one of the boost serialization protocols; it was still trying to use SF.

You have to specifically tell the client which serialization protocol to use, before making any remote calls: