Introduction

RCF (Remote Call Framework) is a C++ IPC framework, providing an easy and consistent way of implementing interprocess communication in C++ programs. It is based on the concept of strongly typed client/server interfaces, a concept familiar to users of IDL-based middlewares such as CORBA and DCOM. However, by catering only to C++, RCF can harness C++-specific language features that allow the formulation of interprocess calls in a simpler and less cluttered manner.

RCF provides a number of IPC messaging paradigms (oneway, batched oneway, request/response, publish/subscribe) over a number of transports (TCP, UDP, Windows named pipes, UNIX local domain sockets) with support for compression (zlib) and a number of encryption and authentication technologies (Kerberos, NTLM, Schannel, OpenSSL). You can use RCF in all kinds of IPC scenarios - from simple parent-child process communication, right up to large-scale distributed systems. Best of all, RCF is 100% standard C++, and will run on any platform with a C++ compiler.

RCF releases and documentation are available on the RCF website. This article presents a brief overview of RCF.

Basics

We'll begin with a simple echo server and client. We want a server to expose to its clients a function that accepts a string, and which returns the same string. Using RCF, the server code becomes:

I_Echo is an RCF interface defined by the RCF_BEGIN/RCF_METHOD/RCF_END macros. These macros correspond to what in CORBA would be an IDL definition, but in this case, the interface definition is placed inline in C++ source code, and requires no separate compilation step, as is the case for IDL interfaces. The interface is simply included in the source code of the server and the client, and compiled along with the rest of the program.

The RCF::Twoway argument in the client stub invocation is a flag that tells RCF to make a two-way client call; the client sends a request, waits for a response, and if none is received within a configurable duration of time, an exception is thrown. The other option is to use RCF::Oneway; a request is sent, but the server does not send a response, and the client stub call will return control to the user immediately. If the directional argument is omitted, RCF::Twoway is used by default.

The client stub is not synchronized in any way, and should only be accessed by a single thread at a time. The server, though, is inherently multi-threaded, although in the example above, it has been written as a single-threaded process. RcfServer::startInThisThread() hijacks the calling thread and converts it to a worker thread of the server.

We can also call RcfServer::start(), and the threads needed to drive the server would then automatically be created. RcfServer::start() returns immediately after launching the server threads. If we did that, we would need to put in a wait loop for the main thread so that it doesn't leave main(), shutting down the entire process.

We can rewrite our client/server pair to use the UDP protocol instead. This time, we'll let the server and the client both reside in the same process, but in different threads:

UDP, of course, is significantly different from TCP. TCP provides reliable, ordered delivery of network packets, while UDP makes no guarantees at all regarding the order of packets that are sent or whether the packets will even arrive at all. On a loopback connection, like in the example above, one might get away with using two-way semantics over UDP, since the packets are not being subjected to the vagaries of a real network. In general, however, you would only want to send one-way calls with UDP.

Interfaces

The interface definition macros function exactly like they did in the previous generation of RCF. The RCF_BEGIN() macro commences an interface definition with the given name and runtime description, and RCF_END() ends the interface definition. In between, one can have up to 25 RCF_METHOD_xx() macros to define the member methods of the interface.

The last two letters in the RCF_METHOD_xx() macros indicate the number of arguments and whether the return type is void. For instance, RCF_METHOD_V3 is used to define a method with three parameters and a void return value, while RCF_METHOD_R2 defines a method with two arguments and a non-void return value.

The net result of these macros is to define the RcfClient<type> class, where type is the name of the interface. This class is then used directly by the client as a client stub, and indirectly by the server as a server stub. RCF interfaces can be defined in any namespace:

Server Bindings

On the server side, the interface needs to be bound to a concrete implementation. This is done through the templated RcfServer::bind() functions. There are several variations that accommodate various memory management idioms. The following calls all accomplish pretty much the same thing, the binding of an Echo object to the I_Echo interface:

{
// Bind to an object...
Echo echo;
server.bind<I_Echo>(echo);
// or to a std::auto_ptr<>...
std::auto_ptr<Echo> echoAutoPtr(new Echo());
server.bind<I_Echo>(echoAutoPtr);
// or to a boost::shared_ptr<>...
boost::shared_ptr<Echo> echoPtr(new Echo());
server.bind<I_Echo>(echoPtr);
// or to a boost::weak_ptr<>...
boost::weak_ptr<Echo> echoWeakPtr(echoPtr);
server.bind<I_Echo>(echoWeakPtr);
}

By default, the binding is available to clients through the name of the interface. The server can also expose several objects through the same interface, but in that case, it needs to explicitly set the binding names:

Marshaling

RCF follows C++ conventions when it comes to determining in which directions arguments are marshaled. In particular, all arguments to an interface method are in parameters, all non-const reference arguments are inout parameters, and the return value is an out parameter. There are currently no provisions for overriding these conventions along the lines of the in/out/inout qualifiers in IDL definitions.

Not everything in C++ can be safely marshaled, and this places some limitations on the types of arguments to interface methods. To wit: pointers and references are allowed as arguments; references to pointers are not allowed as arguments; pointers and references are not allowed as return values.

This means that if an interface method needs to return a pointer -- for instance, a polymorphic pointer -- then the return type needs to be a smart pointer such as std::auto_ptr<> or boost::shared_ptr<>. Alternatively, one of the arguments could be a non-const reference to a smart pointer.

Serialization

The echo example only serialized a std::string object, but it is possible to send almost any C++ class or structure within the limits of the serialization system in use. RCF has its own serialization framework, prosaically named Serialization Framework (SF), but it also supports the Boost.Serialization framework, part of the Boost library.

In general, it is necessary to include serialization code for the classes that are being marshaled. If you have a std::vector<> argument in an interface, you will need to include <SF/vector.hpp> or <boost/serialization/vector.hpp> (or both), and similarly for other STL and Boost classes.

To use your own classes as arguments in RCF interfaces, you will need to define custom serialization functions. In most cases, it's quite simple:

Serialization can quickly become a complex affair, especially when dealing with pointers of polymorphic objects and cycles of pointers. Both SF and Boost.Serialization handle these situations, but in different ways, and catering to both may require writing separate serialization code for each serialization system. The following is an example of sending polymorphic objects, using both SF and Boost.Serialization:

Inheritance

Multiple inheritance is now supported for RCF interfaces. Interfaces may inherit not only from each other, but also from standard C++ classes. Methods in an interface are identified by the combination of their dispatch ID and the name of the interface that they belong to. This information suffices for the server to map an incoming client call to the correct function on the server binding. The example below demonstrates interface inheritance:

Filters

RCF supports compression and encryption of messages out of the box, by way of a filter concept. Filters are applied to both server- and client-side, and can be applied either to the transport layer -- for instance, applying an SSL filter to a stream-oriented transport like TCP -- or to individual message payloads such as, for instance, compression of messages destined for a packet-oriented transport like UDP. The first case we call a transport filter, and the second case, a message filter.

Transport Filters

The process of installing a transport filter on a server-client conversation is initiated by the client. The client queries the server to determine if the server supports a given filter. If the server does, the filter is installed on both ends of the transport, and the communication resumes.

The procedure of querying the server and installing the filter is handled by the client stub. The client calls the ClientStub::requestTransportFilters() function, and if negotiation with the server is successful, the filter sequence is then active and applies to all subsequent remote calls.

Transport filters can be two-way, in the sense that a single read or write request on a filter may result in multiple read and write requests being made on the connection. For example, the first message sent over an encryption filter will initiate some kind of a handshake involving multiple downstream read and write operations.

Message Filters

Message filters do not require any negotiation on behalf of either the server or the client. If a client stub has been imbued with a sequence of message filters via ClientStub::setMessageFilters(), then the message will be passed through the given filters. The encoded message will be prefixed with data describing which filters have been used, thereby allowing the server to decode the message. Should the server not recognize a filter, an exception is passed back to the client.

RCF comes with several filters out of the box: two for compression, based on Zlib, two for SSL encryption, based on OpenSSL and Schannel, and two Windows based filters for the Kerberos and NTLM protocols. These filters can also be chained to each other to create sequences of filters.

The encryption filters can only be used as transport filters, since the process of encryption and decryption requires a two-way handshake. The compression filters can be used either as transport or message filters, but over non-stream-oriented transports such as UDP, it's only meaningful to use the stateless compression filter.

Remote Objects

The RcfServer class allows users to expose single instances of a class to remote clients, but it makes no provisions for remote clients to create any objects on the server. This functionality is instead part of the ObjectFactoryService service. Once an ObjectFactoryService service has been installed into a server, it becomes possible for clients, via the ClientStub::createRemoteObject() function, to create as many objects as the service is configured to allow.

The ObjectFactoryService service implements a garbage collection policy, whereby objects that are no longer in use (i.e., have no active client sessions and have not been accessed for a configurable duration of time) are eventually deleted.

{
// Allow max 50 objects to be created.unsignedint numberOfTokens = 50;
// Delete objects after 60 s, when no clients are connected to them.unsignedint objectTimeoutS = 60;
// Create object factory service.
RCF::ObjectFactoryServicePtr ofsPtr(
new RCF::ObjectFactoryService(numberOfTokens, objectTimeoutS) );
// Allow clients to create and access Echo objects, through I_Echo.
ofsPtr->bind<I_Echo, Echo>();
RCF::RcfServer server(endpoint);
server.addService(ofsPtr);
server.start();
}
{
RcfClient<I_Echo> client1(endpoint);
bool ok = client1.getClientStub().createRemoteObject();
// client1 can now be used to make calls// to a newly created object on the server.
RcfClient<I_Echo> client2(endpoint);
client2.getClientStub().setToken( client1.getClientStub().getToken() );
// client1 and client2 will now be making calls to the same object
}

Session Objects

Objects created by ObjectFactoryService are identified by a token, and can be accessed by any client that specifies that particular token. So once created, such an object can potentially be accessed by many clients. It is quite common, though, for a client to want to create objects that are only accessible by that particular client. In RCF, such objects are called session objects, and are created by SessionObjectFactoryService:

Publish/Subscribe

The Publish/Subscribe pattern is a well-known pattern of distributed programming. One process plays the role of a publisher and sends regular packets of information to a group of subscribers. The subscribers call in to the publisher and request subscriptions to the data that the publisher is publishing.

For packet-oriented transports such as UDP, it is relatively straightforward to build this functionality on top of existing RCF primitives. For stream-oriented transports, TCP in particular, RCF provides some extra features to enable publish/subscribe functionality. The connection that a subscriber calls in on is reversed and then subsequently used by the publisher for publishing.

This functionality is encompassed by the dual PublishingService and SubscriptionService services. The following example describes how these services are used:

Extensibility

RcfServer Services

The RcfServer class accommodates third party extensions through a service concept. Services are notified, among other things, when the server starts and stops. Also, services can be dynamically added and removed from the server while the server is running. A typical service may bind an object to an interface on the server, and may also request the server to spawn a thread (or spawn its own threads) to do some kind of periodic activity.

Server transports are implemented as services. It is thus possible to have several transports serving a single RcfServer object. Some of the services available are:

ObjectFactoryService: Allows clients to create objects on the server

FilterService: Allows the server to dynamically load filters in response to client requests

PublishingService: Enables publishing functionality on a server

SubscriptionService: Enables subscription functionality on a server

Portability

Compilers

RCF has been tested against a large number of compilers, among them Microsoft Visual C++ (from version 6.0), Borland C++ (from version 5.6), and GCC (from version 2.95). For a complete list, see the RCF User Guide.

Platforms

RCF's TCP server implementation is now based on Win32 I/O completion ports, a technology limited to Windows 2000 and later. The TCP client implementation and the UDP server/client implementations are based on BSD sockets, and should be widely portable. On non-Windows platforms, and optionally on Windows as well, RCF leverages the Boost.Asio library to implement TCP servers.

Building

In general, to build an application using RCF, you need to include the file src/RCF/RCF.cpp among the sources of your application. You will also need the header files of the Boost library; any version since 1.33.0 should be OK.

There are several preprocessor symbols that can be used to control which parts of RCF are compiled. These symbols need to be defined globally, i.e., they should be placed in the project settings, and not defined/undefined in source code.

RCF_USE_BOOST_THREADS: Utilize the Boost.Thread library for mutex and thread generation capabilities. If not defined, RCF will use its own internal thread library.

RCF_NO_AUTO_INIT_DEINIT: Disables RCF's automatic initialization/deinitialization facility. If defined, the user will have to explicitly call RCF::init() and RCF::deinit() at appropriate times. In particular, this is necessary when compiling RCF into a DLL in order to avoid premature initialization.

All third party build dependencies (Boost.Threads, Boost.Serialization, Zlib, OpenSSL) are optional. Instructions for building these libraries are beyond the scope of this article, but if you're having trouble building the Boost libraries, an option is to bypass Boost's build utility, bjam, and instead, simply compile the corresponding CPP files into your application.

To use the Boost.Threads library, for instance, you can just include the CPP files in boost_root/libs/thread/src in your project. Then, define one of the symbols in boost_root/boost/thread/detail/config.hpp, whichever one is appropriate (probably BOOST_THREAD_USE_LIB).

Testing

There is a comprehensive set of tests in the /test directory of the download, all of which should compile and run without failures. The tests can be built and run automatically using the Boost.Build utility, or built and run by hand. The tests exercise more functionality than I've mentioned here, and may be useful as a source of information for users.

History

2005-12-23 - Version 0.1 released

2006-04-06 - Version 0.2 released

RCF now compiles and runs on Linux and Solaris, as well as on Windows. Servers and clients can be distributed across multiple platforms, and still communicate seamlessly.

The Asio networking library is a prerequisite for using RCF on non-Windows platforms. Download Asio, make sure that the Asio headers are available to your compiler, and make sure the preprocessor symbol RCF_USE_ASIO is defined when compiling RCF. Asio requires version 1.33.0 or later of Boost, and you'll probably want to define BOOST_DATE_TIME_NO_LIB when using it in order to avoid unnecessary Boost dependencies.

RCF 0.2 has been compiled and tested on the following compilers:

GCC 3.2 (MinGW on Windows)

GCC 3.3 (Linux)

GCC 3.4 (Solaris)

Borland C++ 5.6

Metrowerks CodeWarrior 9.2

Microsoft Visual C++ 7.1

Microsoft Visual C++ 8.0

Various bugs reported on this forum have also been fixed.

Apologies to anyone who's been waiting for this release... I originally planned to release it several months ago, but a busy work schedule (and some stubborn test cases) intervened. Sorry!

2006-07-30 - Version 0.3 released

RCF support for Asio has now been advanced to version 0.3.7 from 0.3.5, by David Bergman. Thanks, David!

For those using RCF to communicate between 32 and 64 bit systems, you can avoid serialization errors caused by size discrepancies between 32 and 64 bit systems, by not using types like long or std::size_t in RCF interfaces. Use one of the boost typedefs instead (boost::int32_t, boost::uint32_t, boost::int64_t, boost::uint64_t).

Finally, thanks to SÃƒÂ¶ren Freudiger at the Technical University of Braunschweig, for lending me an account on their 64-bit Linux machines!

2007-07-11 - Version 0.9c released

RCF 0.9c, a pre-release version of RCF 1.0, is now available from the download section of Google Code. In development for over a year, RCF 0.9c constitutes a major restructuring and upgrade of RCF 0.4. It has been thoroughly put through its paces as the networking backend of a major commercial ECM platform.

Applications using RCF 0.4 should have relatively few difficulties upgrading to RCF 0.9c. If you have any questions on porting your applications to 0.9c, feel free to contact me either on this forum or via email. I will be happy to help you out.

RCF 0.9d-P1 is a preview of RCF-0.9d. It has been fully tested on Windows, against the Visual C++ range of compilers (6.0, 7.1, 8.0, 9.0). The RCF 0.9d release will include full support for Linux and Solaris, and extensive documentation in the form of a User Guide.

If you are upgrading to RCF 1.2 from an earlier version of RCF, there are several breaking changes to be aware of:

SF serialization functions now have a different signature. The redundant version parameter has been removed, so for external serialization functions, the signature is now "void serialize(SF::Archive &, SomeType & t)". For internal serialization functions, the signature is now "void SomeType::serialize(SF::Archive &)".

The global serializeParent() function has been moved into the SF namespace.

The RCF::Exception::getError() function has been renamed to getErrorId().

License

Share

About the Author

Software developer, ex-resident of Sweden and now living in Canberra, Australia, working on distributed C++ applications. Jarl enjoys programming, but prefers skiing and playing table tennis. He derives immense satisfaction from referring to himself in third person.

Comments and Discussions

Thanks for your feedback. I'm not up to speed on the Cygwin environment - what changes would be needed to make it work? RCF 2.0 is using an internal copy of Boost.Asio, which I believe should work on Cygwin.

On auto_ptr/unique_ptr, RCF still needs to support pre-C++0x compilers, so unfortunately we can't just replace auto_ptr with unique_ptr...

What is the reason for the change? With USE_BOOST_THREAD and USE_BOOST_ASIO, pre-v2.0 compiled fine on Cygwin. I believe it is RCF's own #error directive catching a misconfiguration. For whatever reason (haven't looked at the code - just tried a quick compile), I don't think it picks up on the fact that it's being compiled under Cygwin. I'll see if I can spend some time tracing through the code to see why it it flags the error.

A simple way out of auto_ptr/unique_ptr could be solved by perhaps another #define (e.g. RCF_USE_CPP0X)?

We've dropped RCF_USE_BOOST_ASIO and RCF_USE_BOOST_THREADS in order to simplify the codebase. RCF's internal copy of Boost.Asio is now used for both networking and threading. That being said, I can't think of any reason it would not work on Cygwin, so if you can have a closer look and pinpoint the offending code, that would be great.

RCF_USE_CPP0X is definitely an option, and we'll probably need it at some stage. Is auto_ptr<> causing you problems in terms of compiler warnings?

Using external boost, you have "BOOST_WINDOWS" defined, which bypasses the need to define __USE_W32_SOCKETS. In your Boost.Asio, you're missing several platform-select config files that defines BOOST_WINDOWS. I still don't think it's a good idea to bring bits and pieces of boost into RCF.

We strive to write code without compiler warnings, but at times, most libraries cause various compiler warnings outside of our control. I guess it's not a big deal to disable yet another compiler warning.

Strange. For some reason, v1.31 was ok with just defining RCF_USE_BOOST_ASIO. BOOST_WINDOWS was automatically defined under Cygwin. With 2646, it's still complaining that I need to define USE_WIN32_SOCKETS.

RCF 2.0.2648 is still broken on Cygwin. I've spent half a day investigating and here's what I found.

It appears that Cygwin uses a modified version of Boost (I'm using 1.48). Boost may have configured it wrong. Cygwin corrects it and then releases it as a package.

For example, in socket_types.hpp (v1.48) line 20, the official version of Boost had this:
#if defined(BOOST_WINDOWS) || defined(__CYGWIN__)

Cygwin's v1.48 line 20:
#if defined(BOOST_WINDOWS)

In light of this, I think to make RCF Cygwin compatible, you'd probably want to grab Boost from Cygwin rather than the official release.

At this stage, even with RCF_USE_BOOST_ASIO defined, there's still a lot of instances where you've included files from your own Boost.Asio (see *.ipp). If that's all sorted out, then there's Boost.Thread still to go.

Bringing Boost into RCF was a bad idea from the very beginning. Now that you've dropped support for external Boost, support for Cygwin is broken as well.

With RCF_USE_BOOST_ASIO, RCF is intended to use external Boost.Asio only. However I've had a look at the code, and as you pointed out, there are still several RCF headers referencing the internal Boost.Asio headers. I've fixed those includes now - if you could have a go with this update, that would be great:

Probably this line caused a compiler error in a cygwin regression testing environment at some point in time, and was therefore ifdef'd out. Feel free to comment out the ifndef - it won't have any negative effects.

Hi Jarl:
You can't imagine how exciting I am when I get here. Your work can help me to solve a puzzle where I struggled for several weeks. Thanks a lot.
There is still one question that I hope you can help me out.
How can I use RCF to send files from server to client?
Cause I've got this really big file like 500MB that i need to transfer from server to client. The file contains data serialized by protocol buffer.
If I can't send files, what's the best solution you recommanded?
I hope that you can help me. Thank you so much!

RCF 2.0 has built in support for file transfers, and will be released soon. RCF 2.0 will be available for download from the RCF website (http://www.deltavsoft.com), once it is released.

You can also transfer files using RCF 1.3, but you will need to implement it yourself by splitting the file into chunks and sending the chunks separately, and then reassembling the chunks on the other side.

Great work, but I have one question: I have rather large vector of objects, and I receive this error: Client-side message length error. Incoming message length: 2526715. Max allowed message length: 1048576.
How do I set max length of the message?
Thanks