7.0 Legion programming

Legion is designed to support and allow interoperation between multiple programming models. At its base, Legion prescribes the message format for interobject communications, thereby enabling a wide variety of Legion object implementation strategies. In its most useful form Legion presents programmers with high-level programming language interfaces to the system. These high-level interfaces are supported by Legion-targeted compilers, which in turn use the services of a runtime library that enables Legion-compliant interobject communication.

We have implemented a Legion run-time library (LRTL) [36, 8], and we have ported the Mentat [11] programming language compiler (MPLC) to use the LRTL.1 You can thus define Legion object implementations in MPL, which can be translated by MPLC to create executable objects that communicate with one another in a Legion-compliant fashion. Figure 14 shows this programming model.

Figure 14: The Legion programming model.

The object is defined in a high-level language. Through a combination of Legion-aware compiler support and use of the LRTL, a complete Legion object implementation is produced. Based on this object implementation, new Legion objects can be instantiated in the system. These new objects can communicate with one another and with existing Legion objects.

7.1 LegionLOID

The runtime library provides classes for LOIDs, Object Addresses (OAs), and bindings. A majority of library users will deal with naming only at the LOID level, leaving OAs and bindings to the low-level library code. Therefore this section discusses only the LegionLOID class, which manages all of the LOIDs in a Legion system.

The LOID is represented in the library by the LegionLOID class, which is intended to be the starting point for derived LOID classes implementing different types of LOIDs. We imagine that the classes implementing each LOID type will enforce different properties having to do with the structure, content, and meaning of the various fields of the LOID. Therefore, LegionLOID is not intended to be directly instantiated when the internal fields of the LOID are being used and interpreted: the appropriate derived class should be instantiated instead. So far we have not implemented any derived classes that enforce special properties of the LOID fields, so for the time being the LegionGeneralPurposeLOID class manipulates all of the LOID fields without enforcing any structure or special meaning to the fields.

The code for the make_loid() function is presented below as a demonstration of how to build a general purpose LOID (Example E). The function takes a string which represents the object's class identifier and an integer which represents the object's instance number as its parameters. It builds a LOID which names an object in the domain UVaL_LegionDomain_Virginia (a library-defined constant), with class identifier and instance number assigned to the values passed as parameter, and with an empty public key. The function returns a LRef to a LegionLOID instance.

Example F shows some of the functionality of LOIDs (assuming that the function make_loid() exists).

Example F:
The functionality of Legion LOIDs

LegionLOID *a, *b, *c, d;
// Use the function defined above to create two new
// LOIDs
a = make_an_loid("Class Name", 0);
b = make_an_loid("Class Name", 1);
// The copy constructor is overloaded
c = new LegionLOID(*a);
// == and != operators are overloaded
if ((*a == *c) && (*a != *b))
printf("== and != are overloaded\n"):
// A special EmptyLOID type is defined. This is
// useful for sending empty LOIDs as parameters and
// return values
if (d.is_empty())
printf("LOIDs are empty if they have not been
initialized\n");
// LegionLOIDs are packable
LegionBuffer buffer;
b->pack(buffer);
buffer.seek(BEGINNING, 0);
d.unpack(buffer);
f (*b == d)
printf("LOIDs are packable.\n");
// is _class() indicates whether or not an LOID refers
// to a Legion class object
if (a->is_class())
printf("If the class identifier of an LOID is
empty\n",
" then the LOID refers to a Legion
class.\n");
if (!b->is_class())
printf("If not, then the LOID doesn't refer to a
Legion class.\n");
// Finally, LOIDs can be printed neatly, showing
// only the class id and instance number fields
fprintf(stderr,"LOID b: ");
b->show();
fprintf(stderr,"\n");

7.2 Basic module and data container: the Legion buffer

A Legion buffer, represented by the C++ object class LegionBuffer, is the fundamental data container in the Legion Library. LegionBuffer exports operations to read and write data from and to a logical buffer. Instances of different kinds of LegionBuffers export the same interface and perform the same basic function but can have different characteristics. For example, a LegionBuffer instance may copy the data it contains into heap-allocated memory, another may maintain pointers to the data, and a third may read and write its data from and to a file. Instance may also choose to compress or encrypt data, or both. To define its characteristics, each LegionBuffer instance contains:

LegionStorage

LegionPacker

LegionEncryptor

LegionCompressor

MetaData

7.2.1 LegionStorage

The LegionStorage class determines how data are stored. Different LegionStorages have unique performance and operation characteristics. One type, provided in the Library, is LegionStorageScat. It stores its data as a linked list of pointers to chunks that it allocates, or to maintain pointers to data "owned" by other parts of the Library. LegionStoragePersistent stores data in a file. So, while it might be efficient to select a LegionStorageScat that does not copy its data, extra care must be taken to avoid deleting data out from under the buffer and leaving dangling pointers.

Although LegionStorage exports functions in order to access a buffer's data directly (see page 133 in the Reference Manual for a list of LegionBuffer interfaces), these functions typically should not be called directly by a buffer user. Instead, the user should call the functions in the interface provided by the LegionPacker portion of the buffer.

7.2.2 LegionPacker

A LegionPacker determines the data format conversion operations, if any, that are performed on the contained data when they are written to and read from the buffer. LegionPacker is the primary Library mechanism for dealing with the different data storage formats of different machine architectures (e.g., big vs. little endian, 32-bit vs. 64-bit words) that need to communicate with each other. The Library supports three different equivalence classes of architectures: Alpha, Sparc, and x86. For efficiency reasons, Legion assumes a "receiver makes right" data conversion policy [39], in which a message sender (a creator of a LegionBuffer) packs the message in its own native format and the message receiver is responsible for converting the data to the format appropriate for the architecture upon which the message now resides. The Library provides six different types of packers of the form LegionPackerX2Y, where X and Y are two different members of {Sparc, Alpha, x86, etc.}. None of these six packers do any conversion when writing to the buffer, but each converts data in an suitable way when reading from the buffer. LegionPackerX2Y is appropriate for use when the data is stored in format X and the machine currently holding the buffer is format Y. When the data is already stored in the correct format for the architecture upon which it is being stored or when data conversion is done outside of a LegionBuffer, LegionPackerDefault can be used to ensure that no data conversion takes place on reads from the buffer.

The interface provided by a LegionPacker consists of the functions of the form

put_zzz(zzz *source, int how_many)

and

get_zzz(zzz *source, int how_many)

where zzz names a basic C++ data type (i.e. char, short, ushort, int, long, ulong, float, or double). Thus, a LegionPacker exports put_char(), get_char(), put_short(), get_short(), etc.; source points to an array of how_many instances of type zzz; and the put_zzz() function copies this data into the buffer after first performing the acceptable data conversion operation, if necessary for the type of LegionPacker that is instantiated. The get_zzz() function fills the complimentary role; first converting the data and then copying it into source.

The code example below (Example G) illustrates the use of the LegionPacker interface of a LegionBuffer. It also uses the seek() function -- which is actually part of the LegionStorage interface (please see page 133 in the Reference Manual) -- to "rewind" the logical position within the buffer back to the beginning.

Example G:
Using the LegionPacker interface

// Use the default constructor to declare a new empty
// LegionBuffer, which will be configured to contain
// the default storage, packer, encryptor, and
// compressor
LegionBuffer buffer;
// Declare and initialize data to write into the
// buffer
char *in_string = "Hello World";
int in_int_array[5] = {100, 101, 102, 103, 104};
// Insert the string
buffer.put_char(in_string, 11);
// Insert the integers
buffer.put_int(in_int_array, 5);
// Insert a single char
buffer.put_char(&in_string[6], 1);
// Insert a single int
buffer.put_int(&in_int_array[3], 1);
// "Rewind" the buffer back to the beginning so we
// can read out the data we just wrote in
buffer.seek(BEGINNING, 0);
// Declare data structures to read the buffer data into
char out_string[12];
int out_int_array[5];
char out_char;
int out_int;
// Data must be read out in the same order it was put in,
// but not necessarily the same way.
// Read 1st 6 chars
buffer.get_char(out_string, 6);
// Read the next 5
for (j = 6; j < 11; j++)
// one at a time
buffer.get_char(&out_string[j], 1);
// Read the integers
buffer.get_int(out_int_array, 5);
// Read the single char
buffer.get_char(&out_char, 1);

7.2.3 LegionEncryptor

A LegionEncryptor determines the encryption and decryption algorithms, if any, that are applied to the data. The Library currently provides LegionEncryptionDefault, which defines empty encryption and decryption operations.

7.2.4 LegionCompressor

A LegionCompressor determines the compression and decompression algorithms, if any, that are applied to the data. The Library currently provides LegionCompressionDefault, which defines empty compression and decompression operations.

7.2.5 MetaData

Eight bytes of metadata, three of which are currently used, are associated and carried with each LegionBuffer. The metadata indicates the format in which the data is stored, and the algorithms, if any, that were used to encrypt and/or compress the data. The metadata fields and values that are supported by the Library are defined in the Reference Manual (section 7.4).

7.2.6 Packability

LegionBuffers enable the concept of "packable" classes in the Library. A class is packable if it exports the following functions:

pack(LegionBuffer &lb)

and

unpack(LegionBuffer &lb)

Both pack() and unpack() take a single reference parameter, which names a LegionBuffer. The pack() function of a packable class writes its state into the buffer so that the unpack() function of the same class can read it out. The state is typically written to and read from the buffer using the LegionPacker part of a LegionBuffer interface, which encapsulates the data format conversion operations.

Classes are made packable for two reasons: so that they can be passed between heterogeneous architectures within a LegionBuffer and written to a LegionBuffer as part of a "save state" operation. Since these two operations are common in the Library, many parts of the Library operate only on packable classes. For instance, many of the templated data structures are packable and require the ability to call the pack() member function of the contained data. Each function parameter in a Legion object method invocation is also passed within a LegionBuffer, so the easiest and best way to allow an object to be a function parameter is to make its class packable.

Making a class packable -- implementing pack() and unpack() functions for a class -- is generally quite easy. The LegionBuffer exports storage operations for primitive C++ types. For complex types, such as when class X contains an instance of another packable class, Y, X's pack() function can simply contain a call to Y's pack() function. Thus, if Y is packable X does not need to know what data types Y contains in order to pack Y as part of its state. Consider the simple example of a templated array class in Example I. Note that the Array class can be made packable, even though it doesn't know the type of the elements it contains. The class only requires that the contained elements are themselves packable.

LegionBuffer is itself a packable class. Thus, one LegionBuffer can be contained (packed) in another. This is shown at the top of the next example, Example J.

If data is packed as a LegionBuffer it should be unpacked as one. Thus the data that was packed in the top of Example J cannot be unpacked correctly, using the code in the middle of the example. This code will compile and run, but it will not have the desired effect of unpacking the ten characters -- "Hello World" -- that were packed into the buffer. This is because LegionBuffer instances prepend "user data" with metadata. Therefore, hello_world_buf contains metadata at the beginning of the buffer as well as between "Hello" and "World." The correct way to unpack the data is shown in the bottom of the example.

LegionBuffers pack and unpack their bytes raw (without data format conversion), so that each buffer maintains its own metadata. This means that a LegionBuffer created on an x86 architecture can be contained in a LegionBuffer whose data is on an Alpha machine. When the contained buffer is unpacked, a LegionBuffer with appropriate data conversion operations will be instantiated: when the bytes are read out, the data will wind up in the correct format for the architecture of the machine upon which the data resides.

7.3 Legion invocations and messages

7.3.1 Contents of a message

Legion objects communicate with each other via method invocation and return values. To invoke methods and return results, Legion objects send messages to one another in a standard Legion message format.

A Legion message can carry:

Part (or all) of a method invocation, or

The function return value that resulted from an invocation, or

A return value for an out or in/out parameter.

Every Legion message contains, in order:

Source and destination LOIDs

A function identifier

The number of parameters to expect

A computational tag

A list of parameters

A continuation list

An environment.

Source LOID: The source LOID identifies who sent the message.

Destination LOID: The destination LOID identifies the object to which the message is being sent.

Function identifier: The function identifier field should contain a C++ object, packed in the data format of the architecture of the machine from which it was sent. If the message is intended to implement a method invocation, the packed C++ object should match some function identifier in the public interface of the object to which the message is being sent. If the message is the "filled-in" value of an out or in/out parameter, or a normal return value, the packed C++ object should be the constant LEGION_RETURN_FUNCTION_IDENTIFIER.

Computational tag: A single method invocation can be split up into several Legion messages which can come from different sources (see Legion program graphs). A computation tag is a long integer, packed in the message sender's data format, that uniquely identifies a computation or method function invocation within Legion for the duration of the invocation's existence. The computational tag should be assigned by the invoker. Messages that carry function return values and filled-in out and in/out parameters should contain the same computational tag as the messages that carried out the invocation which generated the results. When awaiting results, the invoker matches the computation tag.

Although a computational tag is simply a long integer, the Library provides a C++ class called LegionComputationTag, which encapsulates the integer and exports appropriate functions on computational tags. The Library also provides a class called LegionComputationTagGenerator, which can be used to generate random computational tags. Example K shows typical use of these two classes.

Parameters to expect: This field should contain an integer, packed in the sender's data format, that indicates the total number of parameters being passed in the message function invocation. If the message is part of a return value, this field is ignored.

Parameter list: If the message is part of an invocation, the parameter list contains the values of the parameters contained in the message. The parameter list is packed as an integer that indicates how many parameters are present, followed by the parameters themselves. Each parameter contains an integer that indicates the number of the parameters, followed by a LegionBuffer that contains the value of the parameter. Return values are passed back in a parameter list as well. The C++ object classes LegionParameterList and LegionParameter implement parameters. The code in Example L builds a parameter list that contains two parameters, an integer, and a 14-element character string.

Continuation list: A continuation list describes the location to which the results of a particular computation should be forwarded. A continuation contains a computation tag and result number, which together identify a return value. It also contains the LOID, function identifier, and parameter number to which the result should be sent. The motivation for continuation lists arises from the macro data-flow programming model that is the initial Legion target. In this model, results from method invocations are sent directly to other invocations that use these results as parameters. These data dependencies are determined through analysis of the program code (see [13] for more information). LegionProgramGraphs represent these data dependencies (and are described in section 7.3.5). For further information, refer to the Legion on-line documentation at <http://legion.virginia.edu/documentation.html>.

The LegionMessage class implements Legion messages in the Library. Its most useful constructor takes parameters that correspond to all of the constituent parts described above. Example M shows sample creation of a LegionMessage instance.

Although LegionMessage provides a mechanism for implementing method invocation in the Library, LegionProgramGraph provides a higher level abstraction that is simpler to use.

7.3.2 Legion message database

7.3.2.1 Legion Work Unit

Each Legion object services requests for member function invocations and returns the results of these invocations. The parameters to these functions, as well as the requests themselves, arrive in Legion messages: a complete method invocation may involve composing a number of Legion messages. Conceptually, the Legion message database is divided into two parts. The "bottom" part, called the Invocation Matcher, manages a list of partially complete method invocations for the Legion object: since messages may arrive from different locations and at different times, some parts of message will arrive before other parts. The "top" part of the database, the Invocation Store, maintains two separate lists. The first list contains complete method invocations (i.e., requests with a complete parameter set and security clearance). The second list contains return values that the object has received as a result of its own method invocations on other Legion objects. Once a complete method invocation has been assembled, it becomes a Legion Work Unit, and is promoted to the Invocation Store.

Each object has a server loop that continuously checks the invocation store for ready work units, extracts them as they become available, and performs the requested invocation. A partial interface to the C++ class LegionInvocationStore is given in Example N.

Example N:
Selected elements of the LegionInvocationStore interface

// This class implements the "database" for ready
// work units and stores the results from method
// invocations on other Legion objects
class LegionInvocationStore
{
public:
// Accept function invocations for the given function
int enable_ function (int function_identifier);
// Check to see if there are any ready work units
int any_ready();
int any_ready_for_func(int function_identifier);
// Remove the next work unit from the store
LRef<LegionWorkUnit> next_matched();
LRef<LegionWorkUnit>
next_matched_for_func(int function_identifier);
}

7.3.2.2 Method invocation

(Information here does not account for varying security levels. For information on how security affects invocations, please contact the Legion Resource Group.)

Suppose Legion object A (the invoker) invokes a member function on Legion object B (the invokee). This method invocation proceeds through the invoker and invokee, as shown in Figure 15. The invoker builds a Legion message containing salient information about the member function invocation. Typically, the Legion program graph interface builds this message. The Legion message is then passed to the Legion message layer, which binds the LOID of the recipient to a particular address. The binding process is a key aspect of Legion, and is described in more detail elsewhere (see The binding mechanism). The outcome of the binding process is a <LOID, Legion Object Address list> tuple called a binding. This represents the logical name and current physical addresses of the referenced object. The message and its binding are then passed to the data delivery layer, which linearizes the message for transport over the wire, uses the object addresses to create a physical connection to the referenced object, and sends the message.

Figure 15: Method invocations.

The path through the Legion library for a method invocation that returns a result to the caller. Below the dotted line is considered the Legion library's domain. Application code must interface with the library at the four marked points.

On the receiving end, the data delivery layer of the destination object unpacks the data into a new LegionMessage instance, and passes the message up to the message layer. The message layer then inserts the message into a Legion message database. When the return result reaches its destination, it is handled like any other Legion message until it reaches the Invocation Store, which examines the work unit's contents and realizes that it is a return result, not a method request. The result is then inserted into the return result list, and from there it is available to the original invoking object through the program graph interface

7.3.3 Basic Legion event kinds & handlers

The Library is implemented as a configurable protocol stack. A layer of the stack communicates with other layers through an event mechanism. The basic idea is straightforward: if layer A wishes to communicate with layer B, A announces a LegionEvent. Each LegionEvent has a tag which denotes a LegionEventKind (what kind of event it is). Each LegionEventKind has one or more associated event handlers that may be called whenever an event of its kind is announced. Handlers for a particular LegionEventKind are given a priority to determine the order in which the handlers are called. Since a LegionEvent can carry arbitrary data, this is the method by which data is passed and transformed from layer to layer. If layer B has registered a handler for the kind of event that layer A announces, layer B will get that event. (See Implementing the configurable protocol stack: events.)

7.3.4 Invocation execution and result return

The Library announces a MethodReady event each time a ready method invocation request is inserted into the invocation store. Each request is maintained as a Legion Work Unit. The general algorithm for getting a LegionWorkUnit out of the database and invoking its requested method is as follows:

Remove the work unit from the invocation store

User code can remove work units from the invocation store in at least two different ways, each of which requires a server loop that continuously checks the invocation store for ready work units. One mechanism is to supply and register an event handler for MethodReady events. A simplified version of the code for the handler and the server loop look like that in Example O (top). The accompanying server loop is quite short. The other mechanism, in the bottom of the example, does not require the user to supply an event handler. The user instead checks the invocation store each time, through a longer server loop.

Example O:
Two mechanisms to get a ready work unit out of the invocation store

Once a work unit is removed from the invocation store, it needs to be unpacked to a form suitable for method invocation. There must be specific code to do this for each public method in the invoked object. Example P contains an example invocation construction. The sequence is functionally the same as server stubs in an RPC. First ascertain the requested method and then remove each parameter from the work unit, based on the particular requested method. Each parameter is returned as a LegionBuffer, which must be unpacked in order to get the actual parameters for the method invocation. Once this is done, the method can be called like any C++ member function.

For methods that have return results, these values must be sent to the list of objects defined in the LegionContinuationList part of the work unit from which the method invocation was constructed. The most straightforward way to do this is as follows: for each return result, allocate a new LegionBuffer and insert the return value into the buffer, then call Legion_return() with the buffer, continuation list, and number of the return value as arguments. This sequence is illustrated at the bottom of Example P.

Example P:
An example of method construction, invocation, and return, once a work unit has been removed from the invocation store. Other code structures are possible.

A complete example of a C++ class, its translation into the appropriate Library calls, and some sample method invocations are in the Reference Manual starting on page 122.

7.3.5 Legion program graphs

A program graph represents a set of method invocations on Legion objects and the data dependencies between those invocations and objects. The program graph is a data-flow graph whose nodes represent method invocations and whose arcs represent data dependencies between the method invocations.

Figure 16: Sample code for an invoking object (left) and the corresponding program graph (right).

Suppose that objects A and B each export methods op1() and op2(). Figure 16 shows a simple user program and the resultant data dependencies. It is clear from the code that the parameters to both A.op1() and B.op1() are available locally. Those are the constant parameters. The parameters to A.op2() are not available locally, since they are the result of method invocations that have been executed elsewhere. These are the invocation parameters.

The most straightforward mechanism for making an invocation request is to build a program graph, using the interface provided by the C++ object class LegionProgramGraph. Salient parts of the LegionProgramGraph are in Example Q. A fuller description of the interface constituents is in the Reference Manual (page 130).

Now we can show the necessary library calls to implement the sample code in Example R below.

Start-up: The call to Legion.init() initializes various data structures in the Library. Legion.AcceptMethods() is called because the invoking object may itself be accepting member function requests from other objects.

Object creation: The calls to Legion.CreateObject() create the two objects of interest and return LOIDs to these objects. Local handles for the objects are created based with these LOIDs.

Member function invocation: For each method, we use the object's local handle to create an invocation.2 We can then add the invocation to the program graph using add_invocation(). Every added invocation becomes a node in the program graph. To create arcs parameters must first be packaged into instances of LegionParameter (see Example L). Once packaged, they are added to the graph using add_constant_parameter(). Internal arcs in the graph must be handled differently, because they represent values that are not locally available (they have not yet been computed). Internal arcs are added using add_invocation_parameter(). Once a program graph is constructed, the execute() member function must be called. Calling execute() causes every node in the program graph to be packed up as a Legion message and shipped to the appropriate object for execution. Results from this remote execution then become available and are automatically sent to the objects that require them. In Example R, the return values from A.op1() and B.op1() are forwarded directly to A, so that they can become the parameters to A.op2().

Example R:
Sample user code (top left), the corresponding program graph (top right), and the library calls needed to implement it (bottom). In this case, make_parameter() takes an integer, wraps it up in a LegionBuffer, then wraps the buffer in a LegionParameter.

Return results: Getting return values that are results of method invocation requests is straightforward. The LegionProgramGraph class has a method called get_value(), which takes the parameter number of the result value as one of its arguments. If the result is unavailable, get_value() will block. You can pass the UVaL_METHOD_RETURN_VALUE constant to get_value() to get the return value of the function call. By default, the return values of all method invocations are returned to the invoker. For in/out parameters, add_result_dependency() (not shown) must be used to explicitly ask for the parameter to be returned. This call must be made before execute() is called on the program graph that contains the associated method invocation. Otherwise the parameter will not be returned and a call to get_value() for that parameter will block indefinitely.

7.4 Reference counting

The template class LRef, in concert with the class LRefCntr, is the Library's mechanism for automatic reference counting and safe dynamic memory management. The mechanism is intended for heap allocated C++ objects. It keeps track of references to each object that is shared by different parts of the library and automatically deletes the object when all meaningful references to it have disappeared. Each reference counting object -- i.e., each instance of a class derived from LRefCntr -- maintains an integer that indicates the number of LRefs that point to that object. When a new reference points to an object the reference count within that object automatically increases. When an LRef is overwritten with another value, or when a local variable LRef falls out of scope the reference count in the object to which the reference points automatically decreases. When the reference count falls to zero the object is automatically deleted. All of this happens without any intervention by the programmer or user of LRefs.

The decision to include an automatic reference counting mechanism in the Library was motivated by two observations: memory copies are expensive and often hinder the performance of message passing code, and keeping track of shared pointers and deciding which parts of the code are responsible for deleting which chunks of heap-allocated memory is prone to error and is difficult to document effectively. It is hoped that the automatic mechanism will combine the better performance that comes from avoiding memory copies with the safety and correctness that comes from not having to worry about managing dynamically allocated memory. Obviously, the automatic reference counting mechanism introduces some overhead when compared to simple pointer copies, but we believe that the benefits will outweigh the costs.

To be a casual user of LRef, you need only remember one simple rule of thumb and two simple exceptions.

7.4.1 LRef rule of thumb

Read "LRef<X> t" as "x *t" and then treat the variable t exactly as if it were in fact a C++ pointer to class X.

Just about every operator that is legal on a pointer to a C++ object has been overloaded to work correctly for LRef. Example S shows that LRef can be used just as C++ object pointers would be. The implementation of the MyRCO class in Example S is unimportant beyond the fact that it is derived from LRefCntr and implements the functions that are used to illustrate the point.

7.4.2 Exceptions to the rule of thumb

Never delete the memory that a LRef points to. The memory will be deleted when all references to the memory have been overwritten or go out of scope.

Do not use a LRef alone as a boolean. "if (t)..." will not compile, but "if (t != NULL)..." and "if (!t)..." will work as expected.

LRef can also refer to non-heap-allocated memory, i.e. global and local variables. To insure that these objects are not automatically explicitly deleted by the mechanism, the programmer should call the function makeNonHeapReference() on the reference.

Example S:
Example declaration of a ReferenceCountingObject and its use with LRef.

// Definition and implementation of class MyRCO, which
// is a reference counting object by virtue of being
// derived from class LRefCntr
class MyRCO : public LRefCntr
{
private:
int contained_val;
public:
MyRCO() {contained_val = 0;}
MyRCO(int val) {contained_val = val;}
int set_value(int val)
{ return (contained_val = val);}
int get_value()
{return (contained_val);}
int operator==(myRCO &other_rco)
{return (contained_val ==
other_rco.contained_val);}
int operator!=(MyRCO &other_rco)
{return (contained_val !=
other_rco.contained_val);}
~MyRCO() {printf("Destructor called\n");}
};
// Create three new reference counting object, pointed to
// by variables a, b, and c. Notice that type (MyRCO *) is
// automatically cast correctly to type LRef<MyRCO>
LRef<MyRCO> a = new myRCO(1);
LRef<MyRCO> b = new myRCO(2);
LRef<MyRCO> c = new myRCO(3);
// Show that * and -> work just like pointers
a->set_value((*a).get_value()); // no change to the
// object
c = a;
// The object to which c originally pointed now has no
// more references to it. Therefore, that object's
// destructor will be called automatically. The object
// to which a points now has two references to it, a and c.
a = b;
// The object to which a pointed is not automatically
// deleted because c still points to it.
// All of the print statements below will be executed.
// Comparing objects is still different than comparing
// pointers
if (*a == *b) printf
("a and b refer to object whose vals are ==.\n");
if (*a != *c) printf
("a and c refer to objects whose values are
!=.\n");
if (a == b) printf
("a and b point to the same object.\n");
if (a != c) printf
("a and c do not point to the same object.\n");
// Make a and c point to objects that contain the same
// value
c->set_value(a->get_value());
if (*a == *c) printf
("Now a and c point to objs whose vals are ==\n");
if (a != c) printf
("a and c still do not point to the same obj.\n");

7.5 Exception propagation model

Exceptions are a standard structuring mechanism for building distributed robust applications, as they provide a mechanism for error detection and recovery. As the name implies, an exception has the connotation of a rare event, an assumption that no longer holds true in a wide-area distributed system such as Legion. Examples of possible common exceptions in Legion include:

Security violations: the invoker of an object does not have proper authorization

In designing an exception propagation model we follow the Legion minimalist philosophy of specifying mechanisms and not policies. Thus Legion emphasizes exception propagation and not exception handling. Our view is that exception handling is specific to a particular programming language and should not be part of Legion proper. Instead, Legion provides the mechanisms to enable a wide variety of exception handling models.

In Legion, the propagation of exceptions is specified in a generic manner with computation graphs (see Legion program graphs). Computation graphs can express a variety of exception handling policies, from traditional policies which propagate exceptions back to the caller (as in CORBA) to policies that propagate exceptions to third-party objects. Examples of the latter policy would be to propagate all security exceptions to a Security Monitor object or propagate all Communication errors to a Fault Detector object. For a detailed description of various policies please refer to "Building Robust Distributed Applications with Reflective Transformations" [26].

7.5.1 Standard exception propagation policy

The Legion Library comes with default functions for the common case in which users want to propagate exceptions back to the caller. To raise an exception, users must first create a LegionException instance. Exceptions are described by a Type field and a Subtype field.

LegionExceptionCatcherDefaultEnable() catches all exceptions and does not distinguish between types and subtypes. In the first example the application immediately terminates while in the second, the application can continue execution. Most of the Legion command line utilities use the first.

Interested readers may find an application of the Legion exception propagation model in the standard Legion distribution in the $LEGION/src/Examples directory. Furthermore, more complex policies are possible and are described in "Building Robust Distributed Applications with Reflective Transformations" [26].

7.6 How to use the run-time library

This section describes how to use the Library as is, with no internal modification. We begin by describing the LegionLibraryState class, which encapsulates start-up and initialization routines, and provides a public interface to an object's Legion related state. We then explain how a method invocation is implemented in the Library. Finally, we describe the Library interface from both the invoker and invokee perspectives.

The LegionLibraryState C++ object class provides an interface to important parts of an object's Legion-related state information. This includes the object's own LOID, the object's class LOID, and so on. This class also provides implementations of key object control mechanisms such as object creation, activation, deactivation and deletion. Finally, its interface provides the ClassOf() operation, which encapsulates the mechanism by which a given object's class LOID can be obtained via the object's LOID. We will now examine each of these general features of this class in more detail.

Call Legion.init()

The primary function of the LegionLibraryState object class is to encapsulate the internal state of the Legion library and to provide programmers with the public interface to this state. A simple global object of the class, named Legion, is included as part of the Library. Before using any of the Library features the Library state should be initialized with the init() function.

This function initializes several different parts of the Library, including the LegionClass LOID, the Legion program graph layer, and the Legion matcher and invocation store. After this call, the object can enable or disable functions in the invocation store, set its own LOID (if necessary), and perform any user initialization that must be performed before any methods are serviced (or any messages are sent or received by the object).

Call Legion.AcceptMethods()

The object cannot yet invoke or service methods, however. These services require a second initialization phase which is encapsulated in the class's AcceptMethods() member function of .

After making this call, the object can both accept and invoke methods. Consequently, basic object control mechanisms such as object creation and ClassOf(), which rely on the ability to invoke methods on remote objects, are also enabled. The reason for a two-phase initialization is simply to separate method service configuration from the enabling of method arrival events.

The Library initialization should proceed as follows:

Call Legion.init(). This initializes various data structures in the Library.

Enable acceptable methods (i.e., function identifiers) in the objects invocation buffering mechanism (the Legion invocation store). To accomplish this, call LegionInvocationStore. enable_function(int) for each method that the object will be exporting.

If methods will be serviced by an event handler, add this event handler for MethodReady events. This event handler should contain code to extract work units from the invocation store and call the appropriate method implementations based on incoming function identifiers.

Call Legion.AcceptMethods(). This call initializes the Legion message passing system and notifies the object's creator (its class) that the object is up and ready to accept method requests.

Start the server loop. The details of this depend on how work units are being extracted from the invocation store.

Once the Legion Library state is fully initialized, the object can use the Legion object to determine its own LOID, its vault's LOID, its LegionClass LOID, and so on. For example,

An object can use the LegionUtilityFunctions interface to report to its class when it plans to delete itself, without having been requested to do so by the class. For example, an object before exiting could execute:

Beyond object control services, the LegionUtilityFunctions class encapsulates the important Legion ClassOf() operation, which can be used to determine the LOID of a class object, based on one of its instance's LOID, or based on just a class identifier (that part of the LOID that indicates the object's class):

Unlike simple state accessors such as GetMyLOID(), object control methods such as CreateObject() and ClassOf() all result in Legion method invocations, and thus the cost of these member functions are non-trivial.

7.7 Modifying the Library

A major design object of Legion is an extensible system: making it easy for future implementors to insert modules into the Library. To accomplish this, the Library provides

A layered design and implementation, and

a standard mechanism for interlayer communication.

The layered design of the Library is depicted in Figure 17 (below). The client side, left, is the invoker (the code that is requesting a method invocation on some Legion object). The server side, right, is the invokee (the Legion object upon which the method invocation has been made). While it is convenient to think of the Library in terms of clients and servers, it is also artificial since full Library functionality is provided to both parties. In many cases, an object's role changes as execution progresses: sometimes clients are servers and sometimes servers are clients.

Figure 17: Layered design of the Legion library.

7.7.1 Implementing the configurable protocol stack: events

As Figure 17 illustrates, the Legion protocol stack supports a variety of functions, in order to allow modules to be easily added and configured, an approach similar to that used in the x-Kernel [16]. The problem with the traditional approach to building protocol stacks is that each layer in the stack explicitly calls the layer below or above. This static coupling makes it difficult to dynamically configure the stack.

To provide a dynamically configurable stack, we have chosen a well understood technology, events, and applied it to allow flexibility and extensibility [4]. There are four main classes that implement events:

LegionEvent

LegionEventKind

LegionEventManager

LegionEventHandler

When an event occurs, the system announces it to an Event Manager (a LegionEventManager instance). The event manager notifies interested parties (i.e., Legion Event Handlers) of the event (immediately or later).

Event handlers register themselves with a Legion Event Kind, an event template which contains a unique tag and a default list of handlers, in order to be notified of an event. Since there may be more than one event handler per event kind, a handler is given a priority when registering. In the current scheme, a handler with a lower priority number is executed before a handler with a higher number. Not all event handlers associated with a Legion event are guaranteed to be executed, since an event handler is allowed to prevent the execution of subsequent handlers.

The LegionEventKind class serves as a template for instance of the LegionEvent class (Example U). When an event is created it obtains a list of LegionEventHandler instances from its corresponding LegionEventKind. This allows users to modify the behavior of the protocol stack without having to change existing modules. To enable interlayer communication, Legion events may also carry arbitrary data, which can be updated, modified, and transformed in arbitrary ways by the event handlers processing each events.

A LegionEventHandler takes as its sole argument a reference to the Legion event that it is servicing. Thus, each LegionEventHandler associated with a particular Legion event may inspect and modify the data carried by the Legion event. In general, this is how information is shared between various handlers.

7.7.2 Interfaces

Users may add LegionEventHandlers to a LegionEventKind. When a Legion event is created, it obtains its unique event identifier and a list of suitable LegionEventHandlers from its corresponding LegionEventKind. LegionEventHandlers are ordered, and the handler with the lowest priority are executed first. An example of a LegionEventHandler is in Example V.

A Legion event maintains a logical pointer to the currently executing LegionEventHandler. This allows the event to suspend the execution of its event handlers and to resume that execution at a later time.

In particular, an event handler can:

Inspect and modify the data field of the incoming event

Inspect and modify the list of handlers contained in the incoming event

Create and announce new events

Prevent the next handlers from being executed

Save the current event

There are no restrictions on the code implementing a LegionEventHandler.

7.7.3 LegionEventManager

When users wish to notify the system that something of interest has occurred, they must announce a LegionEvent to a LegionEventManager (Example W). The LegionEventManager is responsible for deciding when to execute the handlers associated with an event. In the current implementation, there are two ways to announce events to an event manager: depending on the chosen method, the event manager will either invoke the handlers immediately or will defer the execution of the event handler and store the LegionEvent in an internal queue. The available methods are: the flushEvents() method, used to execute all pending events; the blockForEventAvailable() method, used to block the thread of control until there are some pending events available; and the serverLoop() method, which repeatedly calls blockForEventAvailable(), followed by flushEvents().

On the sending side, the program graph layer generates a LegionEvent_MethodSend event. The security handler LegionEvent_Can_I may disallow the remote method invocation [37]. If it doesn't, a following handler generates a LegionEvent_MessageSend event for each method invocation. Once the message has been successfully sent, the data delivery layer generates a LegionEvent_MessageComplete event.

On the receiving side, the data delivery layer will generate a LegionEvent_MessageReceive once it has successfully assembled a complete message. The LegionDefaultMessageHandler is the last handler for the event LegionEvent_MessageReceive and generates a LegionEvent_MethodReceive, once the invocation matcher has assembled a complete method invocation. The first handler for LegionEvent_MethodReceive is a security handler, and it implements access control on this object [37]. If the security handler grants access, the method invocation is deposited into the LegionInvocationStore and a LegionEvent_MethodReady event is generated.

7.7.5 Adding new functionality to the Legion protocol stack

To add functionality to the existing stack, users may either define a new LegionEventKind or register their own handlers with one of the predefined events kinds. The latter option is the simpler. Defining a new event kind consists of creating an instance of the class LegionEventKind with a unique identifier. For example:

LegionEventKind LegionEvent_Foobar (UniqueIdentifier);

Once the event kind has been defined, creating and announcing a LegionEvent can be done as shown below:

Adding a handler to an existing event kind is best illustrated through an example. Here, a user can add a security layer to encrypt outgoing messages and decrypt incoming messages. We first define the handlers encryptionHandler() and decryptionHandler(), and register them with the appropriate LegionEventKind (Example X).

// Declaration of the encryption handler
LegionEventHandlerStatus encryptionHandler
(LRef<LegionEvent> ev)
{
// extract the LegionMessage from the data
// field of the event, encrypt the message,
// allow the next handler to be called
returns TRUE;
}
// Declaration of the decryption handler
LegionEventHandlerStatus decryptionHandler
(LRef<LegionEvent> ev)
{
// extract the LegionMessage from the data
// field of the event, decrypt the message,
// allow the next handler to be called
returns TRUE;
}
// Register the handlers with the appropriate
// LegionEventKind
// The encryptionHandler should be the last handler
// before the message is sent by the data delivery layer
LegionEvent_MessageSend.addHandler(encryptionHandler, encryptionPriority);
// The decryptionHandler should be the first handler
// called after the message is delivered by the date
// deliver layer
LegionEvent_MessageRecv.addHandler(decryptionHandler, decryptionPriority);

For encryption, we would like the encryption handler to be the last handler called when sending a message. For decryption, on the other hand, we need to call the decryption handler first when a message is received. These ordering constraints are realized by registering encryptionHandler() with a high priority number and decryptionHandler() with a low priority number. The new protocol stack is shown in Figure 18.

7.7.6 Active messages

The active messages programming model [35] is a message passing scheme that is intended to integrate communication and computation in order to increase the compute/communicate overlap, thereby masking the latency of message passing and increased performance. The basic idea behind active messages is simple: messages are prepended with the address of a handler routine that is automatically invoked upon receipt of the message. Active messages are not buffered and explicitly received, as is common with standard message passing interfaces. Instead, the receiving process invokes the handler routine specified for the message immediately upon message arrival. The handler may execute as a new thread of control, or may interrupt the running computation. The job of the active message handler is to incorporate the received message into the on-going computation.

A Legion version of active messages could be constructed by making Legion methods serve as message handlers, and by replacing the Legion "method ready" event handler with one that creates a new thread to service incoming methods, instead of buffering them in an invocation store. Pseudo-code for such a method invocation handler is given in Example Y.

This method ready event handler would need to be registered with the method ready event kind. The code to do this might look like:

LegionEvent_MethodReady.addHandler(ActiveMessageMethodReady,1.0);

This line of code would need to be executed before any methods arrived at the object. This can be achieved by placing this line of code before any calls to Legion.AcceptMethods().

The effect of this new method ready event handler is to provide an active messages style programming model. In some ways, the model supported here is more general than the traditional active messages model. For example, if a method (i.e., a handler) required two messages from different sources for activation, this requirement would be enforced by the Legion invocation matcher. Programs might be entirely composed of standard single-token active messages, providing a programming model as flexible as the original. On the other hand, programs might also include multi-token active messages, for a more general programming model that might best be called "active methods."

7.7.7 Path expressions

The various method invocation semantics covered thus far have offered a "one size fits all" concurrency control mechanism. For example, the supported remote procedure call model allows exactly one method to be serviced at a time by a given object. The active messages approach, on the other hand, allows any number of operations of all types to be active at the same time in the same object. A more general approach to customizing the concurrency control requirements of operations on an object can be designed based on path expressions [5]. Path expressions permit the programmer to specify:

Sequencing constraints among operations,

selection between operations (mutual exclusion), and

allowable concurrency between operations.

These concurrency control primitives let programmers maintain the sequential consistency of their programs and at the same time indicate potential concurrency to a run-time environment.

Path expression based method sequence could be implemented for Legion objects, again by utilizing the inherent configurability of the Library's protocol stack. As with active messages, supporting a different method invocation semantic requires replacing the Legion method ready event handler. In this case, the method ready handler must examine the function identifiers of available operations and determine if they may be safely fired, given the ordering constraints specified by the program's path expressions. If a method can be safely fired, a new thread is created and allowed to run, starting at the entry point for the given member function (as in the active messages case). On the other hand, if the ordering constraints of a newly arrived method are not satisfied, the method must be buffered (e.g. in a library-provided invocation store) and later extracted and fired, when safe. This need to defer the firing of methods requires that code be executed whenever methods complete execution. One possible way to satisfy this requirement is to use LegionEvent_MethodDone event kind, and announce events of this kind when methods complete execution. A handler for this event kind can then be used to re-evaluate buffered methods with respect to the path expression ordering constraints whenever a running operation completes.

To examine the implementation of the scheme in more detail, we assume a path expression run-time support class, PathExpressionManager, that exports methods to specify the ordering, selection, and sequencing constraints of operations (i.e., Legion method function identifiers). This class would also support methods to determine if a given method is safe to fire, and to determine which (if any) methods are ready to be fired upon the completion of a running operation. The first modification we must make to the Library configuration is to add a new method event handler. It could look like Example Z.

This method handler would need to be registered with the Legion method ready event kind, as in the case of the active messages handler. This method handler would need to be registered with the Legion method ready event kind, as in the case of the active message handler. The other requirement of our path expression solution is that code be executed upon method completion in order to re-evaluate the safety of firing buffered methods. To accomplish this, we use method done events that must be announced whenever a method is finished running. A handler must be registered with the LegionEvent_MethodDone event kind that tries to fire any runnable buffered methods.

LegionEvent_MethodDone.add Handler(
PathExprMethodDoneHandler,0.0);

Finally, an event of this type would need to be announced upon the completion of each method by the object. The data for this event would need to be set to reflect the function identifier of the completed method.

The result of this configuration of the Library would be a run-time environment that could be used to support path expression style method invocation semantics. This run-time system might be used explicitly by a programmer or might be the target of a compiler that accepted a Path-Pascal-like implementation language for Legion methods.

Example AA:
A possible handler for MethodDone events in an implementation of path expressions

7.7.8 Message passing

Thus far, the programming models we have examined have been variations of an object-based method-invocation-oriented model. This is natural, given the object oriented nature of the Legion system. However, Legion can support alternative programming models, such as message passing or distributed shared memory. In this section, we examine the ways in which the Legion library can be configured to support a message passing model. We describe how Legion can be used as run-time support to implement a message-passing interface such as MPI [15] or PVM [32] (see the Basic User Manual for information about the Legion PVM and MPI libraries). These systems allow asynchronous send and receive operations. Messages are buffered until explicitly requested at the receiving process and send operations are permitted to return before the destination process has received the message.

One possible implementation of such a message passing system would be to construct a message-passing Legion base class that exports a single messageDeliver() method. The parameters to this method could be an integer message tag and an uninterpreted string of bytes containing the message. The operation of the send() library function would involve invoking the messageDeliver() Legion method on the intended destination LOID. The implementation of the messageDeliver() Legion method would accept and buffer the received message in an internal message queue. The receive() library function would then consist of a loop that could check the message queue for the desired message tag, dequeue and return it if available or block for a messageDeliver() invocation if not.

Although the above solution is simple to implement using the available library support mechanism, it has the potential drawback of incurring the cost of the mechanism associated with method invocation (e.g., token matching, additional event handlers, etc.) while only requiring support for message passing. An alternative implementation strategy is to insert a handler lower in the Legion Library protocol stack. The natural place for such a message passing handler would be at the Legion message receive event layer. Here, a handler could be inserted to capture a message with certain desired function identifiers (i.e., a special "message passing" function identifier), and enqueue the message contents on a message queue for use by the message passing library. Messages with other function identifiers (those associated with object mandatory methods [21], for example) would be allowed to continue up the protocol stack and through the normal method invocation mechanism. In this scheme, the send() library operation would construct a LegionMessage object containing the message contents and reflecting the appropriate agreed upon function identifier. This Legion message would be added to a LegionMessageEventStructure, which would be placed into a new LegionEvent of the kind LegionEvent_MessageSend. This event would be announced and the message would be sent. The receive() operation would simply loop, examining the message queue utilized by the message receive handler described above, and blocking for available events. Thus, the overhead of the standard Legion method invocation mechanism is avoided for low-level message passing traffic.

Although this scheme could improve performance, it has serious security ramifications. If messages are caught before the token matching process, the automatic MayI() method will not be invoked and the object's security may be compromised. While this may be acceptable for certain performance-critical, security-optional applications, the first message passing implementation described would be more suitable for balanced security/performance applications.

1.We have also developed implementations of PVM (see section 9.0 in the Basic User Manual) and MPI (see section 10.0 in the Basic User Manual) layered on top of the LRTL, and we currently support a Java interface to the LRTL and a specialized programming interface for Fortran called Basic Fortran Support (BFS; see section 4.0).

2.A Legion invocation identifies a particular invocation on one of an object's member functions. An invocation is obtained through a Legion Core Handle. Core handles export functions that allow programmers to ask for invocations and to obtain a description of the corresponding object's interface. A handle can be thought of as a local representation of an object and as a generator of invocations for that object.