JActor

When multiple requests must be sent to other actors in the course of processing a request, the code can become quite muddled by the use of anonymous classes used to implement the callbacks for processing the responses of those requests. This is true even in the simplest of cases. Consider the following example, that builds on the examples in an earlier blog post, JActor API Basics.

import org.agilewiki.jactor.*;

import org.agilewiki.jactor.lpc.*;

public class Test extends JLPCActor {

public void start(final RP rp) throws Exception {

Printer printer = new Printer();

printer.initialize(getMailbox());

(new Print("a")).send(this, new RP() {

(new Print("b")).send(this, new RP() {

(new Print("c")).send(this, rp);

});

});

}

}

The above code just sends 3 requests in succession, with each request sent only after completion of the previous request. The code is already a bit muddled and difficult to refactor. And as the application logic increases in complexity, the code grows disproportionally more complicated.

Fortunately the performance of JActor is good enough that requests with non-trivial processing requirements can often be refactored into multiple request with much simpler processing requirements. But this is not always a reasonable approach. This is where the use of simple machines comes into play.

A simple machine is little more than an array of operations that are typically executed in sequence. Building a simple machine does add a degree of complexity to the code, but the code remains unmuddled and easy to refactor.

import org.agilewiki.jactor.*;

import org.agilewiki.jactor.lpc.*;

public class Test extends JLPCActor {

public void start(RP rp) throws Exception {

Printer printer = new Printer();

printer.initialize(getMailbox());

SMBuilder smb = new SMBuilder();

smb._send(printer, new Print("a"));

smb._send(printer, new Print("b"));

smb._send(printer, new Print("c"));

smb.call(rp);

}

}

The above code is about as complex as the code it replaces, but is a bit more readable and a lot easier to refactor. The benefits to using a simple machine in this case can be debate but as the application requirements become more demanding, the use of a simple machine becomes the clear winner.

Output

a

b

c

There are four parts to a simple machine:

An array of operations.

A program counter which holds an index to the next operation to be performed.

A map of labels/indexes for use by goto operations. And

A map of keys/responses for accessing the responses from requests that have been sent.

The second example implements a loop using a simple machine.

import org.agilewiki.jactor.*;

import org.agilewiki.jactor.lpc.*;

import org.agilewiki.jactor.simpleMachine.*;

public class Test extends JLPCActor {

int counter;

public void start(RP rp) throws Exception {

Printer printer = new Printer();

printer.initialize(getMailbox());

SMBuilder smb = new SMBuilder();

counter = 1;

smb._label("loop");

smb._send(printer, new ObjectFunc() {

public Object get(SimpleMachine sm) {

return new Print(""+counter);

}

});

smb._if(new BooleanFunc() {

public boolean get(SimpleMachine sm) {

counter += 1;

return counter < 6;

}

}, "loop");

smb.call(rp);

}

}

The above code defines a lable, a send operation and a FORTRAN-like logical if operation. As you can see operations can optionally be built with a callback, as this allows the use of current (runtime) values rather than values defined at the time an operation is built.

There are a number of problems with Java serialization and numerous alternatives have been developed. But my focus here is on a particular use case, databases, and a single issue, performance.

Databases generally work with very large byte arrays. This is because seek time is very slow compared to the data transfer rate, so working with larger byte arrays often results in a performance gain. This is true for both hard disks and Solid State Disks (SSD). On the other hand, deserializing very large byte arrays is very CPU intensive. So Java databases optimize the size of the byte arrays they use to balance between the performance of the disk and the performance of the CPU. Fortunately the price of RAM has dropped significantly over the years so large memory caches can be used to hold the deserialized data, which reduces both the need to repeatedly read and deserialize the same data. Unfortunately there is still a need to frequently reserialize the updated data, which is also CPU intensive, and write it back to disk because of the transactional nature of many databases.

Significant performance gains are achieved by not using Java serialization, but working more closely with the data, reading and writing the binary form of integers, floats, strings and the like. This is not difficult to do, especially as most databases work only with well-defined tables where all the data in a given column is of the same type. But still, when reading or writing the data to disk, the entire byte array must be deserialized and subsequently reserialized. Working directly with the binary data is much faster than using Java serialization. But this is still a CPU-intensive process. And the irony is that many database transactions only access or update a miniscule amount of data.

Now in an ideal world, we would only deserialize data as needed, and then only reserialize the data that has changed. Doing this will mean that we can work with much larger byte arrays, resulting is an overall improvement in Java database performance. The data structures for doing this efficiently may be somewhat complex, but that should not be an issue so long as the API is reasonable. The term I use for this technology is JID, or Java Incremental Deserialization/reserialization.

JASocket contains a number of commands, some are for there as an aid in managing the cluster and others are there to illustrate how they work. Here we look at the implementation of some of those commands to aid you in the implementation of your own.

ToAgent

The toAgent command is of particular interest as it is used to send commands to other nodes. The arg string consists of the node address (or resource name), the name of another command and [optionally] the arg string of that other command. This command removes the address from its arg string and creates an EvalAgent initialized with the remainder of the arg string. The EvalAgent is then shipped to the designated node.

The latencyTest command measures the time it takes to send a KeepAliveAgent to another node and get a response. This command has an optional argument--the number of times the request/response is to be performed.

The who command lists every operator logged in on any node in the cluster. The display includes the operator name, node where the operator is logged in, how long the operator has been logged in, the number of commands entered and how long it has been since the last command.

Agents are initialized with their own mailbox. And unless the agent overrides the async method, their mailbox will be an async mailbox.

Agents which have been shipped to another node are initialized with the AgentChannel that delivered them as their parent. Otherwise the parent is the AgentChannelManager, in which case the isLocal method returns true.

The isLocalAddress method can be used to determine if a given node address is for the node where the agent is currently executing.

The JAFactory actor binds actor type names to actor factories, allowing type names to be used in place of class names for serialization/deserialization. There are 3 reasons for this:

Security--When deserializing, only those actor factories bound by an ancestor actor can be used.

Flexibility--A level of indirectness allows the use of different factories in different subsystems over time. And

Configurability--Different type names can be bound to the same actor class with different configurations by passing configuration data (meta data) in the factory's constructor or by using different factories. And this meta data need not be serialized.

The JLPCActor.initialize(Mailbox mailbox, Actor parent, ActorFactory factory) method is used by JAFactory to initialize the actors it creates. And the JLPCActor.getActorType() method returns either the name of the actor type or null if the actor has not been assigned an actor factory.

All actor factory classes, which are used to create, configure and initialize one type of actor, must extend the ActorFactory class:

package org.agilewiki.jactor.factory;

import org.agilewiki.jactor.Actor;

import org.agilewiki.jactor.Mailbox;

import org.agilewiki.jactor.lpc.JLPCActor;

abstract public class ActorFactory {

public final String actorType;

public ActorFactory(String actorType) {

this.actorType = actorType;

}

abstract protected JLPCActor instantiateActor()

throws Exception;

public JLPCActor newActor(Mailbox mailbox, Actor parent)

throws Exception {

JLPCActor a = instantiateActor();

a.initialize(mailbox, parent, this);

return a;

}

}

When binding an actor type to an actor class, JAFactory uses a default actor factory:

package org.agilewiki.jactor.factory;

import org.agilewiki.jactor.lpc.JLPCActor;

import java.lang.reflect.Constructor;

final public class _ActorFactory extends ActorFactory {

private Constructor constructor;

public _ActorFactory(String actorType, Constructor constructor) {

super(actorType);

this.constructor = constructor;

}

protected JLPCActor instantiateActor()

throws Exception {

return (JLPCActor) constructor.newInstance();

}

}

All of JAFactory's methods are thread-safe. This is because JAFactory uses a ConcurrentSkipListMap<String, ActorFactory> to bind type names to actor factories. And there are no Request classes defined to call methods of JAFactory, so you must call the methods directly or via one of the static methods that have been provided.

To define an actor type without an actor factory class, use the method defineActorType(String actorType, Class clazz). Alternatively, the method registerActorFactory(ActorFactory actorFactory) can be used when there is an actor factory class.

The static method JAFactory.getActorFactory(Actor actor, String actorType) returns an ActorFactory, where actor is either a JAFactory or has a JAFactory ancestor. But if no actor factory is found for the given actor type, then an IllegalArgumentException is thrown.

A number of static convenience methods are provided which return a new Actor:

Here we cover the basic API of JActor, which is very easy to use as you will see from the examples provided. --Bill

JLPCActor

Actors extend the JLPCActor class. The methods of an actor need not be thread-safe, as they are always called from the appropriate thread.

Actors interact by sending requests. But before sending or receiving any requests an actor must be assigned a mailbox, which manages its input and output queues.

Actors methods are of two types: synchronous and asynchronous. Asynchronous methods can use an exception handler and can send messages to other actors, while synchronous messages can not.

Asynchronous methods are distinguished by the presence of an RP parameter, a callback which is used to return the result of method. And the return value of an asynchronous method is always void.

Actors which use the same mailbox can call each other's synchronous methods. This is because they always share a thread.

Use the initialize(mailbox) method to assign a mailbox to an actor. Or use the initialize(mailbox, actor) method to also inject a dependency. Actors with an injected dependency "inherit" the request processing abilities of the injected actor, recursiveSo when an inappropriate type of request is sent to an actor, it will actually be sent to the appropriate injected dependency.

The getMailbox() method returns the mailbox assigned to an actor and the getMailboxFactory() method returns the factory used to create mailboxes.

The getParent() method returns the injected dependency, or null. The getAncestor(actorClass) returns the dependency which implements the class provided, or null.

Mailbox

A mailbox has an inbox of incomming messages (requests and responses) that have been sent to the various actors which have been assigned to it, as well as an outbox of messages for each target actor to which those actors are sending messages. (Outgoing messages are organized into message blocks prior to sending them as a means of increasing message throughput.)

Mailboxes are tasks which are assigned to a thread from a threadpool when a message is sent to a mailbox with an empty inbox.

Most mailboxes also support commandeering, which allows the thread of an actor sending a message to process the message directly. But commandeering can only be done when the mailbox of the target actor has not already been assigned to a thread.

Messages are normally only sent when the last message in the inbox has been processed, but calling the sendPendingMessages() method forces all the messages in the outboxes to be sent.

The setInitialBufferCapacity(int) method is used to set the initial capacitity of new outboxes, which otherwise defaults to 10. (Outboxes are implemented as array lists.)

The isEmpty() method returns true when the inbox holds no additional messages that need processing.

The getMailboxFactory returns the factory used to create mailboxes.

JAMailboxFactory

The JAMailboxFactory is responsible for managing a thread pool and for creating mailboxes. It has a convenience constructor with an int parameter that specifies the number of threads in the thread pool.

The close() method closes the thread pool.

The createMailbox() returns a new mailbox that can be commandeered.

The createAsyncMailbox() returns a new mailbox that can not be commandeered. This is used to force message processing of all the actors assigned that mailbox to process the message on a thread that is not the same as the thread used by the actor which sent the message--providing the source and target actors have not been assigned the same mailbox.

The addClosable(Closable) and removeClosable(Closable) methods add and remove Closable objects from a list. When the close() method is called, the Closable.close method is called on all of the objects in this list.

The first time the timer() method is called, a java.util.Timer object is created. All calls to this method then return the same Timer object. And when the close() method is called, it in turn calls the Timer.cancel() method.

Request

All requests passed between actors must extend the Request class, a generic class that takes 2 parameters: RESPONSE_TYPE, which is the class of the result returned when the request has been processed successuflly, and TARGET_TYPE, which a class that the target actor is an instance of.

The isTargetType(targetActor) is an abstract class that must return true when the targetActor is an instance of TARGET_TYPE. (Subclasses of Request must implement this method.)

The processRequest(targetActor, rp) is an abstract class that typically calls a method on the target actor. (Subclasses of Request must implement this method.) For asynchronous actor methods, the rp parameter is simply passed to the actor's method. But for synchronous actor methods, the rp.processResult(result) method must be called with the result returned by the actor's method. (If a synchronous method on the actor has a return type of void, then a result of null must be returned.)

(Note that the processRequest methos is always called from an appropriate thread so as to maintain the thread safety of the target actor. Typically this is the same thread being used by the actor which is sending the message, but not always.)

The send(sourceActor, targetActor, rp) method is used for sending a 2-way request message from one actor to another, with rp being a callback for handling the result. (The rp.processResponse message is always called from an appropriate thread so as to maintain the thread safety of the source actor.)

The send(jaFuture, targetActor) method is used for sending a 2-way request message from non-actor code to an actor. This method waits until it receies a result, which is then returned.

The sendEvent(sourceActor, targetActor) method is used to for sending a 1-way request message from one actor to another.

The sendEvent(targetActor) method is used for sending a 1-way message from non-actor code to an actor.

Getting Started

Download Jactor-4.5.0.zip from here. (The JActor version will change—current version is 4.5.0.) Then extract the jactor-4.5.0.jar file and copy it to a directory, GettingStarted. You will also need some slf4j jar files.

Here is the main method which creates a Test actor and sends a Start request to it.

import org.agilewiki.jactor.*;

public class GettingStarted {

public static void main(String[] args) throws Exception {

// Create a mailbox factory with a pool of 10 threads.

MailboxFactory mailboxFactory =

JAMailboxFactory.newMailboxFactory(10);

// Create and initialize a Test actor.

Mailbox mailbox = mailboxFactory.createMailbox();

Test test = new Test();

test.initialize(mailbox);

// Send a Start request and wait for completion.

JAFuture future = new JAFuture();

Start.req.send(future, test);

// Shut down the thread pool.

mailboxFactory.close();

}

}

The Start request is a singleton, as it has no parameters and is impermiable. The result returned is always null, so the RESPONSE_TYPE is Object. And the TARGET_TYPE is Test. The Start request calls the synchronous method Test.start().

import org.agilewiki.jactor.*;

import org.agilewiki.jactor.lpc.*;

public class Start extends Request<Object, Test> {

public static final Start req = new Start();

public void processRequest(JLPCActor targetActor, RP rp)

throws Exception {

Test a = (Test) targetActor;

a.start();

rp.processResponse(null);

}

public boolean isTargetType(Actor targetActor) {

return targetActor instanceof Test;

}

}

The Test actor sports a single synchronous method, start.

import org.agilewiki.jactor.lpc.*;

public class Test extends JLPCActor {

public void start() throws Exception {

System.out.println("Hello world!");

}

}

Output

Hello world!

Calling a Synchronous Method on Another Actor

There are times when one actor can directly call the synchronous methods of another actor. For example, when initializing the other actor or when both actors use the same mailbox.

Below we have modified the Test actor to create a Greeter actor which shares the same mailbox and then print the greeting returned by the Greeter.greet() method.

import org.agilewiki.jactor.lpc.*;

public class Test extends JLPCActor {

public void start() throws Exception {

Greeter greeter = new Greeter();

greeter.initialize(getMailbox());

String greeting = greeter.greet();

System.out.println(greeting);

}

}

The Greeter actor has only a single synchronous method, greet.

import org.agilewiki.jactor.lpc.*;

public class Greeter extends JLPCActor {

public String greet() throws Exception {

return "Hello world!";

}

}

Output

Hello world!

Sending a Request to Another Actor

Instead of the Test and Greeter actors using the same mailbox, we will have them use different mailboxes, and have the Test actor send a Greet request to the Greeter actor. But as only asynchronous methods can send messages, we will need to change the parmeters of Test.start, which means changing the Start request as well.

The change to the Start request is minor--we only need to change the processRequest method.

import org.agilewiki.jactor.*;

import org.agilewiki.jactor.lpc.*;

public class Start extends Request<Object, Test> {

public static final Start req = new Start();

public void processRequest(JLPCActor targetActor, RP rp)

throws Exception {

Test a = (Test) targetActor;

a.start(rp);

}

public boolean isTargetType(Actor targetActor) {

return targetActor instanceof Test;

}

}

Changes to the Test actor are a bit more interesting. In addition to creating another mailbox for initializing the Greeter actor, we must use a callback to receive the value returned by the Greet request.

import org.agilewiki.jactor.*;

import org.agilewiki.jactor.lpc.*;

public class Test extends JLPCActor {

public void start(final RP rp) throws Exception {

Greeter greeter = new Greeter();

greeter.initialize(getMailboxFactory().createMailbox());

Greet.req.send(this, greeter, new RP<String>() {

public void processResponse(String greeting)

throws Exception {

System.out.println(greeting);

rp.processResponse(null);

}

});

}

}

We also need to define the Greet request, a singleton which calls the synchronous method Greeter.greet() and then sends back the returned value.

import org.agilewiki.jactor.*;

import org.agilewiki.jactor.lpc.*;

public class Greet extends Request<String, Greeter> {

public static final Greet req = new Greet();

public void processRequest(JLPCActor targetActor, RP rp)

throws Exception {

Greeter a = (Greeter) targetActor;

rp.processResponse(a.greet());

}

public boolean isTargetType(Actor targetActor) {

return targetActor instanceof Greeter;

}

}

Output

Hello world!

Composing Actors with Dependency Injection

Dependency injection can be used to compose cactus stacks of actors, with actors able to send messages that will be processed by the appropriate ancestor. (In a cactus stack the links point towards the root, which is the reverse of a tree.)

We will define a simple Printer actor which has a Print request, and then inject this actor into a modified Greeter actor which prints its own greeting.

import org.agilewiki.jactor.*;

import org.agilewiki.jactor.lpc.*;

public class Test extends JLPCActor {

public void start(final RP rp) throws Exception {

Printer printer = new Printer();

printer.initialize(getMailbox());

final Greeter greeter = new Greeter();

greeter.initialize(getMailbox(), printer);

(new Print("Greeting:")).

send(this, greeter, new RP() {

public void processResponse(Object rsp)

throws Exception {

Greet.req.send(Test.this, greeter, rp);

}

});

}

}

Test now creates and initializes a Printer actor, and then creates a Greeter actor initialized with the Printer actor as its parent. A Print request is sent to the Greeter actor and, on completion, a Greet request is sent.

import org.agilewiki.jactor.*;

import org.agilewiki.jactor.lpc.*;

public class Greeter extends JLPCActor {

public void greet(RP rp) throws Exception {

(new Print("Hello world!")).send(this, this, rp);

}

}

The Greeter actor now sends a Print request to itself, passing the rp parameter on so that on completion of the Print request the Greet request also completes.

import org.agilewiki.jactor.*;

import org.agilewiki.jactor.lpc.*;

public class Printer extends JLPCActor {

public void print(String value) throws Exception {

System.out.println(value);

}

}

The Printer actor is quite simple, having a synchronous method which prints the value that is passed.

import org.agilewiki.jactor.*;

import org.agilewiki.jactor.lpc.*;

public class Print extends Request<Object, Printer> {

private final String value;

public Print(String value) {

this.value = value;

}

public void processRequest(JLPCActor targetActor, RP rp)

throws Exception {

Printer a = (Printer) targetActor;

a.print(value);

rp.processResponse(null);

}

public boolean isTargetType(Actor targetActor) {

return targetActor instanceof Printer;

}

}

Print is the first request that we have looked at which is not a singleton. This is because the Print request must hold the value to be printed.

Output

Greeting:

Hello world!

Forcing Parallel Operations

Whenever possible, messages are passed and processed on the same thread. This achieves the highest performance most of the time. But if you need to have an actor which operates on a separate thread, you need only assign it an asynchronous mailbox.

import org.agilewiki.jactor.*;

import org.agilewiki.jactor.lpc.*;

public class Test extends JLPCActor {

public void start(final RP rp) throws Exception {

Timer timer1 = new Timer();

timer1.initialize(

getMailboxFactory().createAsynchronousMailbox());

Timer timer2 = new Timer();

timer2.initialize(

getMailboxFactory().createAsynchronousMailbox());

final long t0 = System.currentTimeMillis();

RP prp = new RP() {

boolean pending = true;

public void processResponse(Object obj) throws Exception {

if (pending) pending = false;

else {

System.out.println(System.currentTimeMillis()-t0);

rp.processResponse(null);

}

}

};

(new Delay(1000)).send(this, timer1, prp);

(new Delay(1000)).send(this, timer2, prp);

}

}

The modified Test actor above creates two Timer actors, initializes each of them with its own asynchronous mailbox. And then sends a Delay request to each of them, but with the same callback, prp.

The prp callback receives the responses from the two Timer actors and, on receipt of the second response, prints the elapsed time and sends back a null response to the Start request.

import org.agilewiki.jactor.*;

import org.agilewiki.jactor.lpc.*;

public class Delay extends Request<Object, Timer> {

public final int ms;

public Delay(int ms) {

this.ms = ms;

}

public void processRequest(JLPCActor targetActor, RP rp)

throws Exception {

Timer a = (Timer) targetActor;

a.delay(ms);

rp.processResponse(null);

}

public boolean isTargetType(Actor targetActor) {

return targetActor instanceof Timer;

}

}

The Delay request calls the synchronous Timer.delay(ms) method.

import org.agilewiki.jactor.*;

import org.agilewiki.jactor.lpc.*;

public class Timer extends JLPCActor {

public void delay(int ms) throws Exception {

Thread.sleep(ms);

}

}

The Timer actor simply blocks the thread it is assigned to for a number of milliseconds.

Output

1003

The total elapsed time to run two timers in parallel for 1000 milliseconds each was 1003 milliseconds. And the reason for the additional 3 milliseconds is due to the inherent latency of passing messages between threads.

Continuations

When a request is received, it comes with an RP continuation. Calling the processResponse(rsp) method returns the response to the actor that sent the request in a thread safe way.

Continuations can be saved and a response sent at a later time. The only restriction here is that when sending a response it must be done within the context of the actor that received the request. (Or another actor with the same mailbox.)

import org.agilewiki.jactor.*;

import org.agilewiki.jactor.lpc.*;

public class Test extends JLPCActor {

public void start(final RP rp) throws Exception {

Greeter greeter = new Greeter();

greeter.initialize(getMailbox());

Greet.req.send(this, greeter, new RP<String>() {

public void processResponse(String greeting)

throws Exception {

System.out.println(greeting);

rp.processResponse(null);

}

});

System.out.println("trigger...");

Trigger.req.sendEvent(this, greeter);

}

}

The Test actor creates and initializes the Greeter actor, sends a Greet request, prints "trigger...", sends a Trigger event, and then, on receipt of the response to the Greet request, it prints the greeting.

import org.agilewiki.jactor.*;

import org.agilewiki.jactor.lpc.*;

public class Trigger extends Request<Object, Greeter> {

public static final Trigger req = new Trigger();

public void processRequest(JLPCActor targetActor, RP rp)

throws Exception {

Greeter a = (Greeter) targetActor;

a.trigger();

rp.processResponse(null);

}

public boolean isTargetType(Actor targetActor) {

return targetActor instanceof Greeter;

}

}

The Trigger request simply calls the synchronous method Greeter.trigger(). There is nothing special about this request to make it an event request. Only it is used as an event by the Test actor which does a Trigger.sendEvent rather than a Trigger.send.

import org.agilewiki.jactor.*;

import org.agilewiki.jactor.lpc.*;

public class Greeter extends JLPCActor {

private RP rp;

public void greet(RP rp) throws Exception {

this.rp = rp;

}

public void trigger() throws Exception {

rp.processResponse("Hello world!");

}

}

The Greeter class saves the continuation on receipt of a Greet request. And on receipt of a Trigger request it uses that saved continuation to send back the requested greeting.

The JASocket package makes it easy to create distributed, scalable software. JAConfig, in turn, makes it easy to manage in production.

The Config DB

JAConfig provides a fully replicated non-transactional, eventually consistent, key/value pair database for maintaining both configuration data and operator passwords. The database also provides change notifications, so servers can react to configuration changes. Every node in the cluster has a copy of this database both on disk and in memory, ensuring that the database is fully robust and supports fast queries. And there is neither a separate log file nor any need for a recovery mechanism--on startup, if the database is not valid its contents are discarded.

The underlying assumption of the database is that changes are infrequent, and that the system clocks of all the nodes in the cluster all have roughly the same time. Key/value pairs in the database always carry the timestamp of when the last change was made. Changes are propagated across all nodes in the cluster and shared when a node connects to another node. For each key, the change with the latest timestamp is retained.

The database is peer-based. So there is no single point of failure. And because there is no master copy, there are no warm or hot backups and fallover time is effectively 0. On the flip side, status information is completely out of scope, as frequent updates will break the underlying assumptions.

Quorum

A cluster can be split into 2 or more smaller clusters by something as simple as a loose cable. If these smaller clusters act independently, inconsistent results can occur. This is managed by knowing the total number of host computers in the cluster and only allowing some activities to occur the the number of hosts currently connected to a given cluster is equal to or grater than (totalNumberOfHosts / 2) + 1. Clusters connected to this number of hosts have what is called a quorum and as there can not be two sub-clusters with a quorum of hosts, at most only one sub-cluster will be active.

Note that we are talking about a quorum of hosts rather than a quorum of nodes. Multiple nodes can run on each host and indeed there may be a large host which runs many of the nodes in a cluster. So if the quorum was based on nodes, it is possible that all the nodes of the quorum are running on the same host, which creates a single point of failure.

Cluster and Host Server Managers

At this time there are two types of server managers, the Cluster Manager and the Host Manager. These managers are started (and monitored) by a kingmaker server, which runs in every node. The kingmaker servers are responsible for having one cluster manager running in the cluster and one host manager running on every host.

The cluster manager uses the data in the config database to start and monitor a number of cluster servers, where each type of cluster server has only a single instance running somewhere in the cluster. The cluster manager and all the cluster servers stop running when the node is not a part of the active cluster (a cluster with a quorum of hosts).

Similarly, the host managers use the data in the config database to start and monitor a number of host servers, where each type of host server has only a single instance running on each host. Unlike the cluster manager and servers, the host manager and servers are unaffected by quorum considerations.

Load Balancing

The cluster and host managers use a server named ranker to determine which node to use when starting a server. A simple ranker is provided which provides a list of nodes ordered by the number of servers running on each node. Alternative ranker implementations can be used as they are developed.

Java Incremental Deserialization/reserialization (JID) provides a near-ideal solution for updating serialized data structures. On an Intel I7, an entry can be inserted in the middle of the byte array of a serialized list with 100,000 entries in 2.4 * 10^-4 seconds (240 microseconds or about a quarter of a millisecond). And an entry can be updated in the byte array of a serialized map with 100,000 entries in 4.8 * 10^-4 seconds (480 microseconds or about half a millisecond).

To achieve this level of performance, JID is not reflection based, nor does it support cyclic data structures. JID instead requires that serializable data structures be built with objects which are instances of the Jid class. Each type of object used in a serializable data structure must also be registered.

Download JActor-4.5.0.zip and JID-2.0.4.zip from here. (The versions will change--the current versions are 4.5.0 and 2.0.4.) Then extract the JActor-4.5.0.jar and JID-2.0.4.jar files and copy them to a directory, GettingJidStarted. You will also need some slf4j jar files.

The j.bat file can be used to compile and run a test with the following command:

j className

The test is comprised of a single file in the GettingJidStarted directory, GettingStarted.jave, with a single method, main.

import org.agilewiki.jactor.factory.JAFactory;

import org.agilewiki.jid.JidFactories;

import org.agilewiki.jid.scalar.vlens.actor.RootJid;

import org.agilewiki.jid.scalar.vlens.string.StringJid;

public class GettingStarted {

public static void main(String[] args) throws Exception {

.

.

.

}

}

Factories

Factories are integral to the operation of JID, as they are needed for deserialization. (JID uses factory objects to create Jid objects, with each factory assigned a type name. See JActor Factories for more information.) Our main method begins with initializing the factories.

JAFactory factory = new JAFactory();

(new JidFactories()).initialize(factory);

JAFactory is the repository of factory objects. JidFactories, when initialized, adds a number of useful Jid factory objects to JAFactory when initialized.

Creating and Serializing an Empty RootJid

Jid objects are used to create tree structures, with the root of the tree always an instance of class RootJid.

The rootJid0 object is created and initialized by the JAFactory.newActor method.

The RootJid.getSerializedLength method returns the length of the byte array needed to hold the serialized RootJid. This method involves a minimum of calculation using information that is updated when the contents of the RootJid is updated. And the length is zero when a RootJid is empty.

The RootJid.save method takes two arguments, (1) the byte array where the serialized data is to be saved and (2) an offset. The returned value is the sum of the offset and the length of the serialized data.

The RootJid.load method is used to load a RootJid with the serialized data created by the method RootJid.save. The load method takes 3 arguments, (1) the byte array holding the serialized data, (2) the offset to where the serialized data is located in the byte array and (3) the length of the serialized data. The returned value is the sum of the offset and the length of the serialized data.

Serializing a RootJid with an Empty String

A RootJid object can hold one Jid object.

rootJid0.setValue(JidFactories.STRING_JID_TYPE);

serializedLength0 = rootJid0.getSerializedLength();

serializedData0 = new byte[serializedLength0];

updatedOffset0 = rootJid0.save(serializedData0, 0);

if (!(rootJid0.getValue() instanceof StringJid))

throw new Exception("unexpected result");

if (updatedOffset0 != serializedLength0)

throw new Exception("unexpected result");

The RootJid.setValue method is used to create and initialize the Jid object held by a RootJid.

Message passing between threads and callbacks can both make exception handling more difficult. On the other hand, the use of 2-way messages (request/response) provides us with a natural default: uncaught exceptions should be passed back to the requesting actor for handling. And that is exactly what JActor does.

JActor supports an exception handler for each request received by an actor, though exception handlers should only be used in asynchronous actor methods, e.g. methods with an RP continuation parameter.

The JLPCActor class provides two methods for working with exceptions: void setExceptionHandler(ExceptionHandler) and ExceptionHandler getExceptionHandler(), where the value null is used to indicate that there is no excepton handler and the default should be used.

If an exception occurs when an exception handler is processing an exception, the new exception is passed back to the actor which sent the current request for handling.

And when a request is sent as an event (Request.sendEvent) and the target actor does not handle an exception, the stack trace of the exception is printed. The printing is done in the JAMailboxFactory.eventException method, which can easily be overridden if a logger is to be used instead of printing.

(The following example presumes that you are familiar with the earlier blog post, JActor API Basics.)

import org.agilewiki.jactor.*;

import org.agilewiki.jactor.lpc.*;

public class Test extends JLPCActor {

public void start(final RP rp) throws Exception {

ThrowsException throwsException = new ThrowsException();

throwsException.initialize(getMailboxFactory().createMailbox());

setExceptionHandler(new ExceptionHandler() {

public void process(Exception exception)

throws Exception {

System.out.println(exception.getMessage());

rp.processResponse(null);

}

});

ThrowException.req.send(this, throwsException, new RP() {

public void processResponse(Object response)

throws Exception {

System.out.println("No exception");

rp.processResponse(null);

}

});

}

}

The Test actor now creates and initializes the ThrowsException actor. Test also defines an exception handler for the current (Start) request and then sends a ThrowException request to the ThrowsException actor.

import org.agilewiki.jactor.*;

import org.agilewiki.jactor.lpc.*;

public class ThrowException

extends Request<Object, ThrowsException> {

public static final ThrowException req =

new ThrowException();

public void processRequest(JLPCActor targetActor, RP rp)

throws Exception {

ThrowsException a = (ThrowsException) targetActor;

a.throwException(rp);

}

public boolean isTargetType(Actor targetActor) {

return targetActor instanceof ThrowsException;

}

}

The ThrowException request is a singleton that calls the asynchronous method ThrowsException.throwException(RP).

import org.agilewiki.jactor.*;

import org.agilewiki.jactor.lpc.*;

public class ThrowsException extends JLPCActor {

public void throwException(RP rp) throws Exception {

throw new Exception("Boo!");

}

}

The ThrowsException actor has a single asynchronous method for throwing an exception.

The Node class provides a default configuration for a node in the cluster and also has a main method for running a simple node, e.g. a node without a console or any initial servers. Configuring a node is a matter of writing a subclass of Node, overriding the various defaults set by Node and adding servers that be started when the program runs. SSHServer and HelloWorld are examples of classes with a main method that creates a node and then create a SSHServer or HelloWorld server respectively.

Here then is the first part of the Node class, including all its public methods:

Note in particular the call to the Node constructor in the main method. The argument provided to the constructor is the number of threads to be created. A 100 is probably excessive. In addition to the threads needed to run the application logic (figure 1 per hardware thread plus 1 for every blocking actor), JASocket requires 1 thread for its Timer and an additional thread for each channel.

The startup method is also worth noting, as it can be called from main to start a server.

Node Directory

Every node has its own unique directory. To ensure this, the directory is given a name based on the cluster port:

This max size is then set for all the sockets, and also specifies the size of the ByteBuffer to be used.

Multicasting

Multicasting is used for discovery and a number of items need to be specified:

protected void startDiscovery() throws Exception {

new Discovery(

agentChannelManager,

NetworkInterface.getByInetAddress(InetAddress.getLocalHost()),

"225.49.42.13",

8887,

2000);

}

The most interesting is the network interface. Generally you can use the network interface which supports the local host IP address. But if you have multiple internet connections, e.g. more than one eithernet card, or if you are using a VPN, you may need to specify something different.

The IP address given above is just an arbitrary IP address in the range of multicast IP addresses. Each IP address (and port number) specifies a multicast group. You must specify a different address or port for each cluster.

The last parameter passed to the Discovery constructor is the delay between broadcasts, in milliseconds. When a node first starts up it sends out a UDP multicast packet. This packet is then rebroadcast repeatedly after the given delay.

To disable discovery, simply override this method with one which does not create an instance of Discovery.

Keep Alive

To use keep alive's, just call the startKeepAlive method on the agent channel manager:

protected void startKeepAlive() throws Exception {

agentChannelManager.startKeepAlive(10000, 1000);

}

The first parameter specifies how long a socket can be idle before being closed, in milliseconds.

The second parameter specifies the delay between keep alive messages, in milliseconds. These are only sent on channels which are inactive.

Keep alive's are important for detecting network failure, but may not be helpful when the entire cluster is running on a single computer. To disable their use, simply override this method with one which does not call startKeepAlive on the agentChannelManager.

Operator Authentication

Operator names/passwords can be authenticated. But straight out of the box, JASocket does not provide any authentication. Note that if you do provide authentication, operator names must not be allowed to contain spaces.

EvalAgent looks for a command name at the beginning of the arg string that it is passed, instantiates the command agent and initializes it with the remainder of the arg string before passing a request to it.

Loops are usually just a while loop with a hasNext() method in the condition and a next() method in the body. But when the response to these methods is asynchronous the loop fails, because the response is not received until after the while loop has completed.

Recursion will work nicely with asynchronous responses even for long loopsStack overflow is not even an issue because the recursive call is made after the loop has exited in the callback that handles the response. Of course you can not use this technique for long loops when the responses are synchronous because of stack overflow.

Message passing in JActor is done synchronously (everything running on the same thread) as much as possible for maximum performance. So it is entirely possible for some responses to be synchronous and, in the same loop, for other responses to be asynchronous.

Case in point, you are looping over a cache. When the next item is in the cache, the response will be synchronous. But when the next item is not in the cache the response will be asynchronous.

The JAIterator class handles both synchronous and asynchronous responses. We will first look at an example that builds on the examples in an earlier blog post, JActor API Basics, before reviewing its implementation.

import org.agilewiki.jactor.*;

import org.agilewiki.jactor.lpc.*;

public class Test extends JLPCActor {

public void start(RP rp) throws Exception {

Printer printer = new Printer();

printer.initialize(getMailbox());

final int max = 5;

(new JAIterator() {

int i = 0;

protected void process(RP irp) throws Exception {

if (i == max) irp.processResponse("done");

else {

i += 1;

(new Print(""+i)).

send(Test.this, printer, irp);

}

}

}).iterate(rp);

}

}

JAIterator is always subclassed, as you must provide a process method that is called with an irp parameter until a non-null response is returned by calling irp.processResponse. Calling the JAIterator.iterate(rp) method then executes the loop, with the response returned to the rp object being the non-null response returned to the irp object.

JAIterator employes a while loop to iterate as long as the responses are synchronous, with an asynchronous response resulting in what looks like a recursive call. So regardless of the modality of the response, the memory footprint is kept small.

Note that in the above, the second argument passed by main to the create method is null. This means that the console will not support user interrupts. Unfortunately, user interrupt processing requires sun JRE-specific methods. So we handle user interrupts in a separate class, IntCon. If using a different JDK, you will need to replace IntCon entirely rather than subclass it: