Service-Oriented System Integration

Boomi's cluster architecture exploits the dynamic nature of Jini networking
to provide high availability and failover. Boomi defines system
integration services with Java interfaces, and registers
implementations of those services in Jini lookup services on the
cluster.

Interaction between services starts by a client performing a Jini
service lookup based on the required service's type. Services can be
deployed on the cluster in a redundant fashion. Since each service
implementation registers a proxy that implements the service's Java
interface, the client can retrieve and use any implementation's
proxy. If a node fails, or is powered down, the service
implementations on that node will not renew their lookup service
registrations, causing the cluster's lookup services to purge those
service proxies. Since a client's access to a service proxy is also
based on leases, the client will have to locate a new service's proxy
from the Jini lookup services. Dynamic service location and
lease-based access to resources inherent in Jini networking thus provide the foundations of
Boomi's high-availablity and failover infrastructure.

In Boomi's system integration architecture, a layer of transport
services represents the gateway between the Boomi server and
external data sources. Transport services act as I/O for the Boomi
system. For instance, HTTP transport occurs via an HTTP-specific
transport service, FTP transport through an FTP transport service, and
input into a database is arranged via a database transport.

As messages enter the system, they are placed into a queue managed by
a queue service. The queue service acts as a message delivery
system, and is responsible for persisting data received from various
transports to guarantee message delivery. Thus, the queue service is
used by all other services. The queue service relies on a relational
database to persist message data.

Figure 3. Data-flow parallelism on a cluster in support of system integration

While the database may form a single point of failure, in practice
Boomi's system is deployed on a clustered database. All incoming data,
as well as interim processing results, are saved in the database. When
the queue service restarts after a crash, it first looks in the
database for yet unprocessed data, and invokes services responsible
for processing those messages.

Messages in the queue are eventually processed according to message
type, message origin, and rules corresponding to additional message
parameters. Such processing occurs by process services that
not only transform data, but also route data to destinations specified
by rules. Thus, process services form the core of the system
integration functionality.

Boomi's process services implement about fifteen system integration
patterns, such as data transformation, intelligent routing, decision
validation, exception handling, or dynamic messaging. For each process
service, users specify via a graphical rules designer what input data
to expect, what transformation actions to take on what message types,
and where results should be routed.

Events in Boomi's system occur not only in response to arriving
messages, but also in an automated fashion induced by a Jini
scheduling service. This service is analogous to the Unix
cron daemon, and fires off events at specified intervals. General
administrative responsibilities, in turn, are performed via a Jini
administration service that controls configuration-level
properties, such as where the database is located, and provisions
security access. The administration service is also responsible for
logging.

Finally, Boomi's Jini archiving service takes data out of the
live data set maintained by the queue, and puts that data on a
secondary storage device for historic preservation. For instance, the
archiving service may pull data out of the database if a data item is
older than a specified number of days, preventing the live database
from becoming too large.

When a service boundary is crossed, Jini discovery, lookup, and remote
method invocation facilitates service interaction. As an example,
suppose that data needs to be read from a directory, transformed to a
different format, and then written out to the directory again. In this
example, the transport service reads the data in from the directory,
and hands that data off to the queue service. The queue service
persists the data in the database management system so that the data
has guaranteed delivery: If the system fails at any point, that data
item can be resent. Next, the queue service sends that data to the
process service, which transforms the data to the desired format. The
process service then places the data back into the queue. Lastly, the
queue hands the results back to a transport service, which then writes
out the data to the directory.

In this scenario, the queue service must find the administration
service to obtain information about the system's database, and to
download the needed JDBC driver. The queue service also has to locate
transport services to send and retrieve data, and process services to
hand data to for processing. As well, the queue service finds the
archive service and passes to it data that is ready to be
archived. Once the data is in the archive service, the archive service
might seek additional archive services on the network in order to
replicate the data.

Figure 5: Service interaction in Boomi's integration cluster

Boomi's cluster-based service deployment relies on the open-source
Jini Rio project. Rio defines the cluster's desired operation in a
declarative fashion. A Rio cluster configuration might specify how
many instances of the process or transport services should be
running. Rio's runtime infrastructure monitors the cluster and tries
to ensure that desired cluster state. Suppose, for instance, that an
administrator requests that four instances of the HTTP transport
service be running. If a cluster node executing an HTTP transport
service crashes, Rio starts a new HTTP transport service instance on
an available node.

Boomi's administration, schedule, and queue services are considered
singleton services: while they are deployed to cluster nodes, only one
instance runs at a given time. On the other hand, the Boomi process
service that does most of the heavy lifting on incoming data, is often
clustered. As well, clustering the archive service provides multiple
backup mechanisms across different cluster nodes. That way, archived
data is replicated across multiple machines.