Cascading: It's all done with mirrors

One of the features planned for version 2.0 of the JMX API
is cascading, also known as federation. Here's what it is, and
how you can build a simplified form of the same thing without
waiting for 2.0.

One of the features planned for version 2.0 of the JMX API
is cascading, also known as federation.
Here's what it is, and how you can build a simplified form of
the same thing without waiting for 2.0.

Update: a subset of the Java DMK[2] product has been released as open source[3]. Daniel Fuchs[4] explains how to use the Cascading API from Open DMK. I would recommend using this in production if you need Cascading.

Cascading

The basic idea behind cascading is that you can "import"
MBeans from one MBean Server into another MBean Server. In the
picture here, the top MBean Server (labeled Master Agent)
imports MBeans from two other MBean Servers (Subagent 1
and Subagent 2).

The different MBean Servers could be in the same Java VM, or more
likely they could be in different VMs, possibly on different
machines.

By "importing", I mean the Master Agent has a "mirror"
for each of the imported MBeans. This mirror shows exactly the
same MBean interface as the MBean it reflects. An operation on
the mirror is forwarded to the remote MBean.

So for example suppose the middle MBean in Subagent 1 has
attributes "Size" and "Capacity" and an operation "reset". Its
mirror in the Master Agent will have the same attributes and
the same operation. If I get the "Size" attribute from the
mirror, it will forward the request to the remote MBean, and
return the result it gets back. If I invoke the "reset"
operation on the mirror, it will forward that request to the
remote MBean, which will do the real "reset" operation.

The end result is that a client of the Master Agent (such
as JConsole) doesn't have to know about the other MBean Servers
at all. It can just interact with the mirror MBeans, and the
result will be the same as if it had interacted with the
corresponding MBeans in the other MBean Servers.

Some of the MBeans in the subagents might themselves be mirrors
for "subsubagents", so you could have a multilevel hierarchy.
This is where the name "cascading" comes from.

(Cascading has existed for years as part of Sun's
href="http://www.sun.com/software/jdmk/">Java Dynamic
Management Kit product (Java DMK), and you can read about
how it works there in the
href="http://docs.sun.com/app/docs/doc/816-7609/6mdjrf87n?a=view">tutorial.)

So what's it for?

There are several cases where cascading is useful.

The most obvious case is where you have a number of different
MBean Servers with interesting MBeans and you want to be able to
manage them all. You can do this by importing the MBeans into a
single MBean Server and attaching a management client like
JConsole to this MBean Server.

This is much simpler for the client than having to connect
separately to each MBean Server. It might even be that the
links between the Master Agent and the Subagents are over a
private network that is not accessible to the client, so it
couldn't connect directly to the Subagents even if it wanted
to.

A second case where cascading could come in handy is if you
don't want to expose all of the MBeans in an MBean Server to a
particular client. You can create a Master Agent that only
imports the subset of MBeans that you do want to expose, and let
the client connect to that.

A related case is where you want to give the MBeans different
names. There's no requirement that the mirror MBean have the
same name as the original MBean in the subagent. You can
construct a new MBean model by importing MBeans and giving them
different names. Of course, you don't have to import all the
MBeans, and you don't have to import them all from the same
place. So this is quite a general mechanism.

A simple implementation

Let's look at how we might implement a basic form of cascading.
The idea is to have a class MBeanMirrorFactory that
allows us to create mirror MBeans. After creating a mirror
MBean, we can register it in the Master Agent under whatever
name we've chosen.

To show how this works, suppose we want to create an MBean
Server that contains all the same MBeans as the Platform MBean
Server, but where every MBean's name starts with "mirror/". So
the MBean called "java.lang:type=Runtime" in the Platform MBean
Server will have a mirror called "mirror/java.lang:type=Runtime"
in the new MBean Server. Here's the code to do that usingMBeanMirrorFactory:

As another example, here's how we might set up a configuration
like the one in the picture above, except that all MBeans from
both subagents are imported. The MBeans from Subagent 1 will be
prefixed with "subagent1/" and the ones from Subagent 2 with
"subagent2/". So we will have mirror MBeans called
"subagent1/java.lang:type=Runtime" and
"subagent2/java.lang:type=Runtime", for example.

The attributes shown are those of the MBean
"subagent1/java.lang:type=ClassLoading" and are the same as we
would see if we attached directly to Subagent 1 and looked at its
MBean "java.lang:type=ClassLoading".

Mirror MBean implementation

The mirror MBean implementation is actually quite simple. The
same class can implement a mirror for any MBean. The secret is
that we do not have to know the management interface at compile
time. We can discover the interface at run time and implement a
href="http://weblogs.java.net/blog/emcmanus/archive/2006/11/a_real_example.html">Dynamic
MBean that forwards every operation on the mirror to
the remote MBean.

Here's a first attempt for the implementation class:

// First attempt. THIS DOES NOT COMPILE.public class PlainMBeanMirror implements DynamicMBean { private final MBeanServerConnection mbsc; private final ObjectName objectName;

Each of the six methods of the
href="http://java.sun.com/javase/6/docs/api/javax/management/DynamicMBean.html">DynamicMBean
is forwarded to the corresponding method in the
href="http://java.sun.com/javase/6/docs/api/javax/management/MBeanServerConnection.html">MBeanServerConnection
interface. The MBeanServerConnection methods have an extra
ObjectName parameter, which here is the name of the remote
MBean.

This is just a little too good to be true, and indeed if we try
compiling it we will get errors, because we haven't considered
exceptions carefully enough.

Exceptions

If we look at a method in the DynamicMBean interface, say
href="http://java.sun.com/javase/6/docs/api/javax/management/DynamicMBean.html#getAttribute(java.lang.String)">getAttribute,
and the
href="http://java.sun.com/javase/6/docs/api/javax/management/MBeanServerConnection.html#getAttribute(javax.management.ObjectName,%20java.lang.String)">corresponding
method in the MBeanServerConnection interface, we will see
that DynamicMBean.getAttribute throws
AttributeNotFoundException, MBeanException, and
ReflectionException. MBeanServerConnection.getAttribute throws
the same exceptions, but also InstanceNotFoundException and
IOException. It is the same story for the other five
DynamicMBean methods.

This makes sense. If we invoke getAttribute on a mirror MBean,
it will invoke getAttribute on the remote MBean. If that gets
AttributeNotFoundException, then the mirror MBean can simply
throw the same exception. But there are two other ways we could
get an exception. One is if the remote MBean does not exist.
The other is if we get a communication failure, for example
because the remote machine is not reachable.

So we need to rewrite PlainMirrorMBean.getAttribute to take
these extra exceptions into account. Happily, the exception
href="http://java.sun.com/javase/6/docs/api/javax/management/MBeanException.html">MBeanException
exists precisely to wrap these general sorts of exception. So
here's what the new version looks like:

We put in the throws clause the exceptions
declared by DynamicMBean.getAttribute. And we catch the two
other exceptions from MBeanServerConnection.getAttribute and
wrap them in MBeanException. We can handle
href="http://java.sun.com/javase/6/docs/api/javax/management/DynamicMBean.html#setAttribute(javax.management.Attribute)">setAttribute
and
href="http://java.sun.com/javase/6/docs/api/javax/management/DynamicMBean.html#invoke(java.lang.String,%20java.lang.Object[],%20java.lang.String[])">invoke
in the same way.

But that still leaves three other DynamicMBean methods that
don't throw MBeanException. How do we handle those?

For
href="http://java.sun.com/javase/6/docs/api/javax/management/DynamicMBean.html#getAttributes(java.lang.String[])">getAttributes
and
href="http://java.sun.com/javase/6/docs/api/javax/management/DynamicMBean.html#setAttributes(javax.management.AttributeList)">setAttributes,
the answer is simple. These methods are supposed to return an
href="http://java.sun.com/javase/6/docs/api/javax/management/AttributeList.html">AttributeList
containing all the attributes that were successfully got or set.
If the attempt to get any given attribute produces an error,
then that attribute is simply omitted from the returned list.
If you want to know what the error was, you have to call
getAttribute or setAttribute (rather than
get/setAttributes) with just the attribute in question
and see what exception it produces.

So if we get an IOException or an InstanceNotFoundException
when forwarding the getAttributes or setAttributes call to the
remote MBean, we can simply consider that every attribute produced
an error, and return an empty AttributeList.

Since DynamicMBean.getAttributes doesn't declare any checked
exceptions, we catch and rethrow RuntimeException, then catch
Exception, which is then all checked exceptions. This is easier
than catching the three checked exceptions that
MBeanServerConnection.getAttributes declares.

We can rewrite setAttributes the same way. So that just leaves
one method from DynamicMBean, getMBeanInfo.

getMBeanInfo()

href="http://java.sun.com/javase/6/docs/api/javax/management/DynamicMBean.html#getMBeanInfo()">DynamicMBean.getMBeanInfo
does not declare any checked exceptions, so we can't wrap an
IOException or InstanceNotFoundException in an MBeanException as
we did for getAttribute and co. We could try returning a
special "empty" MBeanInfo, but that is hacky. We could also
wrap the exception in a RuntimeException, which is less hacky
but still not very satisfactory. No RuntimeExceptions are
mentioned in the specification of
MBeanServerConnection.getMBeanInfo so callers won't necessarily
be prepared to deal with them.

I think the best solution is to get the MBeanInfo once when the
mirror is created and then simply return this value forever after
from the getMBeanInfo() method. This has a number of
advantages.

If the remote MBean is nonexistent or inaccessible, you
might as well find out at once when you try to create the
mirror, rather than waiting until the first time you use
it.

The getMBeanInfo() method will always be called at
least once anyway, when the MBean is registered. This is
because
href="http://java.sun.com/javase/6/docs/api/javax/management/MBeanServer.html#registerMBean(java.lang.Object,%20javax.management.ObjectName)">MBeanServer.registerMBean
returns an
href="http://java.sun.com/javase/6/docs/api/javax/management/ObjectInstance.html">ObjectInstance,
which is an ObjectName plus a class name. The class name is
the value of
href="http://java.sun.com/javase/6/docs/api/javax/management/MBeanInfo.html#getClassName()">MBeanInfo.getClassName()
for the MBean.

If the Java VM where the mirror is registered is running
with a
href="http://java.sun.com/javase/6/docs/api/java/lang/SecurityManager.html">SecurityManager
then every operation on the mirror MBean will need
MBeanInfo.getClassName() in order to construct the
href="http://java.sun.com/javase/6/docs/api/javax/management/MBeanPermission.html">MBeanPermission
that will be checked. If we don't keep a copy of the MBeanInfo
within the mirror, then every MBean operation will
require two round trips to the remote MBean, one to get the
MBeanInfo, and one to do the actual operation.

The principal disadvantage of this solution is that the
MBeanInfo of the remote MBean could change, and the mirror will
never show that. It is rare to have MBeans where the MBeanInfo
changes, and if you do then you can always override
PlainMBeanMirror.getMBeanInfo() to fetch the remote MBeanInfo when
appropriate.

So here's the new constructor that caches the MBeanInfo, and
the new, trivial getMBeanInfo() method that returns it:

Notifications

Another thing we'd like to be able to do with a mirror MBean is
receive notifications that were sent by the original MBean.
That is, if we call
href="http://java.sun.com/javase/6/docs/api/javax/management/MBeanServerConnection.html#addNotificationListener(javax.management.ObjectName,%20javax.management.NotificationListener,%20javax.management.NotificationFilter,%20java.lang.Object)">MBeanServerConnection.addNotificationListener
on the mirror MBean, we'd like our listener to receive the same
notifications as if we had called addNotificationListener on the
remote MBean.

We can do this just by implementing
href="http://java.sun.com/javase/6/docs/api/javax/management/NotificationEmitter.html">NotificationEmitter
(or its parent
href="http://java.sun.com/javase/6/docs/api/javax/management/NotificationBroadcaster.html">NotificationBroadcaster)
and forwarding its methods in the same way as we did for the
methods in DynamicMBean. However, it is better not to implement
NotificationEmitter if the MBean does not in fact emit
notifications. A client should be able to tell whether
addNotificationListener is allowed using
href="http://java.sun.com/javase/6/docs/api/javax/management/MBeanServerConnection.html#isInstanceOf(javax.management.ObjectName,%20java.lang.String)">isInstanceOf(mirrorName,
NotificationBroadcaster.class.getName()). In other words, we
should have one sort of mirror for MBeans that are
NotificationBroadcasters and another sort for MBeans that are
not.

The obvious way to achieve this is to have a subclass of
PlainMBeanMirror, say NotifyingMBeanMirror, that implements
NotificationEmitter, in the same way as
href="http://java.sun.com/javase/6/docs/api/javax/management/StandardEmitterMBean.html">StandardEmitterMBean
subclasses
href="http://java.sun.com/javase/6/docs/api/javax/management/StandardMBean.html">StandardMBean
in Java SE 6. But the drawback of that is that if we subclass
PlainMBeanMirror for another reason, for example to override
getMBeanInfo as we saw
href="#override-getMBeanInfo">above, then we will usually
need to subclass NotifyingMBeanMirror as well and duplicate the
same code in the two subclasses.

An alternative that avoids this problem is to use delegation
instead of subclassing. The idea is that a NotifyingMBeanMirror
wraps a PlainMBeanMirror and delegates the methods of
DynamicMBean to it.

For this to work cleanly, we define an interface that will be
implemented by both PlainMBeanMirror and NotifyingMBeanMirror.

The methods in this interface are useful for code that has
created a mirror to register it in the MBean Server. They are
also useful to NotifyingMBeanMirror because it can use the
interface for its delegation rather than hardwiring the concrete
class PlainMBeanMirror.

You might also have noticed
href="#MBeanMirrorFactory">earlier that
MBeanMirrorFactory.newMBeanMirror returns an MBeanMirror. The
idea is that this method returns a NotifyingMBeanMirror if the
remote MBean is a NotificationEmitter, and otherwise a
PlainMBeanMirror:

public class MBeanMirrorFactory { private MBeanMirrorFactory() {} // there are no instances of this class

The code of NotifyingMBeanMirror is straightforward but a bit
tedious. For the
href="http://java.sun.com/javase/6/docs/api/javax/management/NotificationBroadcaster.html#addNotificationListener(javax.management.NotificationListener,%20javax.management.NotificationFilter,%20java.lang.Object)">addNotificationListener
and
href="http://java.sun.com/javase/6/docs/api/javax/management/NotificationBroadcaster.html#removeNotificationListener(javax.management.NotificationListener)">removeNotificationListener
methods from NotificationEmitter, we again have a problem with
exceptions like IOException and InstanceNotFoundException, and
here we don't have a much better solution than wrapping them in
a RuntimeException. (
href="http://blogs.sun.com/jmxetc/">Daniel Fuchs has
suggested using
href="http://java.sun.com/javase/6/docs/api/java/lang/reflect/UndeclaredThrowableException.html">UndeclaredThrowableException
here.)

Further work

One thing that the Cascading Service from Java DMK does in
addition to mirroring is to track the creation and deletion of
remote MBeans. If you import "java.lang:*" from Subagent 1, say,
and a new MBean appears in Subagent 1 called "java.lang:type=New",
then a new mirror will appear automatically in the Master Agent.
Conversely, if an imported MBean disappears from the subagent then
its mirror automatically disappears too. This works by using a
listener on the
href="http://java.sun.com/javase/6/docs/api/javax/management/MBeanServerDelegate.html">MBeanServerDelegate
to learn of remote creations and deletions.

Another interesting question is whether ObjectNames should
sometimes be rewritten. If I subscribe to notifications from
"subagent1/java.lang:type=Runtime", what will the
href="http://java.sun.com/javase/6/docs/api/java/util/EventObject.html#getSource()">source
of the notifications be? Using the design above, it will be
"java.lang:type=Runtime", which might be unexpected. The
Cascading Service from Java DMK would rewrite the source to be
"subagent1/java.lang:type=Runtime". This discussion can go much
further if we think about whether ObjectName attributes and
parameters should also be rewritten.

The version of Cascading in version 2.0 of the JMX API will
probably be based on
href="http://blogs.sun.com/nickstephen/entry/jmx_extending_the_mbeanserver_to">Virtual
MBeans, and the details will look quite different from
what I've described here. But it will solve the same
problems.

I haven't talked at all about security, not because it is
unimportant but because there's too much to say. Another day
perhaps. The basic question is, how does the Master Agent
connect securely to each Subagent? And is there a way to have
different access to the Subagent MBeans for different users that
might be connected to the Master Agent?

The source code

The source code for the classes I've described above is in
href="http://weblogs.java.net/blog/emcmanus/archive/mirrormbean.zip">mirrormbean.zip,
along with what may be the most twisted unit test you have ever
seen.