Using JGroups Directly From JBoss AS 7 Component

UPDATE (2.09.2013) – Bela Ban pointed to his very interesting post about new ForkChannel feature and easy channel hijacking directly from JBoss AS / Infinispan. Read about it here

JGroups is Bela Ban‘s piece of software for reliable message exchange that is highly configurable and can use either TCP or UDP as a transport protocol. Basically – you run the JGroups on number of clients, they form a cluster and they can send and receive messages within the cluster.

JGroups is used internally by JBoss Infinispan. Infinispan, however, unlike JGroups adds the distributed cache semantics (replicated / distributed modes, entries invalidation, transactional behavior, Map access API, etc.) It even allows you to use the cluster as a compute grid.

Infinispan in turn is used to provide JBoss AS 7 clustering functionalities. Therefore, it means that the underlying JGroups subsystem can and is configured using a standard JBoss AS 7 standalone*.xml file. You can access Infinispan cache from your Java EE component (e.g. EJB) without any problems as described here.

However, there are cases when you’d like to use just the underlying JGroups messaging instead of all the cache semantics Infinispan gives you. And here’s the place things are becoming more complicated. You can always use JGroups directly and store the configuration for it as an application-local resources. It might become arguable if this is or isn’t a violation of the Java EE spec which says that an application should not manage low-level connections, spawn threads, open sockets, etc. This is something that is better to be left to the application server – it also allows us to use one configuration file instead of spreading it across multiple places.

So, the question is – how to access the JGroups subsystem from our EJB application? The whole solution involves few steps which will be described below.

it seems that it creates a new service (ChannelService) and let the JBoss MSC automatically inject its dependent ChannelFactory during the installation. The ChannelFactory will use the UDP protocol stack.

Now in case of JBoss < 7.2 it would be enough because channel service would automatically connect to our channel. However, since JBoss AS 7.2 this behavior was changed and we now need to connect to the channel manually using our own service — JGroupsService. So this is what this code does:

The second part is done in JGroupsChannelServiceActivator#bindChannelToJNDI(-) and it binds the newly created Channel instance to the JNDI under user-defined location. In our case it is java:jboss/channel/myChannel.

2. Register the Activator

We now need to tell JBoss AS 7 to invoke our custom Activator. It is done using standardized JDK ServiceLoader API. In a nutshell it means we need to provide a META-INF/services/org.jboss.msc.service.ServiceActivator file with fully qualified name of our activator class. Take a look at this example.

3. Add Required Modules to our Application

Ok, so we have an activator that should do the magic. If we’d try to deploy it as such we’ll get a bunch of ClassNotFoundExceptions. It is because the JBoss Modules. Our application is not packed with all those JBoss artifacts like JGroups, ServiceActivator API and the JNDI related classes. We don’t want to clutter our app with those libraries – we just want to define the dependencies on modules provided by JBoss AS 7 itself. We do it in the META-INF/jboss-deployment-structure.xml. Note that we could do it in MANIFEST.MF Dependencies: section but Intellij IDEA doesn’t seem to be working with Maven generated MANIFEST.MF:

and there it is. This EJB registers a timer that is invoked every 2 seconds and sends some random String message.

5. Deploy the EJB-JAR

We are now ready to deploy our application to JBoss AS 7 server. The most important part here is to make sure our server will be running with the appropriate configuration, which means one with JGroups protocol stack defined. It is done using <subsystem xmlns="urn:jboss:domain:jgroups:1.1">.

I am using JBoss EAP 6.1 Final and standalone-full-ha.xml config.

Note Because Intellij IDEA doesn’t allow you to easily change the configuration file for your JBoss AS as the Eclipse does, we will need to specify it using VM options: -Djboss.server.default.config=standalone-full-ha.xml.
Alternatively you can go to ‘Run configuration’ -> Startup/Connection -> Run -> and modify ‘Startup script’ to include -c standalone-full-ha.xml.

We also need to make sure JGroups will use IPv4 (it sometimes chooses the IPv6 which can lead to some weird and hard to solve problems.) To do so, add the -Djava.net.preferIPv4Stack=true option to the Server configuration.

6. Run the Client Application

You can find a rather simple Client code here. It just connects to JGroups cluster using specified configuration file. Mind that multicast port number and address should be set to the same values for server and client.

6 thoughts on “Using JGroups Directly From JBoss AS 7 Component”

A couple things…It looks like they’ve removed the automatic connection in the ChannelService as of 7.2. So I’ve had to connect the channel myself to get this working. But then I run into a bit of a problem in that when the app is re/undeployed. The original channel continues running and a different channel instance is injected and I’m not able to connect again because the other channel is still connected. I’ve added a PreDestroy to disconnect the channel and that seems to take care of it but I have a couple questions about that.

Is there some other way I should be handling it rather than connecting and attempting to disconnect with a PreDestroy?

Are you aware if it is possible for the PreDestroy not to be called like say if the app fails to undeploy properly for any reason? That would likely leave other group members not knowing that this node is no longer there.

Yep, they’ve removed automatic connection, so I’ve updated my post accordingly. Did you achieve channel connection as I did – using JBoss services or did you activate the channel using e.g. @Startup of your singleton EJB? Perhaps that’s the problem during redeploys?

I guess there are situations in which @PreDestroy method will not be called but on the other hand – there seem to be AFAIR SUSPECT protocol in JGroups that is responsible for checking if particular node should not be removed from the members because it probably died.

I followed the instruction step-by-step but it seems like it doesn’t work on WildFly 8.0.0.Final.
I keep getting the error “java.lang.NoClassDefFoundError: org/jgroups/ReceiverAdapter” when the server is starting. It looks like the class loader does not familiar with JGroups subsystem even though it is defined in the standalone.xml and I have the modules specifies in my jboss-deployment-structure.xml.

Do you know of any configuration changes required in the code you published in GitHub for the sample to work on JBoss WildFly 8.0?

We try to deploy our web application WAR file on JBoss 6.1 Final server in a cluster environment on two hosts setup with IPv6 protocol.
FYI
– We make use of jgroups v2.12.1 library for handling the cluster communications but it is not working.
– We use clustering over TCP (not UDP)

Can anyone please help or suggest any steps that we shall take care in case of clustering setup over IPv6 enabled hosts ?