VersionPlease use the newest JBoss EAP 6.1.0.Final to follow this document, or you'll meet a lot of bugs.

In this article, I'd like to show you how to setup JBoss AS7 in domain mode and enable clustering so we can get HA (high availability) and session replication among the nodes. It's a step to step guide so you can follow the instructions in this article and build the sandbox by yourself.

Preparation & Scenario

Preparation

We need to prepare two hosts (or virtual hosts) to do the experiment. We will use these two hosts as follows:

Install Fedora 16 on them (Another linux version may also work, but I'll use Fedora 16 in this article)

Make sure the hosts are in same local network

Make sure that they can access each other via different TCP/UDP ports (it's better to turn off firewall and disable SELinux during the experiment or they will cause network problems).

Scenario

Here are some details on what we are going to do:

Let's call one host 'master' and the other one 'slave'.

Both master and slave will run AS7, and master will run as domain controller, slave will under the domain management of master.

Apache httpd will be run on master, and we'll enable the mod_cluster module in httpd. The as7 on master and slave will form a cluster and be discovered by httpd.

We will deploy a demo project into domain and verify that the project is deployed into both master and slave by domain controller. Thus we could confirm that domain management provide us a single point to manage the deployments across multiple hosts in a single domain.

We will access the cluster URL and verify that httpd has distributed the request to one of the as7 host. This helps us confirm that the cluster is working properly.

We will try to make a request to the cluster, and if the request is forwarded to master as7, we then kill the as7 process on master. After that we will go on requesting cluster and we should see that the request has been forwarded to slave, but the session has not been lost. Our goal is to verify the HA is working and sessions are replicated.

After the previous step has been completed we reconnect the master as7 by restarting it. We should see the master as7 is registered back into cluster, and we should also see slave as7 recognizes master as7 as domain controller again and connects to it.

Please don't worry if you cannot digest so all the details now. Let's move on and you will get the points step by step.

Download JBoss AS7

First we should download the AS7 from website:

The version I downloaded is 7.1.0.CR1b, please don't use the version prior to this one, or you will meet this bug when running in clustering mode:

After the download has completed, I have the zip file:

Then I unzipped the package to master and try a test run:

If everything ok we should see AS7 successfully startup in domain mode:

Now exit as7 and let's repeat the same steps on slave host. Finally we get AS7 run on both master and slave, then we can move onto the next step.

Domain Configuration

Interface config on master

In this section we'll setup both master and slave for them to run in domain mode. And we will configure master to be the domain controller.

First open the host.xml in master as7 for editing:

The default settings for interface in this file is like:

We need to change the address to the management interface so slave can connect to master. The public interface allows the application to be accessed by non-local HTTP, and the unsecured interface allows remote RMI access. My master's ip address is 10.211.55.7, so I change the config to:

Interface config on slave

Now we will setup interfaces on slave. First we need to remove the domain.xml from slave, because slave will not act as domain controller and is under the management of master. I just rename the domain.xml so it won't be processed by as7:

From AS 7.1.Final you don't need to rename domain.xml anymore.

Then let's edit host.xml. Similar to the steps on master, open host.xml first:

The configuration we'll use on slave is a little bit different because we need to let slave as7 connect to master as7. First we need to set the hostname. We change the name property from:

to:

Then we need to modify domain-controller section so slave as7 can connect to master's management port:

As we know, 10.211.55.7 is the ip address of master.

Finally, we also need to configure interfaces section and expose the management ports to public address:

10.211.55.2 is the ip address of the slave. Refer to the domain controller configuration above for an explanation of the management, public, and unsecured interfaces.

It is easier to turn off all firewalls for testing, but in production, you need to enable the firewall and allow access to the following ports: TBD

Security Configuration

If you start as7 on both master and slave now, you will see the slave as7 cannot be started with following error:

Because we haven't properly set up the authentication between master and slave. Now let's work on it:

Master

In bin directory there is a script called add-user.sh, we'll use it to add new users to the properties file used for domain management authentication:

As shown above, we have created a user named 'admin' and its password is '123123'. Then we add another user called 'slave':

We will use this user for slave as7 host to connect to master.

Notice that the username must be equal to the name given in the slaves host element. That means that for each additional host you need a user.

In newer versions of JBoss AS7, the add-user.sh will let you choose the type of the user. Here we need to choose 'Management User' type for both 'admin' and 'slave' account:

Slave

In slave we need to configure host.xml for authentication. We should change the security-realms section as following:

We've added server-identities into security-realm, which is used for host authentication when slave tries to connect to master. Because the slave's host name is set as 'slave', so we should use the 'slave' user's password on master. In the secret value property we have 'MTIzMTIz=', which is the base64 code for '123123'. You can generate this value by using a base64 calculator such as the one at http://www.webutils.pl/index.php?idx=base64.

Then in domain controller section we also need to add security-realm property:

So the slave host could use the authentication information we provided in 'ManagementRealm'.

Setup HornetQ

The newer version of JBoss AS7 has enforced the security checking for HornetQ. So you need to some additional configuration in domain controller. Open domain.xml of master and find:

This is the config for hornetq, we need to put 'cluster-user' and 'cluster-password' into it:

There are two "<subsystem xmlns="urn:jboss:domain:messaging:1.3">" config sections in domain.xml. One is for "<profile name="full">" and another for "full-ha". If you have activated both of these two profiles, then you should put the 'cluster-user' and 'cluster-password' settings in both of them.

The 'cluster-user' and 'cluster-password' settings are just user-defined values; you can choose whatever you want. The authentication scheme used by HornetQ here is simple: when a host connects to domain controller, the HornetQ instance running on domain controller will tell the host its 'cluster-user' and 'cluster-password' settings. Then the connected host's HornetQ instance will use this setting to communicate and form a cluster with other HornetQ instances running in the domain. This is a simple but effective way to prevent some outsiders from participating on this domain's HornetQ cluster.

If you do not want to use this security scheme, just disable it in "<subsystem xmlns="urn:jboss:domain:messaging:1.3">":

Dry Run

Now everything is set for the two hosts to run in domain mode. Let's start them by running domain.sh on both hosts. If everything goes well we should see the following in the log on master:

That means all the configurations are correct and the two hosts are running in domain mode now as expected. Hurrah!

Deployment

Now we can deploy a demo project into the domain. I have created a simple project located at:

We can use git command to fetch a copy of the demo:

In this demo project we have a very simple web application. In web.xml we've enabled session replication by adding following entry:

And it contains a jsp page called put.jsp which will put current time to a session entry called 'current.time':

Then we could fetch this value from get.jsp:

It's an extremely simple project but it could help us to test the cluster later: We will access put.jsp from cluster and see the request are distributed to master, then we disconnect master and access get.jsp. We should see the request is forwarded to slave but the 'current.time' value is held by session replication. We'll cover more details on this one later.

Let's go back to this demo project. Now we need to create a war from it. In the project directory, run the following command to get the war:

It will generate cluster-demo.war. Then we need to deploy the war into domain. First we should access the http management console on master (Because master is acting as domain controller):

It will popup a windows prompting you to input an account name and a password; we can use the 'admin' account we've just added. After logging in we will see the 'Server Instances' window. By default there are three servers listed, which are:

server-one

server-two

server-three

We can see server-one and server-two are in running status and they belong to main-server-group; server-three is in idle status, and it belongs to other-server-group.

All these servers and server groups are set in domain.xml on master as7. What we are interested in is the 'other-server-group' in domain.xml:

We could see this server-group is using 'ha' profile, which then uses 'ha-sockets' socket binding group. It enable all the modules we need to establish the cluster later (including infinispan, jgroup and mod_cluster modules). So we will deploy our demo project into a server that belongs to 'other-server-group', so 'server-three' is our choice.

In newer version of JBoss AS7, the profile 'ha' changes to 'full-ha':

Let's go back to domain controller's management console:

Now server-three is not running, so let's click on 'server-three' and then click the 'start' button at bottom right of the server list. Wait a moment and server-three should start now.

Now we should also enable 'server-three' on slave: From the top of menu list on left side of the page, we could see now we are managing master currently. Click on the list, and click 'slave', then choose 'server-three', and we are in slave host management page now.

Then repeat the steps we've done on master to start 'server-three' on slave.

server-three on master and slave are two different hosts, their names can be different.

After server-three on both master and slave are started, we will add our cluster-demo.war for deployment. Click on the 'Manage Deployments' link at the bottom of left menu list.

(We should ensure the server-three are running on both master and slave)

Then enter 'Manage Deployments' page, click 'Add Content' at top right corner. Then we should choose our cluster-demo.war and follow the instruction to add it into our content repository.

Now we can see cluster-demo.war is added. Next we click 'Add to Groups' button and add the war to 'other-server-group' and then click 'save'.

After waiting a few seconds, the management console should tell you that the project is deployed into 'other-server-group'.：

Please note we have two hosts participate in this server group, so the project should be deployed in both master and slave now - that's the power of domain management.

Now let's verify this by trying to access cluster-demo from both master and slave. They should all work now:

Now that we have finished the project deployment and see the usages of domain controller, we will then head up for using these two hosts to establish a cluster

Why the port number is 8330 instead of 8080? Please check the settings in host.xml on both master and slave:

The port-offset is set to 250, so 8080 + 250 = 8330

Now we quit the as7 process on both master and slave. We have some work left on host.xml configurations. Open the host.xml of master, and make the following modifications to the servers section changing from:

to:

We've set auto-start to true so we don't need to enable it in management console each time as7 restart. Now open slave's host.xml, and modify the server-three section:

Besides setting auto-start to true, we've renamed the 'server-three' to 'server-three-slave'. We need to do this because mod_cluster will fail to register the hosts with same name in a single server group. It will cause name conflict.

After finishing the above configuration, let's restart two as7 hosts and go on to cluster configuration.

Cluster Configuration

We will use mod_cluster + apache httpd on master as our cluster controller here. Because AS7 has been configured to support mod_cluster out of box so it's the easiest way.

The AS7 domain controller and httpd are not necessary to be on same host. But in this article I just install them all on master for convenience.

First we need to ensure that httpd is installed:

And then we need to download newer version of mod_cluster from its website:

The version I downloaded is:

Jean-Frederic has suggested to use mod_cluster 1.2.x. Because 1.1.x it is affected by CVE-2011-4608

With mod_cluster-1.2.0 you need to add EnableMCPMReceive in the VirtualHost.

Then we extract it into:

Then we edit httpd.conf:

We should add the modules:

Please note we should comment out:

This is in conflict with cluster module. Then we need to make httpd listen to public address so we can do the testing. Because we installed httpd on master host we know the ip address of it:

Then we do the necessary configuration at the bottom of httpd.conf:

For more details on mod_cluster configurations please see this document:

Testing

If everything goes well we can now start the httpd service:

Now we access the cluster:

We should see the request is distributed to one of the hosts(master or slave) from the as7 log. For me the request is sent to master:

Now I disconnect master as7 by using the management interface. Select 'runtime' and the server 'master' in the upper corners.

Select 'server-three' and kick the stop button, the active-icon should change.

Killing the server by using system commands will have the effect that the Host-Controller restart the instance immediately!

Then wait for a few seconds and access cluster:

Now the request should be served by slave and we should see the log from slave as7:

From the get.jsp we should see that the time we get is the same we've put by 'put.jsp'. Thus it's proven that the session has been correctly replicated to slave.

Now we restart master as7 and should see the host is registered back to cluster.

It doesn't matter if you found the request is distributed to slave at first time. Then just disconnect slave and do the testing, the request should be sent to master instead. The point is we should see the request is being redirected from one host to another and the session is held.

Additional Resources

This article is focusing on AS7 domain mode + mod_cluster integration. If you are interested in using AS7 with mod_jk, please check Using mod_jk with JBoss AS7

If you want to know how to develop an application with AS7, please check [Developing JSF Project Using JBoss AS7, Maven and IntelliJ]

Special Thanks

Wolf-Dieter Fink has contributed the updated add-user.sh usages and configs in host.xml from 7.1.0.Final.Jean-Frederic Clere provided the mod_cluster 1.2.0 usages.
Misty Stanley-Jones has given a lot of suggestions and helps to make this document readable.

1. The demo project itself is a very simple web app that has session replication config enabled in web.xml:

At first half of the article we've setup two hosts and make them form a domain. master is acting as domain controller+host and slave is a host that under the control of master. We then use the domain controller to deploy the web project with one click on both master and slave (by HTTP management console). So we are showing the functions provided by domain controller at this point: Provides a central point to management the project deployments across all the hosts in domain.

Then we use mod_cluster module in http to connect with two AS7 hosts to form a cluster. And then we create a scenario by disconnecting a host that is accepting requests to test the failover are working correctly, and sessions are replicated to the other host. Actually this is achieved by Infinispan, JGroup and mod_cluster modules together, which are defined in domain.xml. They are just configured out of box.

I should explain more details on the clustering in article. I'll add them soon.

why am i getting this error I am jboss 7.1 final on wondows. is this saying that my host can not connect to the unsecure interface. I notice that I also do not see my servers on the domain control panel.

Sorry I couldn't reproduce your problem on my machines. Maybe posting host.xml on both your master and slave to https://gist.github.com/ and provide a link here could be more helpful for me to investigate your problem.

Thanks for the cluster guide. It was really helpfull in getting everything set up. However I used a Windows XP pro OS, and the MS Loopback adapter with 3 custom IP's. I then used these as my "servers". All goes well and the clustering from the JBoss 7.1 AS servers works well. However when I try to setup the httpd server I run into problems.

I first tried a already installed Apache HTTP server with the mod_x.so mentioned just copied from the mod_cluster zip file, and the config you mentioned. No dice. Then I tried both the 1.20 mod cluster httpd and the 1.1.3 without any success. I can get the default "IT WORKS" page, but it seems not to forward to the JBoss AS 7.1 when trying to access cluster_demo.

Here are the questions I have. If you are not running the httpd on the same server as the master jboss, what is the ip supposed to be for the following:

I'm a little bit confused about your problem. You said the mod_cluster doesn't work in http. But then you said you can get the default page when accessing cluster url. Could you please tell me more details on your problem?

For the listening address configurations of httpd, you should set the Listen to the IP address of your host which httpd is running on, and please ensure the IP address are in same local network with your AS7 hosts.

VirtualHost address should be same with the Listen address, which is also the IP address that your httpd is running on.

Allow from should be set to the subnet address that all your machines are in(Your httpd and AS7 hosts are all in same subnet right? Please ensure that's true).

The Apache HTTP zip comes with a page in the htdocs directory with one index.html file that just reads "IT WORKS!" So when I go to http://localhost/ thats what I get. However when I try to go to http://localhost/cluster-demo/put.jsp I just get a 404. So what I am seeing is that its trying to get cluster-demo/put.jsp from the htdocs directory. So the mod_cluster on its own does nothing. Do we need to add a entry to forward all requests of a certain type to the server?

After two AS7 hosts are setup they can be accessed directly by their own IPs. And session replication is working with the help of JGroup and Infinispan configured out-of-box in AS7 domain.xml. But the cluster is useless if we could not have a 'single entry point' for user to access(We don't want the users to know each AS7 node's IP address).

So we need mod_cluster+httpd to provide an address that user could use for accessing the cluster: httpd will provide a listening IP address that user can access, and mod_cluster is used to distribute the requests to one of the node in cluster(depending on each node's load, such as CPU load, memory usage, request numbers, etc.)

I guess you have gotten the idea now :-) So the endpoint address used in your web service should not be one of the AS7 host's ip address. It should be http://<your_httpd_listening_address>:8000 (Because you said your httpd is listening on port 8000).

In my example I've put httpd on the same machine with one of the AS7 host, so the IP address sames like the same with master AS7 ip address. This may create some confusions :-) It could be more clear if I use three machines to show the demo then you could see the httpd IP address is standalone.

Actually we don't have to use mod_cluster to do the request distribution, we could use LVS, or even hardware devices. But with mod_cluster we could get some nice features out of box such as nodes auto-discovery, recovered host auto registering back, etc. Because both mod_cluster and AS7 are developed by JBoss dev team and they are designed to be integrated with each other nicely :-)

So let's go back to your httpd config, you should set it to listen to a public ip address instead of 127.0.0.1, and you could access your cluster by http://<your_httpd_listening_address>:<your_httpd_listening_port>

I am sorry to say but I still cannot get anything into the cluster (but I can send client transactions to both master and slave on port 8082 as defined in domain.xml on the master. The slave does not have a domain.xml as per instruction. It reads domain.xml from the server using port 9999).

I made some changes to the article, to simplify exactly what needs to be configured. Tomorrow I will make some more changes to keep things secure (exactly what needs to be changed on the firewall and selinux configs).

I'm not so familiar with the WebServices configuration in domain.xml (I generally use JAX-RS which has configured out-of-box by AS7), that may need some extra configurations for it to work in cluster environment.

We have put almost 3 weeks into getting the new load balancing working with JBoss AS 7 and mod_cluster. We = two very experienced professional IT people with a shared experience span of around 60 years.

I am very grateful to the community members who have put lots of effort into the new JBoss clustering.

These are my observations:

1) Setting up clustering under Unix / Linux is a breeze. I set it up under 30 minutes on Mac OS X yesterday (that made me very happy, amazing software). I also had two instances running under Centos 6.

2) Setting up the same, using the same domain.xml and host.xml files from Unix/Linux, under Windows 7 is very cumbersome. Despite all our efforts we could not make it work

4) We cannot get anything into the cluster using mod_cluster via httpd. We can send transactions to all cluster instances but we cannot send anything to the actual cluster. And for intelligent load balancing we have to be able to send transactions to the cluster, not to cluster members. Otherwise you get the same inflexible load balancing as under mod_jk where you have to guess how busy a server is going to be.

I can get to the point where I can see the mod_cluster-manager, but I can't access my deployed app via Apache, only directly via JBoss Web. I think there is some subsystem configuration or something that is missing, that tells Apache where to look for the deployment in the AS server. I'll look into it and post back here when I find it.

Today we managed to bring up a cluster under Windows 7 and we managed to get transactions into the cluster using port 8000.

I have Apache httpd running on a Windows XP box and 2 JBoss instances on a Linux server and an iMac.

Please see below for the httpd VirtualHost config (192.168.2.2 is the XP box, no need to make reference to any cluster members by IP address, that is automatic. Please note the URL used by ProxyPass (as per Jean Frederic's suggestion).

Having different logs per VirtualHost is also nice, it makes things more readable.

Session replication doesn't seem to work for me. I followed the steps in this guide, but when I kill server three in master, the process also destroys the session that has been made. So when I run the get.jsp in slave, it shows that time is null (it didn't get any session).

Should be deleted or renamed domain.xml!And then perform the the slave Node of domain.bat, Do not error?Because domain.bat references the domain.conf.bat When you edit domian.conf.bat, you will find set JAVA_OPTS = the% JAVA_OPTS percent-Djboss.domain.default.config =, domain.xml-Djboss.host.default.the config = host.xml this line command.

These instructions work for jboss-as-7.1.0.Final "Thunder" but not for jboss-as-7.1.1.Final "Brontes". On 7.1.1, starting server-three fails with "jboss.binding.jacorb (missing) dependents" as in https://community.jboss.org/thread/197320

Everything works fine, except the last step - http://<ip_address>cluster-demo/get.jsp :-( . I even read the above, and I saw few people having similar problems. I have checked that other-group server instance is up.

I was also able to access mod_cluster-manager.

Could you explain how does it know that it has to connect to JBoss ? probably I am missing something very basic here !

I have observed the following while trying to set up clustering in single pc on domain mode (no slave configuration). If you find some tweaking to these issues, please let me know.

A). to cluster to be correctly operate, socket group name of the
server group where the profile name=”full-ha”,must be in “full-ha-sockets”.The bare minimum requirement is “ha-sockets”though it doesn’t correctly operate the “failover” concept. (if one server is down it will stop responding to user requests and the other servers in the cluster will not serve the requests). Additionally server start with errors (dependency issues) as follows.

C) Even after you start the server and try to start other servers in the group through admin console, it will enter to deadlock state.

However difference between “full-ha-sockets”and “ha-sockets” configuration is “full-ha-sockets” configuration still operate and serve the user request in this situation. (Server which is auto started at boot, will continuously serve the request unless you stop this server, and keep other in the same server group).

So are you starting 6 servers in total (3 in each machine ? ) ? To debug start each of the servers one after another (disable auto start =true in domain.xml host.xml of master and slave) and find out on start of which of the server causes this problem to be showing up in console continously?

Another simple test you can try if it is indeed caused by port - change offset for server two and server three in Master to 151 and 251 and in slave as 101 and 201 (or some odd numbers) (This is just a guess )

1) I made a call to put.jsp, it was directed to master machine and i stopped server three of

master. made a call to get.jsp and it was directed to slave machine. everything fine.

2) Brought back server three of master, now again i made a call to put.jsp and it was

failed 'service temporary unavailable'

3) the error on console when i brought back 'master'

[Server:server-three] 23:46:22,379 WARN [org.hornetq.core.cluster.impl.Di
scoveryGroupImpl] (hornetq-discovery-group-thread-dg-group1) There are more than
one servers on the network broadcasting the same node id. You will see this mes
sage exactly once (per node) if a node is restarted, in which case it can be saf
ely ignored. But if it is logged continuously it means you really do have more t
han one node on the same network active concurrently with the same node id. This
could occur if you have a backup node active at the same time as its live node.
nodeID=6498e4cb-8061-11e1-a0d6-24b6fd012f2a

OK, so I decided to try using linux instead of windows XP. I used Virtualbox with two Ubuntu server installs. I used JBoss AS 7.1 and jdk 7.0.3. After configuration of the cluster I deploy the "cluster-demo.war". The deploy screen then gets stuck in "Please wait: Your request is being processed ...".

if mod_cluster-manager only shows a heading like "mod_cluster/1.2.0.Final" and some links (showing nothing like you say) your nodes have not been registered on the apache-side of mod_cluster. This would also explain that you get a 404 when you try to access the cluster-demo through mod_cluster (port 10001) but it works if you go directly on a JBoss instance.

I had this problem by myself and it stinks like JBPAPP-7516 because it works fine with the latest EAP6 beta and the JBoss AS7.1.2 (you can get a nightly build here).

Also watch your JBoss for any error-messages on the terminal. Sometimes the error-messages are hard to see between all the other messages.

Maybe you should give the latest working nightly-build a try, But be warned, just copying your actual configuration files to the new JBoss installation won't work, because the JBoss makes changes to these files during deployment.

Thanks for the prompt response. I agree with your conclusion that the cluster is not being registered on the Apache side as this explains all I see. There are no errors in my JBoss console and I will try the nightly build next. I don't understand your point about configuration files though - why will JBoss overwrite my config files (domain.xml and host.xml) or did I misunderstand you?

I think we will probably end up running a number of JBoss instances behind a hardware balancer so httpd may be moot, but I still need to prove this works in our dev environment.

the point with the configuration files (I think it was just the host.xml) is that if you deploy something into the JBoss it will write that (with some hash-values and so on) into the host.xml so the deployment is not lost after a restart of the JBoss. Maybe this behaviour does only occur when you deploy over the management-console. But after trying different JBoss 7 versions (including 7.1.0 which refuses to deploy SFSBs annotated with @Clustered) I ran into this problem a couple of times. Just copying the configuration files won't work: The JBoss you copied your configuration to doesn't find the deployments of the old JBoss you copied from and maybe some more issues.

I don't think it's the most efficient way but to solve the issue, but I alway ended up setting up a fresh configuration (so I'm quite fit in setting up the cluster described in this howto ). After all this issue isn't really a big deal. I just wanted give a little hint if you run into a problem that looks like this.

First the caveat: it seems all nodes in a cluster need to run the same version of JBoss 7.

Second: I was unable to suppress this message:

[Server:server-three] 13:36:29,732 WARN [org.hornetq.core.cluster.impl.DiscoveryGroupImpl] (hornetq-discovery-group-thread-dg-group1) There are more than one servers on the network broadcasting the same node id. You will see this message exactly once (per node) if a node is restarted, in which case it can be safely ignored. But if it is logged continuously it means you really do have more than one node on the same network active concurrently with the same node id. This could occur if you have a backup node active at the same time as its live node. nodeID=5b2e186c-9af8-11e1-9675-e3f2a1f44414[Server:server-three] 13:36:29,732 WARN [org.hornetq.core.cluster.impl.DiscoveryGroupImpl] (hornetq-discovery-group-thread-dg-group1) There are more than one servers on the network broadcasting the same node id. You will see this message exactly once (per node) if a node is restarted, in which case it can be safely ignored. But if it is logged continuously it means you really do have more than one node on the same network active concurrently with the same node id. This could occur if you have a backup node active at the same time as its live node. nodeID=5b2e186c-9af8-11e1-9675-e3f2a1f44414

Again, any suggestion about killing this would be great. I'm running on os X (Lion) so the comments about NFS don't really help me much.

ok, maybe I'm a bit late but: Have you created the slave by copying the master? Then try to delete the contents of domain/data and/or domain/servers/yourserver/data . I by myself didn't experience this problem.

I followed these directions and comments by others very carefully and I got everything working *except* for session replication. I tried adding the <cluster-user> and <cluster-password> as was suggested. I tried both 7.1.1-Final and yesterdays latest 7.1.x build (really 7.2.0 Alpha-1). I made the minimal changes required and everything else is stock settings.

Once the other-server-group (server three) begins running I start getting errors on both master/slave as others have reported.

As was suggested in a number of forums postings I tried adding cluster-password. It wasn't clear if you were supposed to add <cluster-name> and <cluster-password> to both master and slave so I added it to both but to no avail (still doesn't work).

There is NO jboss user named this but according to the directions there doesn't seem to need to be one. It's interesting that the exception says 'HORNETQ.CLUSTER.ADMIN.USER' even though I specified to use 'admin'

I should have been more clear but I already tried using a valid user from the ManagementRealm previously (added via add-user.sh) and that didn't help. Like I said based on the error message it doesn't appear like it was honoring the <cluster-user> tag since the exception referred to 'HORNETQ.CLUSTER.ADMIN.USER'

I also tried adding 'HORNETQ.CLUSTER.ADMIN.USER' as a valid user to the ManagementRealm and ran into the same problem.

I was able to setup till the last step but the request is not getting forwarded from Apache Http server to AS7 servers. Both my Apache Http Server and AS7 master are running on a Windows7 machine and the slave AS7 is running on Window 7 laptop. Both the main-server-group and other-server-group (full-ha) are up and running correctly. I am able to access the deployed app on port 8330. Also the Http server is up and running on port 80 so when I put my master ip address I get "It Works!" message. Also, able to access the mod_cluster_manager on port 8000 correctly. Not sure what I am missing currently as the logs are not pointing me to any errors that might be happening.

Hi,
Can someone help me?
I try to install cluster in JBoss 7.1.1 with mod_cluster 1.2.0.
At final step, when i access to /cluster-demo/put.jsp then a message "putting date now" is sent to my master node, However when I stop master node, I try access to /cluster-demo/get.jsp and I see "The time is null".
It seem httpd can redirect request to other node, but without session.
This is my httpd.conf file:
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_ajp_module modules/mod_proxy_ajp.so
LoadModule slotmem_module modules/mod_slotmem.so
LoadModule manager_module modules/mod_manager.so
LoadModule proxy_cluster_module modules/mod_proxy_cluster.so
LoadModule advertise_module modules/mod_advertise.so
LoadModule authz_host_module modules/mod_authz_host.so
Listen 192.168.72.101:8888
<VirtualHost 192.168.72.101:8888>

<Location />

Order deny,allow

Allow from all

</Location>

<Location /mod_cluster_manager>

SetHandler mod_cluster-manager

Order deny,allow

Allow from all

</Location>

KeepAliveTimeout 60

MaxKeepAliveRequests 0

AllowDisplay On

ManagerBalancerName cluster-server-group

ServerAdvertise On

EnableMCPMReceive
</VirtualHost>

I add already some workaround as:
instance-id

proxy-list="IP:Port"

Special one, If I don't add module authz_host_module, I can not start httpd. Httpd alway say that "Order" miss spelling ...

thanks for the post it relay helped me to understand standalone mode clustering of Jboss
I have done the above in windows xp using APACHE 2.2 and HTTPD 2.2 and Jboss AS 7.1.1.Final and it was successful with the application shown below,

</html>

The example stateful session bean is a CDI component associated to the HTTP session. The bean has a counter which is increased every time it is read by the getter-method getCounterValue(). With the counter you can track how often the bean was invoked during the session.

in the apche httpd.config i have done the following

But when i went on further and deployed my own application which uses ejb 3, richfaces and hibernate along with JSF. while we are running a single node jboss standalone cluster the application runs fine and smooth using apache httpd url ie http://127.0.0.1/WiseBank_CBS_v1/login.xhtml.

but when running more then one standalone jboss node of jboss, when ever i access the above url in jboss server node i see lots of error trace which i am unable to debug. attaching the jboss server log.

I was following this guide successfully, up until I got to the point of adding datasources. Where are you supposed to add them, host.xml or domain.xml? I have not deployed any application yet, I am just trying to start the cluster. With the datasources (mysql) configured in domain.xml, the master starts up properly, but when I start the slave, it immediately connects to the database multiple times, eating up ALL of the available connections.

To test everything is working, fire up the domain, deploy an application and start httpd (on windows 7 I needed to run it as administrator otherwise it complained about file permissions). Navigate to: http://172.72.103.134:10001/mod_cluster_manager. You should see:

First, thank you reply.I was the latest apache2.4.4 downloaded from the official website of Apache version, then downloaded from the http://www.jboss.org/mod_cluster mod_cluster-1.2.0.Final-windows-x64-ssl.zip this version, mod_cluster1.2.0 after decompression mod_proxy_cluster.so copid to% APACHE_HOME% \ modules.Of course, there are three files are also copied to this directory.If I only use mod_cluster-1.2.0.Final-windows-x64-ssl.zip this version, I do not know how to start it in Windows.My operating system is Windows XP (32), I double-click http.exe and displays the error message "not a valid win32 application"

1) download only mod_cluster-1.2.0.Final-windows-x64-ssl.zip, then unzip it to some folder. This folder has the same folder structure as in Apache 2.4.4, except the https.conf file. So,you don't need to download Apache 2.4.4 version. Only you need to modify the httpd.conf contents as I have shared above mail chain.

2) You will get the httpd.conf file inside the bin folder. If that file is not there then take the file from bin/default folder or create a httpd.conf file & paste the above contents.

Not sure if anyone is still tracking these comments .. a bit old now. Thanks for the information .. very useful!

I am running into an issue with domain mode configuration. I have Apache 2.2 with mod_cluster, 1 domain controller and 2 host controllers running on a single pc for testing. That is 3 separate instances of JBoss Eap 6.1

I can see both host controllers join the domain. And I can see both nodes recognize mod_cluster with the log output that they are using jvm-route. The number of nodes in the cluster logged appears correct.

When accessing the application through apache; the behavior i am seeing is that when i stop the server that requests are being sent to, the second server picks up the request but the time is null. if i switch back and forth stopping the current server and refreshing the page, the time is set for each individual server. So i have 2 separate times, one for each node and they continue to change as that node is hit.

In further testing i can see that the session is replicated for the servers in the same host, but they are not replicated to servers in the other host.

I am not getting any errors in the server logs or the http log.

Httpd.conf

Domain.xml (using full-ha profile and full-ha-sockets)

Domain host.xml

Slave 1 host.xml

Slave 2 host.xml

Any clue as to what would cause this behavior? Is there additional information I can provide?
Any insight is greatly appreciated.

I am trying to follow this tutorial and install a cluster of jboss on the amazon (ec2).

I'm using the micro instance that have 613 Mb of memory to be the "master server". The memory is not enough and i am giving out of memory exception. My question is: What is server-one, server-two and server-three ? I really need 3 servers running on master ?

At the picture Testing Scenario, i have only two servers running on master server: httpd + mod_cluster AND AS7 server. (only 2 servers). Where is the server-three ?

I think that on master should exist only 1 server (domain controller - administrator os the domain), and optionally a host. I don't understood why 3 servers.

What is the function os each three servers. Can some of them be optional ?

Have you used some messaging features? The messaging feature are only provided by '-full.xml', '-full-ha.xml' configs. If you are using domain mode, make sure you are using '-full' or '-full-ha' profile for your server group.

Please also update to EAP 6.1.0 instead of older version of JBoss AS7, which has fixed a lot of bugs.

JGroup cannot send message to slave. Have you turned off your firewall? SElinux disabled? Does you network support UDP + IP multicasting? These places are worth checking.

I have followed the steps in this guide to form a cluster environment. After that, I have deployed jbpm-console war on server three. But when I start server-three after deployment, it fails to start stating the following error -

I have two Jboss AS7 Server ( Server1 & Server2), from which Server1 is behave as master & Server2 behave as slave and two mod_cluster setup (mod1 & mod2). With single mod1, I am able to track & do the above cluster step up.

Now problem is, how to do the request processing from mod2 for both Server1 & Server2, if mod1 got fail.

After that it works fine with httpd and jboss on the same host (that bug also suggests an explicit external interface definition might be needed for use across a network). It's probably worth a note in this guide for those using 7.1.1; it's fixed in the current EAP6 release.

Generally speaking, mod_cluster uses advertising channel (an IP multicast channel) to advertise itself to the EAP6 worker nodes. If two mod_clusters advertising their addresses in the advertising channel, then the EAP6 server will register into these mod_cluster instances.

I need some time to write a document on how to do this. This weekend I'll try to find some time

first of all I want to thank Li for his great tutorial. Perhaps someone can help me with my problem.

I´m using EAP 6.1 on master (Win7) and slave (Ubuntu Server) for my test environment. I also use mod_cluster-1.2.6.Final-windows-amd64. Everything works fine, if both pcs are within the same subnet. But how can I setup a cluster where the slave is in a different network?

The recommended solution is to put your httpd load balancer and your EAP6 servers in a same network. Because multiple components are either relying on IP multicast or have better performance in a LAN. For example: HornetQ, mod_cluster, JBoss Remoting, JGroups, etc. Putting your worker nodes and load balancers in a WAN will increase your configuration tasks, you have to consider much more things to make it working correctly.

A better solution could be: Building application level clusters in a LAN, and then use an IP level load balancer solution to load balance these clusters in different networks. Another options is to partition your cluster by its functions. For example, put web services cluster in one network, and put web application cluster in another network. And you can also use technologies like CDN to serve the static contents such as images. These are all better to partition your EAP6 servers and httpd load balancers across a WAN.

I was trying to implement your suggestion, but I failed in my special case. I´m new on this topic and perhaps you can help me with an example for my problem.
We´ve got two server in WAN.
Let´s say a.)´s IP are 111.111.111.111 and b.)´s IP are 222.222.222.222. Server c.) Should be the load balancer 333.333.333.333

a.) is Master and b.) is slave, connected with JBOSS Domain Mode. Both server running with the same server app. I´m able to communicate with server a.) or server b.) over two Java Socket IO specific ports. Let´s say port 9010 and 9020.

What I now need is a load balancer running on server c.), who knows both servers and is able to check if the communication over the specific Java Sockets is possible or not. So the client should only know the IP of the load balancer and the load balancer communicates with one of the servers and redirects the answer to the client. Another solution would be that the load balancer tells the client the IP of the server he should use for the communication.
I was able to run a normal load balancer who is redirecting me, if port 80 is not available, but I wasn´t able to load balance that Java Socket connection.
Do you have any suggestions or better a tutorial for me how to solve that problem?

We´ve got a chat app for Android. User can send chat messages and pictures. There is one server which can receive queries over TCP (Java socket IO). We´ve got users from different countries and at the moment only one server located in Germany. Our plan was to spread a few servers over EU countries and cluster them for load balancing with geo tagging. We thought it would solve two problems: More speed for the user living outside of Germany and protection against a server failure.

You told me now, that we should build a LAN cluster. Now we could theoretically build several LAN cluster, but that wouldn´t solve the WAN problem. Do you have any suggestions for us?

Second question: Even if I start to build one LAN cluster in Germany, how can I load balance the TCP connections? Like I told you I wasn´t able to achieve this. Do you have any suggestions for me?

Hi Dennis, there are several technologies for WAN cluster. For example, you can use CDN to load balancing your static web contents such as images. And you may need to cache your database into different locations.

btw, if you can't afford to build a distributed systems across different countries by yourself, there are many clouding vendors you could use. For example, Red Hat has Openshift, and you can also try Amazon EC2.

I am facing some problem to post Jms message from Non-cluster Jboss server to the above cluster setup JMS Queue from the Java Code. I have attached my Client code, Mod_cluster config & Jboss configuration file for your reference. Please guide me, how to discover the Cluster group address from Java code,if possible please share me some sample code which I can use to post message to the Cluster JMS Queue.

You can use your own solution for load balancing and failover. Using mod_cluster or mod_jk plus httpd as load balancer of EAP6 servers is just one option. The session replications are implemented between EAP6 servers.

i start one master and one slave on the same pc,and see the register slave log on master console. but when start the server-three on master from the managerment console, an error occur and the host list on the topest menu is empty.

the error says unhandle master server-three.but i see the server-three start succesfully whiout any exceptioins on the console of master.

I have JBoss 7.1.3 in domain mode and mod_cluster 1.1.3 in two servers, master and slave.

Address of master is: 192.168.0.36

Address of slave is: 192.168.0.170

The profile used is full-ha and serves communicate.

I have added <distributable /> in web.xml and tried to configure httpd.conf following this guide but I don't understand how reach apache address and using load balancer. I can only reach master and slave directly (on port 8330) without using load balancer. I have found some example on the web but none of them works! In the httpd.conf I have set the listener on 192.168.0.36:10001 and at URL http://192.168.0.36:10001/mod_cluster-manager there is a mod_cluster page with these informations:

1. Please download the JBoss EAP 6.x and use the mod_cluster 1.2.x to run the tests. The AS 7.1.x and mod_cluster 1.1.x has bugs.
2. The 'EnableMCPMReceive' directive should be used in virtual host.
3. The 10001 virtual host should be used for MCMP channel only.

Here is an example configuration:

As the example shown above, 10.0.1.32 is the IP address of my machine.I have set two virtual machines, and 10.0.1.32:6666 is for httpd mod_cluster module to communicate with backend JBoss servers(via the MCPM channel).

The httpd version I've used is httpd-2.2.25, mod_cluster version I've used is 1.2.6.Final, and with the JBoss EAP6.2.

Btw, I've written a book regarding on these topics if you are interested in. The book name is JBoss EAP6 High Availability

I discovered that problem was caused by my distribution of Linux (CentOS), because httpd_can_network_connect was 0.

Now, with JBoss 7.1.3 and mod_cluster 1.2.6.Final works, but sometimes I have errors as "Exception acquiring ownership", and "org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is javax.persistence.PersistenceException: org.hibernate.TransactionException: JDBC begin transaction failed".

I think that my two servers not have a good communications and I need to configure connection pool.

I have also other questions :D

1. What's the policy used by load balancer to decide which server use?

2. Is possible to configure it?

3. In clustering, there are rules about hardware of nodes (for example must be the same)?

similarly please let me know if there is any such way/module is there which we can incorporate in domain.xml like above , so that we can put the EAR file in any location and then when we start the server it just scans and picksup the file and deploys it.?

When I use this: "<extension module="org.jboss.as.deployment-scanner"/> " it says that we can't use this in managed domain model.

I have finished the most configurations about that ,but I can't got the HA. I have used mod_cluster + apache httpd on master as my cluster controller ,but the request not be served by slave when the master turn off .

Now,i have a question that if the cluster is unavoidable when the master was power off ,so it doesn't work .could i configure the cluster on slave ? or both of them ?

So,can anyone please tell me how to realize the faction of HA ,or the principle of HA?