The nodes are clustered and using an Oracle database as the message store. JNDI port for node1 is 1099 and for node2 is 1199 I have a simple sender and receiver programs. Sender sends 10 messages at 5 second intervals, Receiver picks them up.

When I connect the Sender to JNDI for node1 and the Receiver to JNDI for node2 then I can CTRL-C node1 (I've set the FailoveronNodeLeave to true) and the receiver still gets the messages. BUT... If I have both the Sender and Receiver connecting to the JNDI for node1 and then stop node1, I see errors like:

12:23:48,879 ERROR [ExceptionUtil] SessionEndpoint[ua-xp0jy3pf-1-ix4gy3pf-hm24yq-s2s2o4c5] send [2b-37djy3pf-1-ix4gy3pf-hm24yq-s2s2o4c5]
javax.jms.JMSException: Failed to route Reference[20154309490524163]:RELIABLE to testDistributedQueue
at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:757)
at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:397)
at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87)
at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeTarget(SessionAdvised$send_7280680627620114891.java)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)
at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:122)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)
at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95)
at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:908)
at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:742)
at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:695)
at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:549)
at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:230)

or

2008-12-24 13:14:02,460 WARN [org.jgroups.blocks.ConnectionTable] (OOB-17,127.0.0.1:7900) connection table is not running, discarding message to 127.0.0.1:7901
2008-12-24 13:14:03,384 ERROR [org.jboss.jms.client.container.ClosedInterceptor] (Thread-40) ClosedInterceptor.ClientProducerDelegate[NO_ID_SET]: method send() did not go through, the interceptor is CLOSED
2008-12-24 13:14:03,385 ERROR [org.jboss.messaging.core.impl.clusterconnection.MessageSucker] (Thread-40) Failed to forward message
javax.jms.IllegalStateException: The object is closed
at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:157)
at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.jms.client.delegate.ClientProducerDelegate.send(ClientProducerDelegate.java)
at org.jboss.messaging.core.impl.clusterconnection.MessageSucker.onMessage(MessageSucker.java:269)
at org.jboss.jms.client.container.ClientConsumer.callOnMessage(ClientConsumer.java:229)
at org.jboss.jms.client.container.ClientConsumer$ListenerRunner.run(ClientConsumer.java:1043)
at org.jboss.messaging.util.OrderedExecutorFactory$ChildExecutor.run(OrderedExecutorFactory.java:120)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)

Hi,try to let node1 use ports-01 and nod2 use ports-02. If still fail, can you enable the DEBUG or Trace log level and see if there is more infomation in the log? Normally, if you correctly setup the cluster, failover will be performed when you shutdown one node.

Thanks for the replies.I realised that I forgot to mention that in the scenario when both sender and receiver programs are connecting to node1's JNDI and I then shut down the node, then the sender program crashes with the exception

This is the same exception seen in the server log that I included initially.

I was hoping that the sender would transparently start sending the messages via the other node in the cluster. When the sender and receiver connect to the different nodes' JNDI, it does seem to cope with shutting down the node and the sender continues to send messages.

Sender code is as follows - basically a setup followed by a loop attempting to send 10 messages (some of the code not included):

I then realised that I wasn't trapping the exception in the sendMessage method which meant that it filtered up to main and exited the program.

Obvious really, but still strange as this wasn't an issue when the sender and receiver programs were connecting to different nodes?

Trapping the exception in sendMessage meant that I could attempt to re-send the message. Going down this route I saw a couple of things.

Doing a CTRL-C shutdown I still saw messages either going missing or being duplicated. I still sometimes got the "Failed to Route" error, but at least I could continue to attempt to send messages. Killing the JBoss instance (kill -9) seemed to fail-over OK with no loss of data

So,

are these observations consistent with what would be expected? (Note in both these scenarios both the sender and receiver initially connect to the same node) Why the difference when both programs are initially connected to a single node - when I need to trap the send exception - as opposed to when they are connected to different nodes and that exception doesn't appear to happen?