When I test the JMS queue using JMeter by sending 100 000 messages to queue it throws the following exception on 2gb RAM :-

--- 19:30:23,747 ERROR [org.apache.catalina.connector.CoyoteAdapter] An exception or error occurred in the container during the request processing: java.lang.RuntimeException: java.lang.IllegalStateException

at org.jboss.as.web.ThreadSetupBindingListener.unbind(ThreadSetupBindingListener.java:67) [jboss-as-web-7.1.2.Final-SNAPSHOT.jar:7.1.2.Final-SNAPSHOT]

--- 19:30:23,430 ERROR [stderr] javax.jms.JMSException: Could not create a session: IJ000453: Unable to get managed connection for java:/JmsXA

Like i say, the jca connection factory, java:/JmsXA, is for invm conncetions, the other is a pure hornetq conenction factory that is used for external clients.

refarding your MDB an dservlet, its not really a test, i need something i can run and reproduce your issue with, but like i saifd before if it works with 4GB then its a memory configuration issue and you need to tune your app to deal with it.