Monitoring ActiveMQ Using JMX Over SSH

If you want to know true confusion and frustration I suggest you attempt to monitor ActiveMQ using JMX over SSH port forwarding.

For those who use ActiveMQ, the JMX monitoring is some pretty impressive stuff. You can dive into connections, queues, topics, subscriptions, etc and get stats about the current state of the system. However, this nirvana of information is practically unreachable when AMQ is in a data center. For the first six months or so working on AMQ I was able to get around this using the web console which gives some level of detail on topics and queues but once we moved into trying to debug bigger and bigger problems the console wasn’t cutting it any longer.

The Investigation Begins

As with anything AMQ, I started looking into the JMX documentaion. Step one was to enable JMX in the activemq.xml file.

<broker useJmx="true" brokerName="BROKER1">

Fire it up, forward port 1099 (the default JMX port), give it a try, and, no go. Come to find out, JMX uses two ports, one which is defaults to 1099 and one which is negotiated at runtime. No worries, there is an “Advanced JMX Configuration” seciton in the documentation which lead me to this xml configuration.

Awesome, I can change the default 1099 port but nothing else. The AMQ documentation was failing me, as usual. Google, here I come. After some searching around I was able to dig up AMQ-892. (Power user tip, search AMQ’s Jira site once the documentation has inevitably failed.) It turns out you can configure the RMI server port and have been able to do so since version 4.1.

I particularly liked the “This necessitates a documentation update in the wiki.” comment from 2007. Way to jump on that. Confident that this is bound to work I forward port 11099 and 11109, give it a whirl, and, failure. It was at this point that I started to despair.

Deeper Down The Rabbit Hole

Quickly running out of options I decided to try running the same configuration on a dev box in the office, and of course I could connect to it flawlessly. To add insult to injury I tried another dev box and this one failed. The only difference I could easily see was one was running a VM image and the other was native but this didn’t point at any easily understandable target. I was done. I conceded defeat. I would soldier on without JMX.

Meanwhile, my co-worker had been pressuring me to start packet sniffing with wireshark. I had resisted, mostly because I dislike getting that low level, but also because RMI is a binary protocol and I didn’t think I could learn much. However, I was bloody and beaten at this point and gave in. I downloaded the package, fired it up and started sniffing away. Sure enough the binary was a bunch of gibberish, but tucked away in there was one string, the hostname of the amq server I was trying to connect to. What the hell is this?!? That question is left to the reader but at least the issue was clear, the hostname was inside the data center and not something I could connect to from my machine. It also jived with the test on the local dev boxes. The box with the VM image had an internal hostname that was not in the office dns, where as the native machine had an addressable hostname.

I attacked the problem with renewed vigor. I found a JConsole FAQ on the AMQ website that I had previously dismissed for its cryptic message. However, this time it made sense. I quickly added

-Djava.rmi.server.hostname=127.0.0.1

to the Java Service Wrapper configuration for AMQ so communication would be forced back through the SSH tunnel, rebooted AMQ, and, wait for it, VICTORY.

The Morale of the Story

So the short answer is add the broker configuration, the management context (the second on up there), and the rmi hostname configuration, port forward 11099 and 11109, and fire up jconsole

13 thoughts on “Monitoring ActiveMQ Using JMX Over SSH”

Beauty, mate! I’ve wresting with this for too long. The missing link for me was the “-Djava.rmi.server.hostname=127.0.0.1” switch. Most of the documentation I’ve seen was for running JConsole on the same machine as ActiveMQ, and I’m trying to do it remotely. Popped in my server’s name in place of 127.0.0.1, and wammo! Thanks for posting your pain.

I’m with you if you report the lack of documentation. My additional info here is to set the port differently by disallowing AMQ to create the connector:

Instead, you let the connector be created explicitly by the JVM using (example):
-Dcom.sun.management.jmxremote.port=11119
-Dcom.sun.management.jmxremote.authenticate=true
-Dcom.sun.management.jmxremote.password.file=${ACTIVEMQ_BASE}/conf/jmx.password
-Dcom.sun.management.jmxremote.access.file=${ACTIVEMQ_BASE}/conf/jmx.access
(in the bin/activemq)

That’s how I got around this issue. This is also the solution to enable authentication/authorization if you want to avoid people to connect to your management console and mess around.