My specific question has to do with JMX as used in JDK 1.6: if I am running a Java process using JRE 1.6 with

com.sun.management.jmxremote

in the command line, does Java pick a default port for remote JMX connections?

Backstory: I am currently trying to develop a procedure to give to a customer that will enable them to connect to one of our processes via JMX from a remote machine. The goal is to facillitate their remote debugging of a situation occurring on a real-time display console. Because of their service level agreement, they are strongly motivated to capture as much data as possible and, if the situation looks too complicated to fix quickly, to restart the display console and allow it to reconnect to the server-side.

I am aware the I could run jconsole on JDK 1.6 processes and jvisualvm on post-JDK 1.6.7 processes given physical access to the console. However, because of the operational requirements and people problems involved, we are strongly motivated to grab the data that we need remotely and get them up and running again.

EDIT: I am aware of the command line port property

com.sun.management.jmxremote.port=portNum

The question that I am trying to answer is, if you do not set that property at the command line, does Java pick another port for remote monitoring? If so, how could you determine what it might be?

6 Answers
6

The documentation suggests that the JMX agent uses a local port -- something unreachable from outside the machine -- unless you specify the following property:

com.sun.management.jmxremote.port=portNum

This is for security reasons, as well as for the reason given by Mr Potato Head. Thus, it looks like Java 6 does not open a default remotely accessible port for JMX.

EDIT: Added after the OP added an answer with more information.

Another option you have is to somehow create a local proxy that listens to all local JMX connections and exports this information. This way, you don't need to have such magic configuration of each JVM instance on the server. Instead the local proxy can connect to all JVMs via JMX and then somehow expose this information remotely. I am not positive exactly how you would implement this, but something like this may be less work than what you otherwise have to do to expose all of your JVMs remotely via JMX.

The protocol connecting JMX client and JMX server is assumed to be 'Java RMI' (aka 'RMI-JRMP'). This should be the default. One can configure other protocols, in particular 'RMI-IIOP' and 'JMXMP'. Special protocols are possible: the MX4J project for example additionally provides SOAP/HTTP and various serialization protocols over HTTP.

Also have a look at the file jre/lib/management/management.properties in your JDK distribution.

So, the possibilities:

Case 0: The JVM is started without any particular configuration

Before Java 6: The JVM does not behave as a JMX server. Any program that is run inside the JVM may access the JVM's MBeanServer programmatically and use it to do interesting data exchanges between threads or to do JVM monitoring, but no management from outside the JVM process is possible.

Since Java 6: Even if not explicitely configured, one can access JMX functionality of the JVM locally (from the same machine) as described in "Case 1".

Case 1: The JVM is started with -Dcom.sun.management.jmxremote

The JVM is configured to work as a local (same-machine-only) JMX server.

In this case (and in principle only for Sun/Oracle JVMs) a JMX client can connect to the JMX server through memory-mapped files found in /tmp/hsperfdata_[user]. This is alluded to in the Sun documentation and called "local monitoring" (and also the Attach API). It does not work on FAT filesystems as permissions cannot be set correctly there. See this blog entry.

Sun recommends running jconsole on a machine separate from the JMX server as jconsole apparently is a resource hog, so this "local monitoring" thing is not necessarily a good idea.

Local monitoring is, however, rather secure, only being usable locally and being easily controlled through filesystem permissions.

Case 2: The JMX server is started with -Dcom.sun.management.jmxremote.port=[rmiregistryport]

The JVM is configured to work as a JMX server listening on several TCP ports.

The port specified on the command line will be allocated by the JVM and an RMI registry will be available there. The registry advertises a connector named 'jmxrmi'. It points to a second, randomly allocated TCP port (an 'ephemeral' port) on which the JMX RMI server listens and through which actual data exchange takes place.

Local as described in 'Case 1' is always enabled in 'Case 2'.

The JMX server listens on all interfaces by default, so you can connect to it (and control it) by locally connecting to 127.0.0.1:[rmiregistryport] as well by remotely connecting to [any outside IP address]:[some port] remotely.

This implies that you have to look at the security implications. You can make the JVM listen on 127.0.0.1:[rmiregistryport] only by setting -Dcom.sun.management.jmxremote.local.only=true.

It is rather unfortunate that one cannot specify where the ephemeral port will be allocated - it is always chosen randomly at startup. This may well mean that your firewall needs to become the swiss cheese of the damned! However, there are workarounds. In particular, Apache Tomcat sets the ephemeral JMX RMI server port via its JMX Remote Lifecycle Listener. The code to perform this little magic can be found at org.apache.catalina.mbeans.JmxRemoteLifecycleListener.

You can do interesting permutations to avoid having to use the RMI protocol. In particular, you could add a servlet engine (like Jetty) to your process. Then add servlets that translate some HTTP-based exchange internally into direct accesses to the JVM's MBeanServer. You would then be in 'case 0' but still have management capabilities, possibly through an HTML-based interface. The JBoss JMX Console is an example of this.

More off-topic, you could use SNMP directly (something I haven't tried) according to this document.

Show and Tell Time

And now it's time for some code to illustrate a JXM exchange. We take inspiration from a Sunoracle tutorial.

This runs on Unix. We use a JVM that is configured as a JMX server using:

-Dcom.sun.management.jmxremote.port=9001

We use lsof to check what TCP ports it is holding open:

lsof -p <processid> -n | grep TCP

One should see something like this, the registry port and the ephemeral port:

The documentation seems to indicate that the JMX agent uses a local ephemeral port, unless you specify the following property:

com.sun.management.jmxremote.port=portNum

Default ports are avoided because you could have many java applications on one system, and if there was a default port, only one application would be able to be managed! The above configuration property is provided for the express purpose of remote management.

If you must insist on using an ephemeral port, then the URL of the JMX agent should be accessible from within the JVM, through the following system property (although this is likely to be a local address):

com.sun.management.jmxremote.localConnectorAddress

Note: I guess you could always open a socket on a remotely-available address and proxy requests on to the local socket, but using the available option seems far more attractive!

@Mr Potato Head: I understood what you meant by unprivileged. I also understand that the port number has no bearing on whether it is externally reachable. However, you're right that you can open a port (privileged or not) in such a way that it is not externally reachable, only locally.
–
EddieFeb 10 '09 at 2:51

However, it's interesting to examine why. Look at the netstat output from a valid local connection. Here are the ports that I see opened up as a result of a jconsole making a local connection to itself. As you can see, port 1650 is the local port being used for the JMX information:

However, it's not sufficient to try to connect jconsole to localhost:1650. Sadly, all that will net you is a "Connection failed: no such object in table" message.

So, the conclusion of my original story is that, if we are going to facilitate remote monitoring using JMX for our customers, we really need to identify unique individual remote access ports for the variety of Java processes that are started in our system. Fortunately, all this requires is the judicious use of the VM argument:

com.sun.management.jmxremote.port=portNum

where we will almost certainly have a sequential pre-specified range of portNum so that the customer can select the correct remote application using the port number.

I've been working recently to figure out how to enable remote JMX management from java code, without requiring the JVM to have been started with special properties set. The solution I settled on is to start my own private RMI registry -- easy enough -- and to expose the JMX service on that registry. I create my own MBeanServer, then create a new JMXConnectorServer. The JMXConnectorServer is created through a call like