Remote Debugging of Java Applications on OpenShift

In this article I am going to show you how to attach a debugger and a VisualVM profiler to the Java application running on OpenShift. The approach described here doesn’t make use of theJolokia bridge. Instead, we are going to leverage the port-forwarding feature of OpenShift.

The whole setup can be divided into three steps:

Enable debug and JMX ports on the JVM

Set up port forwarding

Attach debugger and VisualVM to the forwarded ports

I am going to use OpenShift v3.11 that I installed using Minishift and a test application built with Java OpenJDK 1.8. This is how the complete setup is going to look like:

Hello world application

For those of you who want to follow along, let’s set up a test application which we will use for debugging. If you already have your Java application running on OpenShift, you can jump ahead to the next section.

Let’s deploy a Hello world application that I found on GitHub. This application was originally created to demonstrate how to build Vert.x-based microservices on OpenShift. You can get this application up and running in just two steps.

First, issue this command to build an S2I builder image for Vert.x applications:

$ oc create -f https://raw.githubusercontent.com/vert-x3/vertx-openshift-s2i/master/vertx-s2i-all.json
buildconfig.build.openshift.io/vertx-s2i created
imagestream.image.openshift.io/vertx-centos created
imagestream.image.openshift.io/vertx-s2i created
template.template.openshift.io/vertx-helloworld-maven created

OpenShift started the build of the builder image and you can follow the progress with:

This set of options deserves a bit more explanation. By default, JMX utilizes RMI as the underlying technology for the communication between the JMX client and the remote JVM. And as a matter of fact, there are two RMI ports needed for this communication: * RMI registry port * RMI server port

At the beginning, the client connects to the RMI registry on port 3000 and looks up the connection to the RMI server. After the successful lookup, the client initiates a second connection to the RMI server. Based on our configuration, the client is going to connect to 127.0.0.1:3001. However, there’s no RMI server running on the local machine, so what’s the deal? As you will see in the next section, we are going to forward the local port 3001 back to the remote server.

Next, we need to convey our configuration options to the JVM running inside the OpenShift pod. It turns out that there exists an environment variable JAVA_TOOL_OPTIONS that is interpreted directly by the JVM and where you can put your JVM configuration options. I recommend using this variable as there is a great chance that this variable will work no matter how deep in your wrapper scripts you are launching the JVM. Go ahead and modify the DeploymentConfig or Pod descriptor of your application in OpenShift to add the JAVA_TOOL_OPTIONS variable. For example, you can open the DeloymentConfig for editing like this:

$ oc edit dc hello-world

… and add the JAVA_TOOL_OPTIONS environment variable to the container section of the specification:

This verifies that our JVM options are in effect and the debug port and JMX ports are open. How are we going to connect to these ports? Let’s set up port forwarding on the local machine next.

Setting up port forwarding

OpenShift features port forwarding that allows you to connect to an arbitrary port of a pod running on OpenShift. Port forwarding doesn’t require you to define any additional objects like Service or Route to enable it. What you need though is to start a port forwarding proxy on your local machine. Issue the following command on your local machine to start the proxy and forward the three ports 8000, 3000, and 3001 to the remote pod running on OpenShift:

$ oc port-forward <POD> 80003000 3001

In the above command, remember to replace <POD> with the name of your application pod. If everything worked well, you should see the following output :

Attaching to the JVM running on OpenShift

Having our port-forwarding proxy all set, let’s fire up a debugger and attach it to our application. Note that we instruct the debugger to connect to the localhost on port 8000. This port is in turn forwarded to the port 8000 on the JVM:

$ jdb -connect com.sun.jdi.SocketAttach:hostname=localhost,port=8000

After the debugger attaches, you can list existing JVM threads using the threads command:

Next, let’s check out if we can attach VisualVM to our application as well:

$ visualvm --openjmx localhost:3000

Works like a charm, doesn’t it?

Conclusion

In this blog post, we were able to attach a debugger and VisualVM to the Java application running on OpenShift. We didn’t need to deploy Jolokia proxy or create additional Service or Route objects to make our setup work. Instead, we leveraged the port-forwarding feature already available in OpenShift. The demonstrated method has additional security benefits as we are not exposing any additional ports of the application container.

Hope you enjoyed this article and was able to reproduce this setup for yourself. If you have any thoughts or questions feel free to add them to the comment section below.