How to Trace a Java Application Running on Oracle Solaris

How to combine JSDT and BTrace to dynamically trace a Java application running in an Oracle Solaris environment without changing the source code and without impacting performance.

Published April 2012

The DTrace feature of Oracle Solaris is known for its broad ability to look at almost anything going on in a computer running Oracle Solaris 10 or above (or another OS that has adopted DTrace). Java applications can be traced by DTrace, but there used to be some limitations and restrictions in tracing Java code.

There are many good tracing tools for Java, for example, jvisualvm, which is included in the JDK release, is very handy and provides rich capabilities. Still, most of these tools lack the combination of extreme dynamism, nonintrusiveness, and broad capabilities of the DTrace framework.

Java Statically Defined Tracing (JSDT), which follows User-level Statically Defined Tracing (USDT) for C/C++ code, enables programmers to statically add probes to their code. Those probes, while not activated, do not impact application performance and, when activated, are designed to have minimal impact. This opens the door for the broad DTrace observation scope and its optimal aggregation capabilities.

There is still a potential barrier: the need to add these probes to the code—Java code, in the JSDT case. But Java brings some new capabilities, which do not exist for native languages. For example, you can redefine program classes during the program run. Also, since Java Platform, Standard Edition 6 (Java SE 6), the Attach API enables attaching to any running Java SE 6 or higher JVM and dynamically initiating and executing an agent inside the attached JVM. By combining all this, we can theoretically do dynamic instrumentation of JSDT probes in Java code! That is the topic of this article.

What We Want to Do

Figure 1 shows an example to illustrate the idea. In this example, we would like to explore some behavior without stopping the application, changing the code, or recompiling the application.

Figure 1. Example of Exploring the Behavior of a Java Application

Current Requirements, Limitations, and Caveats

JSDT is supported on Java Hotspot VM 1.7. Use version 1.7.0_04 to avoid an issue of failure to create the first provider. I tested the process described in this article on Oracle Solaris 11, and everything should work properly on Oracle Solaris 10; however, I did not check other operating systems that support DTrace and I didn't check other JVMs.

The BTrace client connection initiation to the target application fails from time to time. Just try again if it fails.

BTrace currently does not clean ("de-instrument") the instrumented classes; it just de-activates the probes. This behavior prevents repeated instrumentation of the same probes and same classes. I hope cleaning will be implemented soon.

JSDT Basics

Defining JSDT probes is easy to do but requires a simple initialization. The first step is to define Java interfaces for each provider (extends com.sun.tracing.Provider). The interface methods will also be the corresponding DTrace probe names, as shown in Listing 1.

How to Dynamically Instrument JSDT Probes

We saw how to define the probes statically by adding them to our source code. Now let's try to do the same thing without changing the source code. To use the Attach API and the Java code instrumentation capabilities, we can take advantage of the great BTrace package, which provides a rich set of dynamic tracing capabilities that try to follow the DTrace standards while tracing Java applications. BTrace has very high value by itself, but here we'll just "take a ride" by using it with our JSDT stuff to instrument a running application.

BTrace Basics

BTrace is a tool that is built on the efficient ASM byte code framework. BTrace lets you dynamically instrument running Java application classes by using special Java annotations. The annotations act as directives to indicate where tracing code should be inserted in the target application, for example:

BTrace creates and invokes an agent in the target JVM (through the Attach API), and then it uses a client to communicate with that agent to perform the instrumentation and to get output tracing data, if desired.

In its default "safe" mode, BTrace puts some restrictions on the injected code to avoid potential undesired side effects on the target JVM. One of the main restrictions is no calls to methods other than BTrace library methods. (BTrace provides a rich set of methods in its utils package.) In our case, we will use the "unsafe" mode, so we can define and call the DTrace provider class methods.

Listing 2 is a kind of "Hello World" BTrace script that instruments the Thread.start() method entry and prints a message. The script does not call any external to BTrace methods, so it can be compiled in safe mode. However, we will later issue calls to JSDT to provide class methods, and we will have to use unsafe mode then.

The DTrace provider classes are required for both the BTrace script compilation and the target application runtime. The providers need to be initialized prior to the first call to a provider method. The best method is to initialize the providers during the class instrumentation. This can be done with static initialization. Static initializers contained in the BTrace class we define do not work properly, but importing a factory class with static initialization works, at least lazily. A provider factory class with static initialization might look like Listing 3.

The static code is lazily initialized at least just before the first probe method call.

Triggering the Probes from a BTrace Script

In order to trigger JSDT probes from BTrace code (as of BTrace 1.2), we need to do some setting and tweaking:

Change to BTrace unsafe mode by editing the <Btrace-install-dir>/bin/btrace script and changing -Dcom.sun.btrace.unsafe=false to -Dcom.sun.btrace.unsafe=true.

Add the providers to the BTrace compilation class path by adding the providers' JAR file to the -cp chain in the Java invocation command at <Btrace-install-dir>/bin/btrace. If you also use any other classes in the BTrace script (such as target application classes you would like to refer to), you need to add them, too.

The target Java application loads our providers with the boot class loader. Therefore, we need to make the providers reachable to the boot class loader. Unless you are doing something else with the boot class loader, you have these options for adding to the boot class path:

The easiest way, which does not involve adding a flag to the target application, is adding the provider classes to the classes directory under your active JRE (jre/classes), for example, /home/ahurvitz/java/jdk1.7.0_04/jre/classes. If a classes directory does not exist there, create it. To verify that the target application class path includes this directory by default in the boot class path, you can run the JDK jinfo command like this:

# jinfo -sysprops <target-java-pid> | grep "sun.boot.class.path"

The other option involves adding a Java option (flag) to the target application, which is less convenient. To do this, add a -Dsun.boot.class.path=<current-boot-class-path>:<providers-jar> flag to the target application. Replace <current-boot-class-path> with the value of the sun.boot.class.path property, which you can retrieve by running the following command:

The extremely simple Java program in Listing 4 will be used as the target application to trace in the following example. Assume that we'd like to trigger DTrace probes for every entry into and exit from the makeOneIteration() method, passing a counter object as a parameter.

After compiling this program, we'll run it. The program kindly prints its process ID (pid) for the next steps. This pid is the <target-java-pid>, which we will refer to while running the btrace command.

Now that we have a program to trace, let's start. (It is assumed that nothing has been installed yet.)

Configure BTrace by editing <Btrace-install-dir>/bin/btrace, as follows. (It is recommended that you save a copy before you edit the file.)

Change unsafe mode from false to true:

${JAVA_HOME}/bin/java ... -Dcom.sun.btrace.unsafe=true ...

Add the provider's JAR and any other class you use in the BTrace script, for example, objects you refer to) to the boot class path. In the following example, we are also adding the TraceTarget class, because we are referring to its object in the BTrace script.

Summary

DTrace, JSDT, and BTrace provide an easy way to dynamically instrument a Java application with DTrace probes, which opens up many possibilities for ad hoc exploration of running Java applications on Oracle Solaris 10 and above.

See Also

About the Author

Amit Hurvitz has worked on the ISV Engineering team at Oracle Hardware Engineering (formerly Sun Microsystems) for 10 years. Prior to that, he worked on C compiler optimizations and was a C++ and Java Platform, Enterprise Edition developer.