Software generalist, backend engineer, bit-herder, building distributed systems on the JVM for FullContact.

Modding the JVM for Fun and Profit: Part 1

Lately I’ve been curious about JVM internals. I’ve been building systems on the JVM for several years now,
and I’ve poked and profiled more than my fair share of services. Increasingly, I find myself wanting to reach inside
the JVM and see more. More importantly, I learn by doing and by dropping logging in places it doesn’t belong, and I
really want to learn what the inside of the JVM is like and start modifying it.

There are organizational benefits to building your own JVM. For example, the Twitter JVM team ships their own
JVM with modified G1GC, heap profiling, performance and bugfixes and all sorts of other changes only hinted at by
their talks. Often times, one might wish to customize behavior to better fit your usecase without necessarily needing
to inject your jar on the classpath -Xbootclasspath/p to override system classes for your production JVMs.

Every so often, my coworkers and I consume a beer or two too many and kick around
statements in the form “I wish the JVM did {X}” or “I wish the JVM didn’t do {X}.” If it was enough beers,
we talk about maintaining our own JVM :). We’d have a few goals such as making GC logging happen asynchronously, or allowing
one to trigger minor GCs from code (there are some reasons to do this.)

Realistically, it’s probably a bad idea to have your own JVM release unless you’re Twitter or Google. Since we are
safely in the realm of enthusiasts, lets build us a JVM.

Goals for part 1 of this journey:

Download the JVM source code

Build JVM source code

Check that our JVM works

Make a change (we’ll change some GC logging)

Run JVM and observe change

I’m working on Ubuntu 16.04, but this should work on any proper Unix. OSX’s
instructions seem a little more finnicky, so I didn’t try. However, I assume it should work with some coaxing.

We’ll start out simple. First off, we need to acquire the JVM source code. Since we’re wanting to make tweaks to
existing stable JVMs, we’ll start with the releases “forest.” It’s called a forest, because it’s a set of
Mercurial subtrees that comprise the JVM. Get it?

Under build/linux-x86_64-normal-server-release/images/j2sdk-image, make has built and collected a full JVM
installation for us. You can see that it works by checking the version. This can be tarballed up or turned
into a dpkg or rpm.

Now down to business. I’ve never liked that -XX:+PrintGCApplicationStoppedTime outputted values in seconds. We’ll
helpfully change this to milliseconds. I have Evan Jones’ mmap-pause repo
handy locally, so I’ll use his
MakeGarbage.java
tool to create some GC activity.

The way the JVM project is laid out, most of the VM code lives under the hotspot/ directory, while the jdk/
directory contains more application-y code, that is, the Java standard library and JNI code to back it. Each tree has
both common and {platform,cpu}-specific code, for example hotspot/src/os_cpu/linux_ppc. We’ll be primarily working in
hotspot/src/share/vm for this mod.

hotspot/src/share/vm/services/runtimeService.cpp

1234567891011

// Print the time interval for which the app was stopped// during the current safepoint operation.if(PrintGCApplicationStoppedTime){gclog_or_tty->date_stamp(PrintGCDateStamps);gclog_or_tty->stamp(PrintGCTimeStamps);gclog_or_tty->print_cr("Total time for which application threads ""were stopped: %3.7f seconds, ""Stopping threads took: %3.7f seconds",last_safepoint_time_sec(),_last_safepoint_sync_time_sec);}

Through some simple grepping, we can find the message in question. We’ll modify the message and properly convert the
quantities to milliseconds.

hotspot/src/share/vm/services/runtimeService.cpp

1234567891011

// Print the time interval for which the app was stopped// during the current safepoint operation.if(PrintGCApplicationStoppedTime){gclog_or_tty->date_stamp(PrintGCDateStamps);gclog_or_tty->stamp(PrintGCTimeStamps);gclog_or_tty->print_cr("Total time for which application threads ""were stopped: %3.7f milliseconds, ""Stopping threads took: %3.7f milliseconds",last_safepoint_time_sec()*1000.0,_last_safepoint_sync_time_sec*1000.0);}

Admittedly, this was a very simple change. In part 2 we’ll explore adding a new runtime method to invoke minor GCs
from client applications, and asynchronously emit GC logging to avoid disk writes adding to our GC pauses. Then,
we’ll package our JDK as a Debian package such that it can become our system’s default Java. Until next time!