Oracle Blog

Jonathan Gibbons' Weblog

jtreg: old and new

jtreg 4.1 is now available, and one of the new features is a new "agentvm" mode. Here, by way of a short historical review, is a comparison of the new agentvm mode against the existing othervm and samevm modes.

In the beginning...

The spec for jtreg calls for tests to execute in a scratch directory which can be cleared before every test. Early versions of Java™ did not provide the ability to change the current directory when executing a command in a sub process, and so the first versions of jtreg were split into a complicated shell script and a Java program using the JavaTest™ test framework. The shell script analyzed the command line arguments, determined the scratch directory to be used, and set it as the current directory before finally executing the test harness itself.

Even back then, there was support for two modes: "othervm", being the ability to run every test in its own JVM, different from (other than) the JVM used to run the test harness, and "samevm", being the ability to run every test in the same JVM used to run the test harness. However, to begin with, the use of "othervm" was strongly recommended, because the risk of tests affecting one another was quite high, and the abilility to recover from over-zealous tests was quite limited. And, although "samevm" was conceptually well defined for running API tests (@run main Foo, etc) it was less well defined for compilation tests (@compile, etc) and there is still some legacy code in javac that was added to work around the limitations (e.g. -XDstdout, to force the compiler to write to the stream used by the jtreg @compile/ref= option.)

And then ...

As jtreg evolved, the maintenance of the wrapper script became increasingly difficult. Eventually, the script was translated into Java code, and merged with the harness itself. However, the issue of the current directory remained an issue, and so after analyzing the command line arguments, jtreg performed a check to see if the JVM being used was acceptable for the execution of the test run itself. This is primarily an issue for samevm mode, when the correct current directory must be set, the correct version of Java must be in use, and so on. If any of the checks failed, jtreg restarted itself using the correct parameters -- the correct version of Java, the correct current directory and so on. In this case, the Java code was behaving as "a better wrapper script", and although there were now potentially two JVMs involved, making the name "samevm" somewhat ambiguous, the name of the mode stuck. (Well, the tests themselves were mostly all executing in the "same VM"...)

And along the way, samevm mode has been improved such that it is now the recommended mode to use for all the tests in the OpenJDK langtools repositories. To be fair, making that happen has involved changes to the test harness, the tests and in a couple of cases, to the tools themselves. But the payoffs have been substantial, and we can now execute over 1700 tests in just one sixth of the time it would using using othervm mode. (11 minutes vs. 66 minutes on my laptop)

But ...

While we have been able to fix the tests in the langtools repositories to use samevm mode, it has not been so easy to find the resources to do the same for the tests in the jdk repositories. Common problems are tests not closing open files when they exit, and tests trying to set a security manager. These conditions are not an issue when running tests in othervm mode, but both are problematic in samevm mode. If files are left open in the scratch directory, on Windows that will prevent the files being deleted, which in turn will cause problems for all the subsequent tests in the test run. Setting a security manager always been forbidden in samevm because originally it could only be set once anyway, and even now, it is possible to set a security manager that you cannot remove. Together, these and other problems, significantly reduce the number of tests in the jdk repositories that can be run in samevm mode, and that can therefore benefit from the corresponding improvement in performance.

Ideally, it would be possible to make sure that most tests can be run in samevm mode. But updating tests is a risky business at the best of times, and even harder when the original authors are no longer available. And so, in practice, it has been easier to improve the test harness...

And so...

The new agentvm mode provides a way to work around these problems. It's still a problem if a test leaves open files when run on Windows, or if it sets a security manager that cannot be unset, but with the new agentvm mode, such problems do not affect the rest of the tests in the test run.

In agentvm mode, jtreg runs in one JVM and creates JVMs with the required characteristics (version of JDK, directory, VM options) as needed to run the tests. More significantly, when each test completes, jtreg will do some housekeeping to restore the JVM involved and the scratch directory to a standard initial state. If any of the housekeeping fails for any reason, the JVMs are allowed to exit. But if the housekeeping is successful, the JVMs are kept available for reuse by any subsequent tests that need JVMs with the same characteristics.

Thus, for "well-behaved" tests, agentvm mode will provide similar performamce to samevm mode, because there will not be any overhead to create a new JVM for each test. But, for less well behaved tests, agentvm mode will automatically degrade to something more like othervm mode, starting new JVMs as necessary for each test.

How many JVMs does jtreg use in agentvm mode at any time? Typically, just two: one to run the test harness, and one to run the tests. That's the same as samevm mode, except it's better, insofar as the JVM used to run the tests will be restarted if there are any problems. There may be three, because agentvm mode allows us to relax the restrictions inherent in samevm mode about not allowing different JVM options or even versions of Java for the compilation and execution of tests. It can do this while still retaining the performance of samevm mode, by creating and using different agent JVMs with the appropriate parameters.

Thus, with agentvm mode, it is now possible to compile tests using a standard JVM but to execute the compiled classes with special VM options, such as for profiling. Previously, this combination was only possible with othervm mode.

And, with agentvm mode, it is now possible to compile tests with a standard version of JDK, and to execute the tests using a version of the Java platform that does not include javac. Previously, jtreg could only test versions of the Java platform that included javac.

Note: if any tests are explicitly marked /othervm, they will obviously be run in their own JVM, separate from the reusable JVMs used for the rest of the tests.

To try out the new agentvm mode, just use the -agentvm option instead of -samevm or -othervm. Or, if you have scripts or Makefiles already set up to run jtreg in samevm mode, there's a backdoor environment variable you can set to use agentvm mode instead, until you're ready to commit any change to your script or Makefile: just set JTREG_USE_AGENTVM_FOR_SAMEVM to true.

Finally...

Note that when you use agentvm mode, it doesn't make the problems of running a test in samevm mode go away -- it just makes jtreg better able to identify and then tolerate such issues. When issues occur, the performance will degrade to similar to that of othervm mode. To keep the performance up to the level of samevm mode, the issues in the tests reported by jtreg will still need to be fixed. But that's a different story...

The parallel javac is especially important now that SSDs are replacing HDDs. I noticed that javac was performing only about %30 faster on a high-end SSD. Making javac multi-threaded, especially IO-wise, should improve the performance substantially.