Why benchmark?

Five questions motivated me to design and run these benchmarks. First, I was curious about how the performance of Java 1.4.2 (the latest official version from Sun) compares to that of Microsoft's relatively new .NET 2003 suite of languages. Both Java and the .NET languages are "semi-compiled" (or, looking at the flip side of the coin, "semi-interpreted"). By this I mean that source code is compiled into intermediate-level code and then run by a combination interpreter/just-in-time compiler. With Java, the intermediate language is called bytecode and the interpreter/compiler is called a Java Virtual Machine (JVM). Source code in the .NET world is compiled into the Microsoft Intermediate Language (MSIL) and is run on the .NET Common Language Runtime (CLR) engine.

The .NET languages benefit from many of the same features that have made Java so popular, including automatic resource management/garbage collection and type safety. They also add interesting new features and conveniences such as cross-language debugging, easy GUI design, and virtually idiot-proof application deployment. But what is the performance penalty of these new features? By adding layers of complexity to its programming model, has Microsoft given up its speed advantage over Java?

Microsoft makes it especially easy to compare the overhead of the Java and .NET frameworks by including J# in the .NET suite. This language is syntactically identical to Java (although it implements only version 1.1.4 of the Java spec, which is by now quite out of date), so any differences in speed between Java and J# should be attributable purely to differences between the Sun and Microsoft runtime overhead.

Second, I wanted to assess Microsoft's claim that the same routine coded in any of the .NET languages is compiled into identical MSIL code which will ultimately run at the same speed. This led me to keep the benchmark very simple, so that I could make sure the routines in each of the .NET languages really were functionally identical. Would all four languages really run at the same speed?

Third, I was curious to see how much slower Java or the .NET languages are than a fully compiled language like C, especially when the C program is unburdened by the runtime overhead of the CLR. I first tried to eliminate the CLR from the Visual C++ benchmark by turning off the language's "managed" features with the #pragma unmanaged directive, but I was surprised to see that this didn't lead to any performance gains. After that strategy failed, I recompiled the Visual C++ program with Gnu's gcc C compiler in order to give C every opportunity to shine in its native, unmanaged, CRL-free form.

Fourth, I wanted to find out how semi-compiled languages compare to fully interpreted languages like Python, Perl or PHP. It is often said that as hardware continues to get faster and cheaper we will reach a point where the extra speed of compiled languages will be largely unnecessary. But if there is still an order-of-magnitude difference between the performance of a routine coded in C and the same algorithm coded in Python, we would be wise to keep our C skills up to date. To test this, I wrote another version of the benchmark in Python. I then re-ran the Python benchmark with the Psyco just-in-time compiler to see if we could combine Python's spectacular readability and rapid development with the speed of a compiled language. Greedy perhaps, but worth a try.

Finally, I thought it would be interesting to see how Sun's latest Java release compares to earlier versions. Sun has makes strong claims about performance improvements in the 1.4.2 version of its compiler and JVM relative to the earlier 1.3.1 release, and I wanted to see if the performance lived up to the hype. So I added Java 1.3.1 to the benchmark roster.