Although Scala compiles to Java bytecode, it is designed to improve on many of the perceived shortcomings of the Java language. Offering full functional programming support, Scala’s core syntax contains many implicit structures that have to be built explicitly by Java programmers, some involving considerable complexity.

Creating a language that compiles to Java bytecode requires a deep understanding of the inner workings of the Java Virtual Machine. To appreciate what Scala’s developers have accomplished, it is necessary to go under the hood, and explore how Scala’s source code is interpreted by the compiler to produce efficient and effective JVM bytecode.

Let’s take a look at how all this stuff is implemented.

Prerequisites

Reading this article requires some basic understanding of Java Virtual Machine bytecode. Complete virtual machine specification can be obtained from Oracle’s official documentation. Reading the whole spec is not critical for understanding this article, so, for a quick introduction to the basics, I’ve prepared a short guide at the bottom of the article.

A utility is needed to disassemble the Java bytecode to reproduce the examples provided below, and to proceed with further investigation. The Java Development Kit provides its own command line utility, javap, which we will use here. A quick demonstration of how javap works is included in the guide at the bottom.

And of course, a working install of the Scala compiler is necessary for readers who want to follow along with the examples. This article was written using Scala 2.11.7. Different versions of Scala may produce slightly different bytecode.

Default Getters and Setters

Although Java convention always provides getter and setter methods for public attributes, Java programmers are required to write these themselves, despite the fact that the pattern for each has not changed in decades. Scala, in contrast, provides default getters and setters.

Let’s look at the following example:

class Person(val name:String) {
}

Let’s take a look inside the class Person. If we compile this file with scalac, then running $ javap -p Person.class gives us:

If any val or var is defined inside the class body, then the corresponding private field and accessor methods are created, and initialized appropriately upon instance creation.

Note that such an implementation of class level val and var fields means that if some variables are used at the class level to store intermediate values, and are never accessed directly by the programmer, initialization of each such field will add one to two methods to the class footprint. Adding a private modifier for such fields does not mean the corresponding accessors will be dropped. They will just become private.

Variable and Function Definitions

Let’s assume that we have a method, m(), and create three different Scala-style references to this function:

How are each of these references to m constructed? When does m get executed in each case? Let’s take a look at the resulting bytecode. The following output shows the results of javap -v Person.class (omitting a lot of superfluous output):

In the constant pool, we see that the reference to method m() is stored at index #30. In the constructor code, we see that this method is invoked twice during initialization, with the instruction invokevirtual #30 appearing first at byte offset 11, then at offset 19. The first invocation is followed by the instruction putfield #22 which assigns the result of this method to the field m1, referenced by index #22 in the constant pool. The second invocation is followed by the same pattern, this time assigning the value to the field m2, indexed at #24 in the constant pool.

In other words, assigning a method to a variable defined with val or var only assigns the result of the method to that variable. We can see that the methods m1() and m2() that are created are simply getters for these variables. In the case of var m2, we also see that the setter m2_$eq(int) is created, which behaves just like any other setter, overwriting the value in the field.

However, using the keyword def gives a different result. Rather than fetching a field value to return, the method m3() also includes the instruction invokevirtual #30. That is, each time this method is called, it then calls m(), and returns the result of this method.

So, as we can see, Scala provides three ways to work with class fields, and these are easily specified via the keywords val, var, and def. In Java, we would have to implement the necessary setters and getters explicitly, and such manually written boilerplate code would be much less expressive and more error-prone.

Lazy Values

More complicated code is produced when declaring a lazy value. Assume we’ve added the following field to the previously defined class:

In this case, the value of the field m4 is not calculated until it is needed. The special, private method m4$lzycompute() is produced to calculate the lazy value, and the field bitmap$0 to track its state. Method m4() checks if this field’s value is 0, indicating that m4 has not been initialized yet, in which case m4$lzycompute() is invoked, populating m4 and returning its value. This private method also sets the value of bitmap$0 to 1, so that the next time m4() is called it will skip invoking the initialization method, and instead simply return the value of m4.

The bytecode Scala produces here is designed to be both thread safe and effective. To be thread safe, the lazy compute method uses the monitorenter/monitorexit pair of instructions. The method remains effective since the performance overhead of this synchronization only occurs on the first read of the lazy value.

Only one bit is needed to indicate the state of the lazy value. So if there are no more than 32 lazy values, a single int field can track them all. If more than one lazy value is defined in the source code, the above bytecode will be modified by the compiler to implement a bitmask for this purpose.

Again, Scala allows us to easily take advantage of a specific kind of behavior that would have to be implemented explicitly in Java, saving effort and reducing the risk of typos.

The Printer class has one field, output, with the type String => Unit: a function that takes a String and returns an object of type Unit (similar to void in Java). In the main method, we create one of these objects, and assign this field to be an anonymous function that prints a given string.

The hidden Hello$.class contains the real implementation of the main method. To take a look at its bytecode, make sure that you correctly escape $ according to the rules of your command shell, to avoid its interpretation as special character:

The method creates a Printer. It then creates a Hello$$anonfun$1, which contains our anonymous function s => println(s). The Printer is initialized with this object as the output field. This field is then loaded onto the stack, and executed with the operand "Hello".

Let’s take a look at the anonymous function class, Hello$$anonfun$1.class, below. We can see that it extends Scala’s Function1 (as AbstractFunction1) by implementing the apply() method. Actually, it creates two apply() methods, one wrapping the other, which together perform type checking (in this case, that the input is a String), and execute the anonymous function (printing the input with println()).

We can see that the anonymous function here is treated just like any val variable. It is stored in the class field output, and the getter output() is created. The only difference is that this variable must now implement the Scala interface scala.Function1 (which AbstractFunction1 does).

So, the cost of this elegant Scala feature is the underlying utility classes, created to represent and execute a single anonymous function that can be used as a value. You should take into account the number of such functions, as well as details of your VM implementation, to figure out what it means for your particular application.

Going under the hood with Scala: Explore how this powerful language is implemented in JVM bytecode.

When a class implements this trait and calls the method isNotSimilar, the Scala compiler generates the bytecode instruction invokestatic to call the static method provided by the accompanying class.

Complex polymorphism and inheritance structures may be created from traits. For example, multiple traits, as well as the implementing class, may all override a method with the same signature, calling super.methodName() to pass control to the next trait. When the Scala compiler encounters such calls, it:

Determines what exact trait is assumed by this call.

Determines the name of the accompanying class that provides static method bytecode defined for the trait.

Produces the necessary invokestatic instruction.

Thus we can see that the powerful concept of traits is implemented at the JVM level in way that does not lead to significant overhead, and Scala programmers may enjoy this feature without worrying that it will be too expensive at runtime.

Singletons

Scala provides for the explicit definition of singleton classes using the keyword object. Let’s consider the following singleton class:

The synthetic variable MODULE$, through which other objects access this singleton object.

The static initializer {} (also known as <clinit>, the class initializer) and the private method Config$, used to initialize MODULE$ and set its fields to default values

A getter method for the static field home_dir. In this case, it is just one method. If the singleton has more fields, it will have more getters, as well as setters for mutable fields.

The singleton is a popular and useful design pattern. The Java language does not provide a direct way to specify it at the language level; rather, it is the responsibility of the developer to implement it in Java source. Scala, on the other hand, provides a clear and convenient way to declare a singleton explicitly using the object keyword. As we can see looking under the hood, it is implemented in an affordable and natural way.

Conclusion

We’ve now seen how Scala compiles several implicit and functional programming features into sophisticated Java bytecode structures. With this glimpse into the inner workings of Scala, we can gain a deeper appreciation of Scala’s power, helping us to get the most of this powerful language.

We also now have the tools to explore the language ourselves. There are many useful features of the Scala syntax that are not covered in this article, such as case classes, currying, and list comprehensions. I encourage you to investigate Scala’s implementation of these structures yourself, so you can learn how to be a next-level Scala ninja!

The Java Virtual Machine: A Crash Course

Just like the Java compiler, the Scala compiler converts source code into .class files, containing Java bytecode to be executed by the Java Virtual Machine. In order to understand how the two languages differ under the hood, it is necessary to understand the system they are both targeting. Here, we present a brief overview of some major elements of the Java Virtual Machine architecture, class file structure, and assembler basics.

Note that this guide will only cover the minimum to enable following along with the above article. Although many major components of the JVM are not discussed here, complete details can be found in the official docs, here.

Decompiling Class Files with javap

Java ships with the javap command line utility, which decompiles .class files into a human-readable form. Since Scala and Java class files both target the same JVM, javap can be used to examine class files compiled by Scala.

That’s a little more interesting. However, to really get the whole story, we should use the -v or -verbose option, as in javap -p -v RegularPolygon.class:

Here we finally see what’s really in the class file. What does all this mean? Let’s take a look at some of the most important parts.

Constant Pool

The development cycle for C++ applications includes compilation and linkage stages. The development cycle for Java skips an explicit linkage stage because linkage happens at runtime. The class file must support this runtime linking. This means that when the source code refers to any field or method, the resulting bytecode must keep relevant references in symbolic form, ready to be dereferenced once the application has loaded into memory and actual addresses can be resolved by the runtime linker. This symbolic form must contain:

class name

field or method name

type information

The class file format specification includes a section of the file called the constant pool, a table of all the references needed by the linker. It contains entries of different types.

The first byte of each entry is a numeric tag indicating the type of entry. The remaining bytes provide information about the value of the entry. The number of bytes and rules for their interpretation depends on the type indicated by the first byte.

For example, a Java class that uses a constant integer 365 may have a constant pool entry with the following bytecode:

x03 00 00 01 6D

The first byte, x03, identifies the entry type, CONSTANT_Integer. This informs the linker that the next four bytes contain the value of the integer. (Note that 365 in hexadecimal is x16D). If this is the 14th entry in the constant pool, javap -v will render it like this:

#14 = Integer 365

Many constant types are composed of references to more “primitive” constant types elsewhere in the constant pool. For instance, our example code contains the statement:

println( "Calculating perimeter..." )

Usage of a string constant will produce two entries in the constant pool: one entry with type CONSTANT_String , and another entry of type CONSTANT_Utf8. The entry of type Constant_UTF8 contains the actual UTF8 representation of the string value. The entry of type CONSTANT_String contains a reference to the CONSTANT_Utf8 entry:

Such complication is necessary because there are other types of constant pool entries that refer to entries of type Utf8 and that are not entries of type String. For example, any reference to a class attribute will produce a CONSTANT_Fieldref type, which contains a series of references to the class name, attribute name, and attribute type:

Field and Method Tables

A class file contains a field table that contains information about each field (i.e., attribute) defined in the class. These are references to constant pool entries that describe the field’s name and type as well as access control flags and other relevant data.

A similar method table is present in the class file. However, in addition to name and type information, for each non-abstract method, it contains the actual bytecode instructions to be executed by the JVM, as well as data structures used by the method’s stack frame, described below.

JVM Bytecode

The JVM uses its own internal instruction set to execute compiled code. Running javap with the -c option includes the compiled method implementations in the output. If we examine our RegularPolygon.class file this way, we will see the following output for our getPerimeter() method:

Each instruction starts with a one-byte opcode identifying the JVM instruction, followed by zero or more instruction operands to be operated on, depending on the format of the specific instruction. These are typically either constant values, or references into the constant pool. javap helpfully translates the bytecode into a human-readable form displaying:

The offset, or position of the first byte of the instruction within the code.

The human-readable name, or mnemonic, of the instruction.

The value of the operand, if any.

Operands that are displayed with a pound sign, such as #23, are references to entries in the constant pool. As we can see, javap also produces helpful comments in the output, identifying what exactly is being referenced from the pool.

We’ll discuss a few of the common instructions below. For detailed information about the complete JVM instruction set, see the documentation.

Method Calls and the Call Stack

Each method call must be able to run with its own context, which includes things such as locally-declared variables, or arguments that were passed to the method. Together, these make up a stack frame. Upon invocation of a method, a new frame is created and placed on top of the call stack. When the method returns, the current frame is removed from the call stack and discarded, and the frame that was in effect before the method was called is restored.

A stack frame includes a few distinct structures. Two important ones are the operand stack and the local variable table, discussed next.

Execution on the Operand Stack

Many JVM instructions operate on their frame’s operand stack. Rather than specifying a constant operand explicitly in the bytecode, these instructions instead take the values on the top of the operand stack as input. Typically, these values are removed from the stack in the process. Some instructions also place new values on top of the stack. In this way, JVM instructions can be combined to perform complex operations. For example, the expression:

The first instruction, dload_1, pushes the object reference from slot 1 of the local variable table (discussed next) onto the operand stack. In this case, this is the method argument sideLength.- The next instruction, aload_0, pushes the object reference at slot 0 of the local variable table onto the operand stack. In practice, this is almost always the reference to this, the current class.

This sets up the stack for the next call, invokevirtual #31, which executes the instance method numSides(). invokevirtual pops the top operand (the reference to this) off the stack to identify from what class it must call the method. Once the method returns, its result is pushed onto the stack.

In this case, the value returned (numSides) is in integer format. It must be converted to a double floating point format in order to multiply it with another double value. The instruction i2d pops the integer value off the stack, converts it to floating point format, and pushes it back onto the stack.

At this point, the stack contains the floating point result of this.numSides on top, followed by the value of the sideLength argument that was passed to the method. dmul pops these top two values from the stack, performs floating point multiplication on them, and pushes the result onto the stack.

When a method is called, a new operand stack is created as part of its stack frame, where operations will be performed. We must be careful with terminology here: the word “stack” may refer to the call stack, the stack of frames providing context for method execution, or to a particular frame’s operand stack, upon which JVM instructions operate.

Local Variables

Each stack frame keeps a table of local variables. This typically includes a reference to this object, any arguments that were passed when the method was called, and any local variables declared within the method body. Running javap with the -v option will include information about how each method’s stack frame should be set up, including its local variable table:

In this example, there are two local variables. The variable in slot 0 is named this, with the type RegularPolygon. This is the reference to the method’s own class. The variable in slot 1 is named sideLength, with the type D (indicating a double). This is the argument that is passed to our getPerimeter() method.

Instructions such as iload_1, fstore_2, or aload [n], transfer different types of local variables between the operand stack and the local variable table. Since the first item in the table is usually the reference to this, the instruction aload_0 is commonly seen in any method that operates on its own class.

About the author

Sergey is an experienced software developer with a background in computer science and basic algorithms and over 20 years of experience. Over the past decade, he has specialized in Java platforms, but he is also fluent in Python and has working knowledge of C/C++ and Scala. [click to continue...]

Sergey is an experienced software developer with a background in computer science and basic algorithms and over 20 years of experience. Over the past decade, he has specialized in Java platforms, but he is also fluent in Python and has working knowledge of C/C++ and Scala.