Java Champion Dick Wall on the Virtues of Scala (Part 2)

This series of interviews spotlights Java Champions, individuals who have received special recognition from Java developers across industry, academia, Java User Groups, and the larger community.

Dick Wall

Dick Wall, a Java programmer for more than 15 years, began writing Java applets using Symantec Cafe on a Mac. He has worked with large simulations of energy distribution systems and with geographical information systems, and was an Android advocate at Google. Currently, he does research into human genetics using Scala at Locus Development. He also partners with Bill Venners at Escalate Software, where he provides Scala training and consulting, and currently he works almost exclusively with Scala. In addition, he is the creator and co-host of the popular Java Posse podcast, and he founded the Bay Area Scala Enthusiasts (BASE) in Silicon Valley.

In Part 1, Wall provided some inside dope on the Java Posse, explored the potential of genetic insight for human health, and shared observations about alternative languages for the JVM. Here we explore his involvement with Scala.

Oracle Technology Network: What path led you to devote so much energy to Scala, a relatively obscure language that is currently ranked somewhere between 51 and 100 in popularity on the Tiobe Index?

Wall: It’s been a long journey to Scala, which has about every element I would include in a language. I taught myself BASIC at a pretty young age when my parents bought me a computer (and we were using them at school as well). In college I learned more-complete languages such as Pascal, Forth, and C (followed by C++). I also crossed paths with some functional languages such as Lisp and Miranda. I remember being interested in Miranda but ultimately finding it frustratingly academic and impractical—so ultimately I dropped it. Lisp I continued to use (mostly in the form of eLisp—embedded in Emacs) for many years, until I came to the conclusion that Lisp was not so much a language as a language construction kit in which all developers created a new language in their own ideal image of a language, so you never really learned Lisp—you just spent each new project learning an entirely new dialect of Lisp constructed by a different programmer.

So I settled on C (and more often C++) as my primary language for development, as a pragmatic choice. My first jobs involved programming in C++ as well. The problem was that C++ kept getting more complex. Much of the simplicity, elegance, and development speed of Pascal (and even sometimes BASIC) was missing, and over time the problem became worse.

In the early 1990s, I discovered two new languages: Python and Java. Python was the one I liked more. It offered both simplicity and productivity, but there were problems. Execution could be slow, the tooling support was not very good, and it lacked a standard graphics UI system. To this day, I still prefer Python to Java for any quick task—particularly file, database, or XML processing—and I often write Python scripts instead of shell scripts to automate tasks.

Java turned out to be a compromise, but a good one, and became my career language for more than 15 years. It was simpler than C++, performance (particularly in the JDK 1.2+ releases) was acceptable (even fast in more recent releases), and it gained a lot of industry acceptance.

“I was delighted to discover that Scala was a pragmatic language that gave me many of the features I remembered (and liked) from functional languages such as Miranda.... I'd found a language that had every element I would have chosen to include."”

Dick Wall Java Champion

However Java (the language) has its own baggage, which has accumulated over the years. Backward compatibility no matter what has indeed cost dearly. Newer languages such as Microsoft’s C# show what would be possible with a Java-like language, but Java has not added any ambitious features for some time. It’s still a good compromise language (as good as it ever was), but we have other options now—many of them pointed out by competing ecosystems such as .NET.

I was always on the lookout for a language that would give me both Python-like productivity and simplicity for just writing something and quickly having it work and that also offers strong performance, toolability, and type safety (all of which I like in Java). Scala is simply the first language that offers all those features in a package that suits me. Programming in Scala feels like programming in Python (if you can think it, you can do it), but with the benefit of having a compiler looking over your shoulder and telling you that you have the wrong type here or the wrong method name there.

The final “aha!” moment came about a year and a half ago. I had a quick task to complete, and I started writing it in Python (as I have for many years) but then realized that I could probably write it just as fast in Scala. I tried, and indeed I managed to write it just about as fast. Some of that is due to familiarity from using Scala all the time, but it is telling that using a statically typed language can be as productive as using a dynamically typed clean room language such as Python.

I was delighted to discover that Scala was a pragmatic language that gave me many of the features I remembered (and liked) from functional languages such as Miranda. It felt like my career programming expedition had finally come full circle. I’d found a language that had every element I would have chosen to include.

Scala is still relatively obscure at present, but it’s getting a lot of interest and already has some big backers. I believe that it will continue to grow in popularity, and I’m not worried about its future.

Pushing the JVM Hard

Oracle Technology Network: You have written that “Scala pushes the JVM hard, perhaps harder than any other mainstream language (particularly if you consider the type system), and this makes it an ideal spyglass through which to examine how the JVM copes with tasks that were often not even conceived when it was first created.” This sounds like a diagnostic point. How might Java developers make use of it?

“Scala, in my experience, does push the JVM harder than other JVM languages I have seen. Scala tries to be more than Java in every way”

Dick Wall Java Champion

Wall: The JVM was originally designed to support code written in Java, and its feature set mirrors the needs of that primary language. Alternative-language designers often need to find creative ways around these missing features when implementing a language to run on the JVM.

Scala, in my experience, does push the JVM harder than other JVM languages I have seen. Scala tries to be more than Java in every way. It is more object-oriented, more functional, more statically typed, more flexible, and more concise, and in many other ways, it pushes the boundaries of what is possible. In doing so, it highlights some shortcomings in the JVM—although not as many as you might think—but the venerable JVM holds up pretty darn well in most respects.

Here are some examples:

Traits – Scala provides traits that can be thought of as interfaces (which is also the case in Java) but that provide both state and behavior (which Java can’t). The result is that some clever work has to be done by the compiler to weave classes that use traits together at compile time, without relying on support from the JVM. This works well in practice but means that if the traits change (as they often do in new Scala library releases), binary compatibility will be broken for those libraries. This leads to the problem that a new release of Scala often requires rebuilds of libraries for that new release.

Type-safe erasure – For Java backward-compatibility reasons, generic type safety is something that is handled by the compiler and is dropped before the JVM gets a hold of the generic container objects. This becomes more apparent in Scala than it is in Java, mainly because Scala’s type system is more sophisticated and because Scala has a feature called pattern matching that enables it to differentiate between a list of strings and a list of ints easily in code but that is practically impossible in the JVM, because that inner type information is lost.

Tail call recursion elimination – Functional languages use recursion as a technique for avoiding mutable state, and many functional languages can do this without any downside (they turn the tail calls into a loop at runtime, so you don’t have the overheads or risks associated with descending the execution stack). Unfortunately, the JVM can’t do this itself. Scala tries but is often dealing with incomplete information at compile time (it doesn’t know what other classes might extend the one it is compiling, for example), so it can miss opportunities to eliminate tail calls.

On the whole, though, the JVM stands up remarkably well to a language that it was not designed to run.

Scala Is Not More Complicated than Java

Oracle Technology Network: Is Scala more complicated than Java?

Wall: Ahh, the big question, and one without a straightforward answer. I believe that programming in Scala can be as simple as or even simpler than programming in Java and can lead to code that is more readable, concise, and understandable. It can also provide more flexibility and scope for better solutions to many tasks. At least some of the claims that the Scala language is complicated are rooted more in unfamiliarity with the language, compared to a known quantity such as Java, than in technical concerns about language complexity. So is Scala more complicated than Java? The answer is a resounding “No.”

I suggest that people look at the Kojo project to see just how simple Scala can be.

That’s not to say that it can’t be more complicated in some cases. Scala gives you a lot more flexibility and power than Java does, and if you choose to exercise all that flexibility and power, you will need to know more than you would with a language such as Java. For example, Scala enables developers to extend the actual compiler with compiler plug-ins, which involves manipulating the abstract syntax tree created by the compiler—and that’s quite complicated. But not everyone will have to do that to use Scala.

Most of our tasks as developers are getting progressively more complex. If they weren’t, something would be wrong with progress. When I started programming years ago, a program interacted with people via text interface, multiple execution threads weren’t a consideration in most programs, and the average program size was under 100 KB. These days I work on systems that interact with Web UIs; multiple Web services, databases and datasources; and far larger and more complicated data running on machines with gigabytes of memory and multiple cores.

Given that the complexity bar for a given project is raised to a certain height, I choose to use a language and a set of libraries that will get me as close to that bar as possible so that the last few feet are all I need to worry about. Using Java leaves quite a gap for clearing that bar. Adding in libraries such as Wicket or Camel can help close the gap more (but I would say the learning curve for some of these libraries is every bit as high as that of the more advanced Scala features), and ultimately you are still going to have to work harder for those last few feet.

Escalate Software

Oracle Technology Network: You partner with Bill Venners to teach Scala to developers through your company, Escalate Software. What training do you offer?

Wall: Bill and I both were reasonably early adopters of Scala and are both self-taught. It took a long time for both of us to pick up the language, especially coming from Java, and the training is intended to accelerate that learning experience for others and make it easier to get up to speed. Also, I discovered while working as a developer advocate at Google that I like to teach, although I wouldn’t want to do it full-time. Bill is a natural teacher too, so it just made sense for us to give it a try.

We offer training from the ground up in Scala. We assume no Scala knowledge ahead of time for our “Applied Fundamentals” course, which takes three days, is very hands-on (participants spend as much time programming in Scala as learning about it), and gets developers up to a level of competence that enables them to use Scala and the standard libraries on real projects. The course also covers the Scala IDEs (we concentrate on IntelliJ for the training, but the skills are transferable to other IDEs). The only requirements for that course are some kind of programming language skills. We aimed the course particularly at those with skills in Java-like languages such as Java and C#, but we have successfully trained many different language programmers, including Smalltalk, Ruby, Python, and Perl programmers. There is an emphasis on functional programming techniques as well, because that tends to be the new skill set for most of the programmers who take the course, but we cover object-oriented features of Scala as well.

We now follow up those two days of applied fundamentals with our advanced course, which is two days long and assumes a working knowledge of Scala. People can choose to do either course, or both together in one week.

Scala Is More Productive than Java

Oracle Technology Network: On your Escalate Software site, you say, “Developers in Scala typically find themselves several times more productive than in Java on large projects once the initial learning barrier is crossed, and we can help with that.” This is a very strong claim. What is it based on?

“I can put my hand over my heart and say that I am much more productive in Scala than I was in Java, and I see no reason why the many people I know using Scala wouldn't say the same without some reason.”

Dick Wall Java Champion

Wall: The claim comes partly from my own experience and partly from talking with other Scala developers (I know a lot of them from my community work). In my personal experience, I am now as productive in Scala as I am in Python, my previous most productive language, meaning that I can turn around non-trivial projects maybe three times as fast as I could do the same projects in Java. Other Scala developers I have talked to claim that within a month or two of starting to use the language, they become as productive in Scala as they were in Java and rapidly improve on that. Of course, people are always going to argue about these claims, but I can put my hand over my heart and say that I am much more productive in Scala than I was in Java, and I see no reason why the many people I know using Scala wouldn’t say the same without some reason.

Here are some key features of Scala that improve performance:

Functional programming features – Function literals, closures, pattern matching, recursion, and other easily available language features improve productivity. Ask anyone who has used a language with closures how much they miss them in Java, and you will probably get quite an extensive answer.

Boilerplate busters – Case classes, traits, and other advanced features can all but eliminate the need to repeat yourself anywhere in the code. You haven’t experienced the full power of refactoring in a statically typed language until you have done it in a language with traits. Case classes eliminate all noise from value objects as well (like equals and hashCode implementations) and simultaneously cut down a common source of errors.

Concise code – My Escalate partner Bill Venners likes to quote “Vigorous writing is concise” from Strunk and White, meaning that the message, not the ceremony, is what’s important. Java has a fair amount of ceremony in its coding—Scala has more focus on the message.

All these are true for dynamically typed languages such as Groovy, Python, and Ruby too. The difference is that Scala also has the compiler (or often the IDE), which taps you on the shoulder from time to time and tells you, “You got that bit wrong,” rather than letting it fail at testing or runtime. That’s another serious time saver.

Oracle Technology Network: You state that “Scala also works with all Java libraries and frameworks, even existing project code, often improving on their APIs in the process.” Can you give examples of how it improves on some APIs?

Wall: Scala runs on the JVM and is in almost every way a superset of the Java feature set. This means that any library you can use from Java you can use in Scala as well, although sometimes you won’t need to.

Scala has features such as function literals and traits that Java does not have, and these can lead to more-powerful APIs for using libraries. A common activity for widely used libraries is that a developer wraps the Java library with a light Scala wrapper that enables you to use some of those features to make the API more concise. Perhaps the most extensive example is in the parallel collections included in Scala 2.9 that use the parallel arrays and fork/join work from JSR 166 behind the scenes, but with a far smaller and simpler API. Other examples are GUI libraries that are often extended to use closures for event handlers (Scala Swing, for example), among many others. These adapters can be pretty small but can lead to impressive productivity and simplicity improvements. For example, go here for some examples.

SubCut: Open Source Bioinformatics

Oracle Technology Network: Do you have any open source projects you would like to share with Java developers?

Wall: I have one, called SubCut. A sort of cross between dependency injection and service location written in Scala, it was written for use at Locus Development, but I thought it was useful enough to open-source it. Over time we hope to develop more, but a lot of what we do is very specific to problems we are solving and unlikely to be useful to others. That said, I think we will be contributing back to a lot of other open source bioinformatics projects out there.