About Tal Weiss

Tal is co-founder and CEO at Takipi where he leads a team of hackers that build tools that help fix Java and Scala code in production. His main interests are tools for developers, solving nasty bugs (preferably multi-threaded) and Jazz drumming.

The Dark Side Of Lambda Expressions in Java 8

This post may not make me any new friends. Oh well, I was never really popular at school anyway. But let’s get to the point. Java 8’s biggest feature in terms of the language is undoubtedly Lambda expressions. It’s been a flagship feature for functional languages such as Scala and Clojure for a few years, and now Java has finally joined in.

The second biggest feature (depending of course on who you ask) is Nashorn – the new JVM JavaScript engine that’s supposed to bring Java up to par with other JS engines such as V8 and its node.js container.

I’ll explain. The Java platform is built out of two main components. The JRE, which JIT compiles and executes bytecode, and the JDK which contains dev tools and the javac source compiler. These two components are fairly (but not fully) decoupled, which is what enables folks to write their own JVM languages, with Scala rising to prominence in the last few years. And therein lies some of the problem.

The JVM was built to be language agnostic in the sense that it can execute code written in any language, as long as it can be translated into bytecode. The bytecode specification itself is fully OO, and was designed to closely match the Java language. That means that bytecode compiled from Java source will pretty much resemble it structurally.

But the farther away you get from Java – the more that distance grows. When you look at Scala which is a functional language, the distance between the source code and the executed bytecode is pretty big. Large amounts of synthetic classes, methods and variables are added by the compiler to get the JVM to execute the semantics and flow controls required by the language.

When you look at fully dynamic languages such as JavaScript, that distance becomes huge.

And now with Java 8, this is beginning to creep into Java as well.

So why should I care?

I wish this could be a theoretical discussion, that while interesting, has no practical implication on our everyday work. Unfortunately it does, and in a very big way. With the push to add new elements into Java, the distance between your code and the runtime grows, which means that what you’re writing and what you’re debugging will be two different things.

To see how let’s (re)visit the example below.

Java 6 & 7

This is the traditional method by which we would iterate over a list of strings to map their lengths.

This will throw an exception if an empty string is passed. The stack trace will look like –

at LmbdaMain.check(LmbdaMain.java:19)
at LmbdaMain.main(LmbdaMain.java:34)

Here we see a 1:1 correlation between the stack trace we see and the code we wrote, which makes debugging this call stack pretty straightforward. This is what most Java devs are used to.

Now let’s look at Scala and Java 8.

Scala

Let’s look at the same code in Scala. Here we’ve got two big changes. The first is the use of a Lambda expression to map the lengths, and the second is that the iteration is carried out by the framework (i.e. internal iteration).

val lengths = names.map(name => check(name.length))

Here we really start to notice the difference between how the code you wrote looks, and how the JVM (and you) will see it at run time. If an exception is thrown, the call stack is an order of magnitude longer, and much harder to understand.

at Main$.check(Main.scala:6)
at Main$$anonfun$1.apply(Main.scala:12)
at Main$$anonfun$1.apply(Main.scala:12)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.immutable.List.foreach(List.scala:318)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.AbstractTraversable.map(Traversable.scala:105)
at Main$delayedInit$body.apply(Main.scala:12)
at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
at scala.App$$anonfun$main$1.apply(App.scala:71)
at scala.App$$anonfun$main$1.apply(App.scala:71)
at scala.collection.immutable.List.foreach(List.scala:318)
at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
at scala.App$class.main(App.scala:71)
at Main$.main(Main.scala:1)
at Main.main(Main.scala)

* Remember, this example is very simple. With real-world nested Lambdas and complex structures you’ll be looking at much longer synthetic call stacks, from which you’ll need to understand what happened.

This has long been an issue with Scala, and one of the reasons we built the Scala Stackifier.

And now in Java 8

Up until now Java developers were pretty immune to this. This will change as Lambda expressions become an integral part of Java. Let’s look at the corresponding Java 8 code, and the resulting call stack.

Stream lengths = names.stream().map(name -> check(name));
at LmbdaMain.check(LmbdaMain.java:19)
at LmbdaMain.lambda$0(LmbdaMain.java:37)
at LmbdaMain$$Lambda$1/821270929.apply(Unknown Source)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:512)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:502)
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.LongPipeline.reduce(LongPipeline.java:438)
at java.util.stream.LongPipeline.sum(LongPipeline.java:396)
at java.util.stream.ReferencePipeline.count(ReferencePipeline.java:526)
at LmbdaMain.main(LmbdaMain.java:39)

This is becoming pretty similar to Scala. We’re paying the price for shorter, more concise code with more complex debugging, and longer synthetic call stacks.

The reason is that while javac has been extended to support Lambda functions, the JVM still remains oblivious to them. This has been a design decision by the Java folks in order to to keep the JVM operating at a lower-level, and without introducing new elements into its specification.

And while you can debate the merits of this decision, it means that as Java developers the cost figuring out these call stacks when we get a ticket now sadly lies on our shoulders, whether we want to or not.

JavaScript in Java 8

Java 8 introduces a brand new JavaScript compiler. Now we can finally integrate Java + JS in an efficient and straightforward manner. However, nowhere is the dissonance between the code we write and the code we debug bigger than here.

In this case the bytecode code is dynamically generated at runtime using a nested tree of Lambda expressions. There is very little correlation between our source code, and the resulting bytecode executed by the JVM. The call stack is now two orders of magnitude longer. In the poignant words of Mr.T – I pity the fools who will need to debug the call stack you’ll be getting here.

Questions, comments? (assuming you can scroll all the way below this call stack). Let me know in the comments section.

I was involved in commenting on some very early implementations of lambda functions is Java. I totally agree with you – but it’s even worse than you think.

In languages that are truly functional you can have functions as objects. Java 8 lambdas makes you think that this is the case with method references. Here is a simple abstract example.

Say you have an event producer object to which you can add listeners (maybe they are stored in a HashMap in the producer). With Java 8 you could ass a listener like this (using the new method reference syntax)

eventProducer.addEventListener(this::eventListener)

Later on you might want to dispose of the listener object and in order to make sure that it is garbage collected you will have to inform the producer that you no longer wish to listen to its events. You might call something like this

eventProducer.removeEventListener(this::eventListener)

Where you would expect it to be able to remove the reference to the method stored in the HashMap.

Unfortunately, a massive failing of these method references is that

this::eventListener != this::eventListener

Each reference to a method – even if it is the same method – is a different object. This is awful :-(

IMHO by not enhancing the JVM to embrace function pointers (Brian Goetz dismissed this very early on in the development of Java 8) there is almost no point to using the functional parts. As you have explained the debugging becomes a nightmare, but additionally the language itself becomes inconsistent and has surprises in it – and I’ve always been a fan of the principle of least surprises.

I just learned this the hard way. I wrote a bunch of code expecting removeListener to work, but it doesn’t :/ Unbelievable. There were millions of posts about adding listeners with method references and I loved the possibility… But none of them mentioned you can’t remove the damn thing. This makes method references completely useless in this scenario. I wish I read this comment earlier. So sad :((

You WOULD kind of hope that every time you reference a method, it would be the same object, but it’s not that big of a deal to actually set it into a variable (you know, like almost every other object you use). Make it final if you’re worried that it might accidentally get changed somewhere.

My point of view is that debugger has to show current stack trace in a smarter way, for instance with a small set of entities to be processed in each stream, a counter about how many entries has been processed, watch point and break point to intercept when a filter is applied on an entity… and so on.

Sincerely that is a complex dynamic UI design that has to scale with the complexity of the lambda expression under debug, to avoid a developer to split his expression into parts for debugging, just losing the “benefits” of this new support in the language !

And even if a great debugging interface is available in some great IDE, the trouble remains when a stack trace (NullPointerException, InvalidArgumentException, StackOverflowError… ?) pops up in log files.

Wow. If a little noise in stack traces is the greatest expense, it seems Java 8 Lambdas were very cheap. Java developers have long been used to long stack traces from: web/app servers and various frameworks. The stack traces are very clear about which lines are your application code and which aren’t, so I can’t imagine any Java developer spending more than a couple extra seconds to figure out what the offending lines were.

And, did you know that when you can get an exception, you can get the stack trace data programmatically? You can easily write a stack trace printer that (for example) formats and colorizes a stack trace to highlight which code is interesting. You should even only show lines that were from your app if you wanted. There are libraries that do this now. It’s not hard to add this in to make your life easier.

Clearly having a two line stack trace is preferably to a 13 line one, all other things being equal. But given how easy it is to read even at 13 lines and how easy it is to improve the presentation of those 12 lines should you be so inclined, really the dark side of lambdas is pretty bright.

Plus, there seams to be something that no one is taking into account: almost all of that stack trace has to do with the streams API, NOT the lambda stuff. Yes, there’s a lot going on behind the scenes with streams, but you gotta show me a big stack trace just from lambdas to throw me off. Even that barely will, because of what you said.

Just looked into it: there is only one additional class in the stack trace when it comes to a simple lambda call.

at LambdaStackTraceTest.lambda$0(LambdaStackTraceTest.java:19) at LambdaStackTraceTest$$Lambda$2/1523554304.run(Unknown Source) at LambdaStackTraceTest.referenceRunner(LambdaStackTraceTest.java:29) at LambdaStackTraceTest.main(LambdaStackTraceTest.java:19)

The first one is the actual lambda I wrote, the second is the added thing. The third line is the method calling the lambda, and the last is the main. If you use a method reference, the top line is simply replaced by the method being referenced.

Those huge stack traces are due to the complexity of streams, not because of lambdas.

I would agree with Riccardo that the general developer won’t embrace lambdas. Of course time will tell if this was a good addition to the Java language or not. Perhaps Java has grown too big and we need to remove some more things. Thanks for sharing Tal!

I am learning Scala now in preparation for Apache Spark. I noticed what Tal brings up in the Map Reduce also. Whenever there is an error, the system presents a huge list of errors that scans several pages.

I think Java 8 is very good. I think lambdas are an okay way to save a few lines of code but I don’t see them as super clarifying. Generics are much more confusing and I can only use them in simple way. I learnt Java around 1.5 if I remember correctly, it was somewhat uphill but doable. I wouldn’t like try starting out with Java now. It would take a massive effort to get to a basic industry level. On the other hand javafx 8 is easy and very good for GUI’s and there are a lot of helper classes for multithreaded code that can make it easy to program for. Single thread Java is very safe and well defined, C on the other hand has lots of undefined behaviour that is a tarpit for anyone who hasn’t read and fully understood exactly what you are and aren’t allowed to do from the speifications. I’d like to say that multithreaded Java is safe but I have found on Linux OS level lockups sometimes happen especially when you are manipulating a lot of memory in threads. This also used to happen with browsers in linux where you opened a page with hundreds of images, but they seem to have fixed that. They should maybe talk to browser guys about that one.

Newsletter

Join them now to gain exclusive access to the latest news in the Java world, as well as insights about Android, Scala, Groovy and other related technologies.

Email address:

Recent Jobs

No job listings found.

Join Us

With 1,240,600 monthly unique visitors and over 500 authors we are placed among the top Java related sites around. Constantly being on the lookout for partners; we encourage you to join us. So If you have a blog with unique and interesting content then you should check out our JCG partners program. You can also be a guest writer for Java Code Geeks and hone your writing skills!

Disclaimer

All trademarks and registered trademarks appearing on Java Code Geeks are the property of their respective owners. Java is a trademark or registered trademark of Oracle Corporation in the United States and other countries. Examples Java Code Geeks is not connected to Oracle Corporation and is not sponsored by Oracle Corporation.