In this interview, Neal Gafter talks with Bill Venners about the proposal for adding
closures to the Java programming language.

Last Thursday, December 14, 2006, Neal Gafter gave a presentation entitled Closures for Java at the
JavaPolis conference in Antwerp, Belgium. Neal Gafter is a software engineer at Google, where he works on
Google Calendar. Together with Gilad Bracha, James Gosling, and Peter
von der Ahé, he has also been working on a proposal to add closures to Java. In his JavaPolis talk,
he described the goals of adding closures to Java, compared them to anonymous inner classes, and
explained and showed the proposed new syntax. Shortly after his talk, Gafter spoke with Bill Venners about
the proposal for adding closures to Java.

Using Closures for Control Statements

Bill Venners: What problems do closures try to solve?

Neal Gafter: What I'm try to accomplish with closures is to allow you to
write methods that are part of APIs that act like control statements in the
language. Because a lot of APIs do have specialized needs. You have the
concurrency framework where you'd really like something that acts like the
synchronized block, but for the concurrency locks. We did add a statement in
JDK 5, the for-each loop, which is specially designed to work with collections.
There are a lot of APIs that would benefit from having statement forms, and
there are proposals to add even more statement forms in JDK 7. My hope is
that by adding one language construct, we won't have to have a never-ending
series of additional statement forms because we keep having more and more
APIs that we care about. Instead, we can have one language extension and
those APIs can include those statement forms.

Bill Venners: One of the examples you gave us in your talk was
withLock. Can you elaborate on that?

Neal Gafter: withLock is a method that I hope would be
added to the concurrent framework. It basically looks just like a
synchronized statement. You say, withLock,
open-paren, then a lock variable, close paren, open curly brace, then a bunch of
statements, and then close curly brace. It looks just a like a built-in statement
form, but it's a method invocation, an invocation of this withLock
method. And the withLock method takes two parameters, the
lock, and a closure. It does the sort of boilerplate that you would have to do to
synchronize a block of code using the concurrency framework yourself, but it's
much more convenient to use.

The withLock example is a poster-child for closures. If you don't have closures,
and you're never going to have closures, you really want to put something like
that in the language. But, again, it's a never-ending series of language
additions.

Bill Venners: What makes a block of code a closure, instead of just a
block of code?

Neal Gafter: The syntax I just described, where you have a function
declaration that looks like it has a block of code after it, that's not legal syntax
today. That is a new syntax in the language, and we would define that syntax to
mean that you take the block at the end and wrap it in a closure and pass it as
an extra parameter to the method.

There is also syntax where you can write closure literals, where you can
explicitly pass two or three of them as parameters to a method, and that's a
new syntax too. The way you write a closure is, open curly brace, the argument
declarations, an arrow (equals-greater-than), a bunch of statements, and then
a closed curly brace—that's a syntax that's not in Java right now. That
would be the way you write a closure literal in the language. It would be
wrapped by the compiler in an interface type when it's passed to the method
to which you pass it in as a parameter.

Bill Venners: In your talk, you highlighted the differences between
closures and anonymous inner classes, such as the meaning of this and
return. When you have a closure, you can return from...

Neal Gafter: ...the enclosing method. Actually, there are some places where there are no
enclosing methods. If you wouldn't be allowed to write a return anyway, then
you won't be allowed to write a return inside of a closure. But a return from a
closure simply returns from the enclosing method. The fact that it's inside of a
closure doesn't change the meaning of the return statement.

Closures Under the Covers

Bill Venners: What happens under the covers? In your talk, you showed
the interface that has an invoke method in it. Could you describe what really
happens?

Neal Gafter: The closure is turned at compile time into an instance of
the interface. If you take the example of the withLock method,
that method is declared to take a function type as its last parameter. At compile
time, the compiler looks at the closure and says, oh, I actually need something
of this interface type. (Function type is an interface type.) So it builds an
instance of that interface, just like an anonymous inner class, but it's not an
anonymous inner class, because the scoping rules for a closure are different
from an anonymous inner class's scoping rules. It creates an object that
implements that interface where the body of the invoke method
does what the code inside the closure says it should do. And it passes that
instance as an extra parameter to the withLock method. And
inside the withLock method, in order to cause that code to be
executed, you have some parameter, like block—you just
say block.invoke(). That's all you need to do, and it simply calls
the invoke method on that interface from the object that was
passed in.

Bill Venners: If that block executes a return, how does it return from the
enclosing method, which called withLock?

Neal Gafter: There are actually a few different answers to that question.
The simplest answer is that a return from a closure will be implemented using
exceptions. So the compiler will create and throw an exception, and that will
pass out through the invocation of withLock to the caller that
was calling withLock, where it will be caught. And the value that
you're returning will be pulled out of the exception, and the code
that catches it will actually do the returning from the enclosing method.

As a practical matter, if you use that pattern, where you put a return inside of a
closure, it will get inlined by HotSpot, and it will turn into a jump, which is exactly
what it would have done if you used the statement in the language instead of a
closure. If the closure only executed once, it won't, but if the code is executed
more than a couple of times, and the return is executed very frequently in the
closure, then HotSpot will inline that. And it will turn into the same code at the
machine code level that it would have, if you had done it via a control statement
added to the language.

Using Closures with Existing APIs

Bill Venners: One thing I really liked in your presentation was about
methods that exist in existing APIs today that you could pass a closure to, even
though they never imagined that would be possible.

Neal Gafter: Let's take a particular example. In the concurrency
framework there is an interface called Executor. It represents a
thread pool, among other things. One of the things you typically do with a thread
pool is you submit a Runnable to it. It takes the
Runnable, and assigns it to a thread to run now, or as soon as
a thread is available. That method is called execute. If you want
to use this method, but you want to use a closure, because of the convenience
of the syntax, you can. Remember I said the closure conversion takes the
closure and figures out what kind of interface type it needs and that's the kind of
interface that's built at compile time to be passed it. So if you say,
executor.execute(), and then open-close curly braces, and write a block
of code inside the curly braces, that block of code is turned into a closure. It's
turned into an object that implements Runnable, and the
run method does whatever that block says it should do. That
object is passed to the execute method of the executor
framework.

No change needs to be made to the executor framework because a closure is
always converted to whatever interface type you need statically in the context of
the invocation of the code. You can use a closure in place of a
Runnable, you can use it in place of a Callable—whatever
interface you need—you can write a closure literal in that place and
pass it to that API.

Bill Venners: So if make a new API after closures are in the language,
and I say explicitly that I take a closure...

Neal Gafter: It's just a parameter, right? All you have to do is declare it
as the last parameter of the interface.

Bill Venners: So all I have to do to be ready for a closure being passed
to me is that the last parameter is an interface?

Neal Gafter: You can receive a closure in any parameter position. The
only advantage of having it in the last parameter position is you get to use the
special syntax that looks like a statement. That's the point. If it is the first
parameter instead of the last parameter, it's no problem. But you would use the
closure-literal syntax with the fat arrow instead of putting a block after the
invocation. But you can do that. In fact, if you have an API where you need to
pass in more than one closure, you have to use the closure literal syntax.

Bill Venners: Basically any existing method call at the end, if it's an
interface... It has to be an interface, can't be a class?

Neal Gafter: It has to be an interface, and an interface with one method
in it.

Bill Venners: So that narrows it down a little bit the existing APIs that
will be able to be called with closures.

Neal Gafter: It does, but there a lot of those—all of the call-back
registration stuff in Swing, the executor framework, and so on. There are
Callables, Runnables—a lot of places
where you could take advantage of this, even
java.util.Comparators that you use for sorting. You can write a
block of code that describes the Comparator right in the
sort invocation. That works just fine. In many of the APIs where
people currently use anonymous instances in the invocation, you can use
closures instead—but not all of them. If you have something where you
really need more than one method to be in there, you can't easily use closures
for that. Actually, you can, but not easily. You have to write a method that makes
an instance of a class and then invokes the closures for each one. It's not
pretty. It's really mainly suitable for the ones that have a single method.

Exception Transparency

Bill Venners: Tell me about exception transparency.

Neal Gafter: The idea of exception transparency is that sometimes you
have an API to which you pass in a closure, and if the closure throws an
exception, you want your method to be known to throw the same exception so
that exception checking works.

A really good example of that is the withLock that works with the
new locks. If the block of code in the withLock statement throws
some exceptions, you want the compiler to know that those exceptions are
going to be propagated out from the withLock call. In the
closure specification, therefore, we've added some stuff to the machinery of
generics. It's only a little bit; it's not very complicated. We added just enough to
make it possible for you to express that the exception propagates out.

Bill Venners: How does this affect the way checked exceptions work?

Neal Gafter: If the block throws, say, a
NumberFormatException, then the compiler is going to
complain that it's not caught. But if you can declare that in the enclosing
method, then the compiler won't complain. You can't declare the
withLock method to throw
NumberFormatException. That would be silly, because not
everyone who uses withLock has to deal with
NumberFormatException. In fact, there is no particular
exception that you can declare in withLock. But using this new
mechanism, you can say whatever type is thrown by the block is thrown by
withLock. If no checked exceptions are thrown by the block, then
nothing is thrown by withLock. So you have a type parameter for
the exception in the API declaration (such as the withLock
method), and you say, "The closure throws this exception X, and so do I." That
allows you to get the exception checking for APIs where the implementation will
propagate the exceptions out.

The reason you need a special syntax for that, the reason you can't just use an
ordinary type parameter, is if you use an ordinary type parameter, that will work
as long as there is exactly one exception that's thrown. If there is no exceptions
thrown, what are you going to fill in there? And what if there are two exceptions
thrown? Then you're just out of luck.

The idea is that if you declare a type parameter with the throws
keyword, then you're allowed to put multiple types in there for the exceptions,
and the way you list them all out, you can't use commas, those would look like
extra type parameters, you use a vertical bar. But you don't normally have to
specify those type parameters explicitly. You just pass a closure that happens
to throw three things, and the caller has to catch those three things. That just
works. But if you do want to explicitly pass a type parameter, which is the three
different exceptions that can be thrown by a block, you just list the three in that
type parameter position with a vertical bar between them.

Bill Venners: So the compiler will always be able to tell by looking at
each individual invocation of a method that takes a closure what the exceptions
thrown by the block will be, right?

Neal Gafter: The set of exceptions that are
thrown by a closure are part of the type of the closure. The interface to which
that's converted must declare those exception types. The closure that's
received by invoke must declare that in the throws clause of the
method of that interface.

Bill Venners: Can you give an example?

Neal Gafter: For example, withLock is a method. It's
second parameter is an interface that contains a single method called
invoke. Whatever exceptions the passed block throws must be
in the throws clause of that method. And the closure conversion checks that.

Writing the withLock method, you don't
know what that's going to throw. That's why you use a generic type parameter.
But the compiler, when it's looking at the closure, knows exactly what is thrown.
It can see that you're calling Integer.parseInt(), for example,
and it knows that it throws NumberFormatException. The
compiler knows what exceptions are thrown by the closure, and it matches
those and makes sure that the interface that the closure is converted to
declares all the exceptions that are actually thrown. If it's a generic, then the
compiler knows what a generic is supposed to be, it's supposed to be
whatever is actually thrown by the closure. It fills in the type parameter on that
basis.

Closures and Functional Programming

Bill Venners: The other thing you showed at the end was some
functional programming styles. I had written a blog post about how it becomes
increasingly difficult to add features to a language over time because you have
to be backwards compatible with more and more existing features. If there is a
better way to do it, like iterations, you leave in the old way. So we have an
Iterator, Enumeration, and with closures we'd
be able to pass a closure to an each method.

Neal Gafter: Yes, but I don't think there is any point in doing it. I don't
think there is a point in adding an each method to
Collection, or
Collections, or anything like that. I don't think it does anything for
you. And if you do it, you don't need to put it in the interface. If you want to put
a static method in Collections, called each that takes a
Collection and a closure, and loops over the
Collection and calls the closure on each of them, you can do
that just fine, there is no backward compatibility problem.

Bill Venners: So you say it we don't need it because we already have
the for-each?

Neal Gafter: I'm saying that there is no problem leaving the for-each in,
and as a practical matter I don't think you need it. It doesn't do anything for you
that you can't already do.

Now, if you had closures before JDK 5, you probably wouldn't have wanted to
add that statement to the language. But I don't think there is any harm in having
it there.

Bill Venners: So basically your original justification for adding
closures was adding control statements. So instead of adding a bunch of
them, you can use closures. The for-each was one of those things. We already
have that one, so we don't need to use closures to do that.

Neal Gafter: You don't, but the point is there are lots and lots of APIs in
the world that would benefit from closures besides collections. It's not
the only API in the world that needs to have API-specific control constructs. It's
the only one that got them so far, but it's not the
only one that deserves them.

Bill Venners: At the end of your talk, you were discussing some of the
functional programming styles that can be done with closures. You talked
about map. Can you talk about how useful the functional styles might be? When
you add closures, can you program in a different style in Java?

Neal Gafter: There is a style of programming that you see in the functional
programming community. Some people like it. I think the main benefit from
adopting that style is when you start using that with the concurrency framework
that takes advantage of that. For example, just having that map operation might
be only useful in some cases, but if you can do it in such a way that you invoke
the function concurrently, especially if the function is particularly expensive, that
becomes a very concise way of writing something that is otherwise very difficult
to do. That is what I was alluding to when I mentioned the fork-join concurrency
framework that Doug Lea has waiting in the wings. There is no easy way
for clients to use that fork-join concurrency framework because there is no way
of expressing the API in Java now that makes it convenient to use. If you have
closures, you could have this map interface that takes one List of T
and this function that maps from T to U,
and returns a List of U. That's
mildly useful, but when you add a parameter that's an Executor
so that you can do that mapping in parallel, it becomes much more useful
because you have a way of very conveniently doing fork-join parallelism. There
are some cases where there is a huge benefit to do that kind of thing
concurrently, if that function is expensive, for example.

I think the most important uses of stuff from the functional programming
community will be in a concurrent setting. I don't really have in my mind all of
what that looks like, but I hope to get Doug Lea involved early, before I
actually put out a prototype or a compiler that does closures. I want to get Doug
Lea involved so I can include some of his fork-join concurrent framework, have
him express those using closures, so that other people would get
a feel for what those actually would look like in JDK 7.

The Next Steps for the Closures Proposal

Bill Venners: Given you work at Google now, what would be the process for getting
this proposal into JDK 7?

Neal Gafter: I'm working on this on my own time. The main reason I'm
working on this on my own time is that Google's JCP representative has
decided that Google does not support this work. So for that reason, I cannot
propose a JSR on Google's behalf. But what I can do is provide a prototype that
someone, for example, Sun Microsystems, perhaps, could use a basis for a
JSR proposal. As a practical matter, I think Sun wants to roll all language
changes that might go into JDK 7 into a single JSR anyway. If that's the case, it
doesn't make sense to have me be the spec lead for the language change JSR
for JDK 7. It will probably be someone at Sun Microsystems, and closures would be
included in that, I hope, as a candidate for JDK 7. Doing the prototype, I hope,
increases its chance of being considered, because people will have a chance
to try it out and see what it actually buys for them.

Bill Venners: What has the response been from your viewpoint?

Neal Gafter: It's very mixed. People seem to have strong feelings about
it, one way or another. I don't think there's really a consensus. There are a lot of
people who have experience with languages like Ruby that feel that having
closures will increase the joy of programming in Java and make it possible to
do things where they currently have to fight with the compiler to coerce it to do
what they want. There are some people that think that you can actually do with
anonymous inner classes all of the things that you can do with closures. I,
frankly, don't believe that. That was really the point of my presentation, trying to
make that clear. And there are some people that understand the difference in
power, but believe that Java programmers are not up the task of taking
advantage of closures without shooting themselves in the foot.

Bill Venners: How can they shoot themselves in the foot?

Neal Gafter: For example, Josh Bloch's position is that if we have
closures in the language, then we effectively have an extensible language, and
if you have extensible languages, different groups will be programming in
different dialects of the language, and that means the programmers will not be
able to move from one project to the other and understand what the code does.
I think that's true, however, programmers that use different API sets already
program in different dialects of the language, and there is already a problem
that when people start looking at code using an unfamiliar API, they don't
understand the code until they learn the API. And closures are just APIs in that
sense.

I don't think closures increase the level of difficulty of understanding code
that uses an unfamiliar API at all compared to the situation we're in today. If you
use one of the twelve XML APIs, and you use a different one when switching
from one project to another, you simply have to learn it. If a project uses certain
closure methods that you're unfamiliar with, you need to read the documentation and
see what those methods do. If you use a good IDE, you can hover over the
method name, and it will point you to the JavaDoc. The JavaDoc pops up on the screen
and tells you. That's all there is to it. I think that's better than adding two or three
language constructs every two or three years, because with those language
constructs, you don't get the popups that tell you what the language construct
does. You need to learn them whether you're going to use them or not, because
they're pervasive. Whereas with APIs, you do have the JavaDoc to help you.

I think it is really an honest difference of opinion. My hope is that having a prototype out
there for people to be able to use, and having people using it, will give some
actual evidence about whether or not this is a problem, and to what extent it's a
problem, and how much does it really solve people's problems to have
closures in the language versus not having closures.

About the author

Bill Venners is president of Artima, Inc. and editor-in-chief of
Artima Developer. He is author of the book, Inside the Java Virtual
Machine, a programmer-oriented survey of the Java platform's architecture
and internals. His popular columns in JavaWorld magazine covered Java
internals, object-oriented design, and Jini. Bill has been active in the Jini
Community since its inception. He led the Jini Community's ServiceUI project,
whose ServiceUI API became the de facto standard way to associate user
interfaces to Jini services. Bill also serves as an elected member of the Jini
Community's initial Technical Oversight Committee (TOC), and in this role
helped to define the governance process for the community.