Economy Size Geek - Interview with Rich Hickey, Creator of Clojure

This month's column diverges from the normal pattern of
covering my struggles with technology. When I started this column, I was really
looking forward to the Software Development issue, because I was hoping to
spotlight a language I am learning—Clojure (pronounced like
“closure”).
But, I
found myself still in the process of bending my brain around it as my
deadline loomed, so instead of trying to teach you while I am still
learning, I decided it would be more interesting to hear from the man
who created the language instead, Rich Hickey.

Rich Hickey, Creator of Clojure

DE: What did you do before you started the Clojure project?

RH: I'm a consultant, so I work on various things. I
think the big thing I've done recently is I worked on the national exit
poll.

DE: What other languages did you use before inventing your
own?

RH: I was a C++ developer for a long time. I taught it at
NYU for a little while. I worked in scheduling systems and broadcast
automation all in C++. Then I moved from that to Java and C#. At the same
time, I also started doing some Common Lisp.

DE: Most people who create their own language start from
scratch—what was it that drew you to do a Lisp?

RH: I discovered Lisp after ten years of C++ and said to
myself, “What have I been doing with my life?” I realized it was
just much more productive for me. I fell in love with it right away.
Basically, I said to myself, “At some point in my life, this is really
what I want to be doing.” I spent several years subsequent to that
continuing to do my commercial work in Java and C#, but all the while
hoping at some point to be able to get Lisp into the mix. That's what
drove me to doing Clojure, because I saw that either .NET or the JVM were
requirements for all the customers I was encountering. Lisp really
couldn't reach those platforms, so I decided to make a Lisp that
could.

DE: Why put it on the JVM?

RH: It's designed to be useful for the
work I was doing, where you have customers that have requirements that things run on one of
these standard platforms—platform not meaning the OS but these
infrastructures, like the JVM or .NET.

DE: And putting it on the JVM, did that get you out of doing
a lot of the low-level stuff?

RH: Absolutely! I mean as a separate concern from the
practicality of being able to access all the libraries, being on the JVM
gives you a really nice separation of concerns between the high-level
language, which is what I got to focus on, and the runtime, which is
something that the JVM does an excellent job with. It's got a great
garbage collector and a very sophisticated runtime compilation
infrastructure. So as a language developer, you don't have to worry
about emitting machine code or anything that low level. But, I'm not on
the JVM just to get a free ride in terms of making it easier to
implement. It was very much a part of the design of the language that you
be able to touch and reach the underlying host platform, because there's
a lot of value there.

DE: Libraries are part of that value, right?

RH: The amount of libraries available are really fantastic.
That meant as a new language, Clojure had a ton of libraries right out of
the gate. People didn't have to wait for a Clojure library for
sockets or for talking to the database or for doing a GUI. They had
libraries for those things right away. So they were productive with
Clojure right away.

DE: You seem very pragmatic. You talked about productivity
and libraries and how you get to use this on a daily basis. That seems
like a different view of what a language is and how it fits.

RH: It's a little bit different. There's no taking
away from Common Lisp being very powerful. I just think that it was
standardized back before the Internet was commonplace. Then it was hard to
move because there was a change in the marketplace in the 1990s that shut
down artificial intelligence and a bunch of other things that were where
Lisp lived. It wasn't really about the language, but it did cause
some stagnation. So coming out fresh, not being compatible, really gave me
a blank slate. Being a commercial developer, I knew what I needed to have
be present in order for it to be practical.

DE: Clojure is not object-oriented, why not?

RH: Well it's not object-oriented the way Java, C# or C++
is. That's not really the way you would structure things. It is in
some ways a rejection of my heritage, as an object-oriented programmer. I
think after having done it for two decades, I don't believe in it
anymore. I just don't think it's the right way to start. It can
help you organize your code, but it brings along with it some complexity
that I have found in real systems always ends up biting you—and it is
related to mutability. By default, an object-oriented program is a graph of
mutable objects. That's a very, very difficult thing to think about,
debug and keep running. I've worked on very big object-oriented
systems, and you always essentially run into the problems related to that
architecture. I think that even before you get to concurrency, there are
complexity problems with mutable objects that basically affect every large
object-oriented application. When you add in concurrency, those problems
become much clearer.

So a functional approach was something that I had already
started doing, even in programs I was writing in C#. For instance, there
were parts of the national exit poll system that were very functional, even
though it's a C# system, because the way to defend yourself against
this complexity is to write in a more functional style with a lot more
immutability. The problem is that it's not very idiomatic in C# or Java to
do so. I wanted to make a language where it was—where the default was to
do the right thing. When you needed mutability, there would be a good story
about how to do that compatibly with concurrency.

DE: So is that why you have the transaction system built in?

RH: Once you say by default everything should be immutable,
real systems can't work that way everywhere. There have to be places
where people can see the effect as if something were changing. So how do
you mimic that? The Clojure system says, “Well, we're gonna take
something immutable and put it into a cell or a reference, and we'll
swap out what's in that reference from one immutable thing to
another.”
And, that will look like change to the program. It's a
little bit different from a variable in that you're promising yourself
the only thing you'll ever put in there is an immutable thing. But,
that gives you a great separation of concerns. Because then you have these
immutable values that most of your program is manipulating. You can say
about those references, “This is what happens in the concurrent
program.” And, one of the things could be transactions, and that's
the STM (Software Transactional Memory).

So it's a nice story. You can switch back and forth
between the different constructs depending on how much sharing there will
be in your application. But, yes, that's why it's built in. Because
without it, you just leave people wondering, “Okay, well, when I need
to change, what do I do? Do I go back to locks?”, and that kind of
thing.

DE: Clojure introduces a lot of different ideas all at once.
How do you get started?

RH: Well, you start small. I think the way in for most
people is to see the way Clojure uses associative data structures (maps in
Clojure). They're called dictionaries or hashes in other
languages. So people that have used Python and Ruby have gotten a good
feel for working with objects as if they were just generic dictionaries.

You will be facing challenges because some of the things that
you're used to aren't there. On the other hand, you also
are handed a lot of things that are much easier and clearer and fit together
better. So you get a little bit of a bump to try to change your habits,
but you're getting immediate payback in a program that's easier to
understand, easier to test and much closer to the way you're thinking
about the problem.

DE: Is that the power you see—that it makes it easier for
you to code the problem you're trying to tackle clearly?

RH: Absolutely. I mean, that's really the great thing
about a Lisp or one of the highest-level functional languages like a
Haskell, the program pretty much consists of stuff that matters.
Anything that doesn't matter, you can make go away. Whereas, if you look
at a Java, C# or C++ program, it's full of things that don't
matter—things
that are there because of the way you have to do things or the syntax and
the language.

DE: Well, I noticed that in May 2009 you guys announced
Clojure as 1.0. It seems like open-source projects often linger below 1.0.
How did you decide this is 1.0?

RH: I think one of the nice things about being a Lisp is
that most of the additions you make to the language aren't really in
the language, they're just libraries. It is very easy to add and not
break anything. Clojure has grown really rapidly and yet in a
non-disruptive way, because new things are just new things. If you
don't use them, they don't impact you.

DE: How does your process for evolving the language work?

RH: It's not very formal. I'm trying to stay in
touch with what people are doing and what's working and what needs
refinement. Sometimes there are performance things I want to tackle.
Other times there are new things in the libraries people need that
I'll tackle. It's not like there's a big road map, because I
don't really believe in getting too far ahead of yourself. Things are
done incrementally.

DE: How does code get into Clojure? Is it
just you, or is
there a team?

RH: On the core stuff it's me. There's one other
person who can apply patches that I've approved. Then contrib is much
wider. In contrib, I think there are about a dozen people who can commit
and there are 100 registered contributors.

DE: And how do you register to become a contributor?

RH: You sign a contributor's agreement. It's a lot
like the one that Sun used, and then you're a contributor.

DE: If I want to make a difference in the language where do
I get started?

RH: Well you should become familiar with what Clojure has got
already. Usually, you'll get good at the core of Clojure, and then
you'll start building apps and using parts of contrib. Then perhaps
you could find something that contrib isn't covering and contribute a
whole new library there. Or, you'll find something in contrib that
could be enhanced. You can talk to the person who owns that part and say,
“Hey, I've got ideas”, and work together. It's not
difficult. There's not a lot of hurdles. The core language being much
more focused I think is critical, because I don't think languages really
are built by teams of people usually. You have to have a vision, and you
really don't want to be going in more than one direction at a time. In
particular for Clojure, it's essential to me that the core stays very
small. Being simple at the very core of things is part of what makes it
good.

DE: Are you surprised by the growth?

RH: It's out of control. When I released
it, I had the
realistic expectation that if 10–100 people used it, that would be
amazing. Because that's all you can expect. But for some reason, it
took off. It was not something I anticipated. While we're talking,
we're at 2,999 members of the Google group. Who could know that was
going to happen? It's been crazy. Very crazy.

DE: Is it exciting for you to kind of get some validation
that you weren't the only person who wanted a Lisp you could use?

RH: Yeah, I think there's the Lisp aspect, and then there
are plenty of people using Clojure that Lisp is a kind of hurdle for initially.
They're not coming looking for the Lisp part. They want the dynamic
development. They want the immutability. They want a good concurrency
story. They want functional programming. The Lisp part is not the
appeal.

DE: I noticed in one of your posts talking about Clojure in
Clojure. What does that mean?

RH: Well, Clojure was written from scratch. So I started
writing it in Java. A lot of the bootstrap, the underpinnings of Clojure,
were written in Java. The basic data structures were all written in Java,
and the first compiler was written in Java. Then, once you had a compiler
and the data structures, you could write the rest of the language in
Clojure. So what I want to do is go back to those parts that are written
in Java and rewrite them in Clojure now that Clojure exists. Recently,
I've been doing a lot of work so it has the features and performance
needed to do even the lowest-level parts of implementing itself. We can go
back and re-implement Clojure in Clojure.

DE: So is the goal to make it so that you have more kind of
idiomatic flexibility? Or just for completeness?

RH: Well, the goal is to get rid of the Java code. It
demonstrates that Clojure has sufficient performance and expressiveness to
do everything the Java part did. The other part is it will make moving
forward a lot easier. David Miller has been porting all the Java to C# to
port it to .NET. So when there's a lot of Java, there's a lot of
C#. When most of Clojure is written in Clojure, only a very little bit
will be Java-specific. And, only that little bit would need to be ported to
.NET. In addition, I think a really important target for Clojure will be
JavaScript. So moving Clojure mostly into Clojure means reducing the
footprint of a port to a very small amount, and I think that's a
valuable goal.

DE: When you move to these other kinds of
things, do you worry
about the low-level stuff you have counted on the JVM for?

RH: The whole thing is not to try to make all these
platforms the same. It's to say, “Well, I know how to accomplish
things quickly in Clojure.” And, when my target platform is the
browser,
I can leverage my knowledge about Clojure to do that as well. You'll
still want the same kind of ability to access the hosts that you have for
Java when you're in JavaScript. You end up with a body of expertise. You
will have libraries that will work in all those places that are not
host-specific. So, more and easier portability is an objective of Clojure in
Clojure. And, it'll just be more fun for people to hack on it.

DE: What is your typical Clojure user like?

RH: There isn't a typical user. It's very much split
among people with very different backgrounds. You have people who are
solid Lispers who are looking forward to being able to reach the rest of
the world easily, who actually know very little Java or none. Then you
have Java people who are looking to get that expressiveness and agility,
but they want to know it will be solid enough to do everything they were
doing in Java in terms of performance and threading. We get people from
Ruby and Python who know they love dynamic languages already, but have
found performance or other issues there. Or they're just trying to up
their game and learn more about functional programming. And then we'll
get people from the functional camp (Haskell or ML) who are looking for the
practicality of being on a platform like the JVM. What's great is they
can all help each other.

DE: They all have a different piece. Your platform provides
something that they can each bring and they each need.

RH: That's a great way to put it. It's made for a
fantastic community where somebody who's struggling to learn the
functional programming side can turn right around and be the expert about
CLASSPATH for somebody else because he knows Java. So that's been
great. I just can't say enough about the community—it's fantastic! I
think there's just a tremendous amount going on. A lot of libraries.
The numbers and the growth are a big part of the appeal of Clojure.
There are books. One book is out, and more books are on the way. So I think
it's very approachable now in terms of getting a lot of help.

DE: What do you see that's exciting for you about the
future?

RH: I think there are lots of challenges still to come,
for people who are looking to try to take advantage of all these CPUs they
are going to have. I'm excited about Clojure in Clojure and growing
that. In general, I'm trying to work on the general problem of
dealing with time in programs. I think that's part of what Clojure is
about. I still have a lot of ideas around that that I haven't
implemented yet.

DE: When you say time in programs, what do you mean?

RH: Well, when we talk about things changing, and
concurrency and mutability and things like that, those are all about time.
I don't think the languages that we've had up till now have been
explicit enough about time. So we've run into a lot of problems,
because we're not thinking about time explicitly. Clojure makes time
explicit. I want to do more work on that, because I think that will be a
really important thing as we move to these multicore platforms.

DE: So are you trying to create something that is missing
from other languages?

RH: Well, I mean I'm borrowing as much as I can from
wherever I can for sure. I definitely stand on the shoulders of giants.
But in this area, I never claim any novelty for Clojure. Clojure is mostly
about trying to take existing good ideas that may not have been put
together and put them together. It's certainly missing from the more
traditional languages like Java, C# and C++. But, there are other ways to
address it that are present in Erlang and Haskell, for instance.

DE: Any final thoughts?

RH: I would encourage everybody to try it out. It's a
welcoming community, and beginners are always welcome and treated nicely.
And, we're happy to have more users.