Friday, April 14, 2006

Lisp is Not an Acceptable Lisp

It's been over four months since Eric Kidd posted his infamous Why Ruby is an acceptable LISP article. You know, the one that got approximately 6.02e23 comments, ranging from "I agree!" through "I hate you!" to "I bred them together to create a monster!" Any time the comment thread becomes huge enough to exhibit emergent behavior, up to and including spawning new species of monsters, you know you've touched a nerve.

What amazes me is that nobody's pointed out the obvious counter-observation: Lisp is not an acceptable LISP. Not for any value of Lisp. There's nothing magical about this, nothing partisan. If Lisp were acceptable, then we'd all be using it.

You've all read about the Road to Lisp. I was on it for a little over a year. It's a great road, very enlightening, blah blah blah, but what they fail to mention is that Lisp isn't the at the end of it. Lisp is just the last semi-civilized outpost you hit before it turns into a dirt road, one that leads into the godawful swamp most of us spend our programming careers slugging around in. I guarantee you there isn't one single Lisp programmer out there who uses exclusively Lisp. Instead we spend our time hacking around its inadequacies, often in other languages.

So for four months I've been waiting for someone else to say it, but so far it's not happening. Why aren't we admitting it?

Oh, right. Religion. I keep forgetting about that.

Lisp programmers are just like all other programmers: they want to write code and get cool stuff done, which presupposes they've already learned the last programming language they'll ever need. There's all this real-life stuff (jobs, family, stability, all the usual suspects) intruding on you as a programmer, demanding that you quit dorking around looking for the One True Language, and settle down on whatever barren rock you happen to be squatting on at the moment, and call it Good. So most Lisp programmers — and that's not many, since not many programmers make it even close to that far down the Road — see that last outpost of technical civilization, peer balefully into the swamp, and decide to check into the Lisp hotel for good. Not realizing, of course, that all its rooms are in the swamp proper.

We all know it's tricky to have a rational discussion about a religion. Non-Lispers will be able to read this without getting their feathers ruffled. Some Lispers aren't too far gone, so let's assume we're talking to them, and take a look at some of Lisp's problems that make it flat-out unacceptable. At least for LISP, you know, the idealized one.

Problem 1: Which Lisp?

Sorry, folks, but you can't trivialize this one. Let's say I'm a new would-be Lisper, just finished walking down that long damn Road, and now that I'm here, I'm ready to start using it. Which "it" should I use? The answer is "it depends", and that's pretty unfortunate, because right there you've just lost users. With Python or Ruby or Java, you've only got one language to choose from. Or at least you can be comfortable that there's a single canonical version, and the rest (e.g. Jython) are highly experimental territory.

Pick Scheme, and you have to pick a Scheme. Pick Common Lisp, and you have to pick a Common Lisp. Heck, there are even two or three flavors of Emacs-Lisp out there.

Most newcomers eventually (and independently) decide the same thing: Scheme is a better language, but Common Lisp is the right choice for production work. CL has more libraries, and the implementations are somewhat more compatible than Scheme implementations, particularly with respect to macros. So newcomers heave a deep sigh, and they learn to accept LISP-2, names like rplaca, case-insensitivity, '(ALL CAPS OUTPUT), and all the other zillions of idiosyncracies of a standard Common Lisp implementation. Eventually, if they stick with Lisp at all, they learn they can override most of these defaults in nonportable ways, which makes things infinitesimally more bearable.

Whatever. If you're a Lisper, you dealt with all this crap years ago, and now you're committed. If you're not a Lisper, then you're not very likely to become one any time soon. In fact your probability of learning Lisp is decreasing over time, as other languages continue to close the gap in the Lispy areas, and simultaneously increase their lead in non-Lispy areas where Lisp is making little (if any) progress.

Let's look at some of those areas. But first, let me mention one last problem in the "which Lisp" space. It's dirty laundry that needs airing. The problem: Paul Graham. I mean, the guy's a genius, and I love reading his essays, and his startups are doing great things, etc. etc. You can't fault him. But he's created something of a problem.

Before Paul Graham, Lisp was dying. It really was, and let's not get all sentimental or anything; it's just common sense. A language is always either gaining or losing ground, and Lisp was losing ground all through the 1990s. Then PG came along with his "I'm not talking to you if you're over 26 years old" essays, each a giant slap in our collective face, and everyone sat up and paid attention to him in a hurry. And a TON of people started looking very seriously at Lisp.

Lisp might or might not have experienced a revival without Paul's essays, but it's moot: he showed up, and Lisp got real popular, real fast. And then he said: "Don't use it!" Sort of. I mean, that's effectively what he said, isn't it? By deciding to pre-announce Arc, he Microsofted Lisp. Killed it with vaporware. It's a great strategy when you're an evil empire. I don't think that's exactly what Paul had in mind, but let's face it: that's what happened.

So Common Lispers grumble about Paul in the hallways. If I read newsgroups (every time I try, the overall ugliness of humanity drives me away within hours; I only re-attempt it every decade or so) I see them grumbling there too. He's put them in a tough spot, because he did use Common Lisp for Viaweb, and his arguments in favor of Lisp (in the general sense) have been compelling enough to bring in newcomers by the droves. But he's not throwing his weight behind CL. He's not even taking the marginally-acceptable route (from a damage-control perspective) of recommending Scheme. Instead, Paul's (*gasp*) starting a new religion.

Arc's going to be a new religion, of course, because programmers just haaaaaave to make it that way. If it ever appears, anyway. But will it? That's the tricky thing about Cathedral-style software; you never can tell. My prediction: someone will get tired of waiting, and they'll Torvalds Arc into obsolescence before it's ever released. (If you don't get the reference, it's what Linux did to GNU Hurd).

Long story short: nobody knows what the hell Lisp they're supposed to be using, and it's absolutely killing adoption.

Problem 2: Worthless Spec

Oh, ouch, did I have to put it quite like that? I mean, c'mon, let's be fair, there are literally hundreds of people out there who disagree.

Unfortunately, the simple fact is that the spec is ancient. Every time someone talks about updating it, someone screams about time or money or whatever. The problem is (like the problem with RSS) a people-problem, not a time or money problem. This is absolutely true in the Scheme world, too. There are a bunch of old-timer stakeholders who want to have their say. So you're basically asking a goverment (complete with lobbyists, political parties, the works) to design Lisp if you go that route. The naysayers are right about one thing: it'll never happen.

Your only other option is to design a new language, and you won't get any help from Lisp people, because they will hate you. They love pointing to the trail of bodies left in the wake of every pioneer who's tried this before, none of whom has emerged with a "successful" Lisp. Of course, they haven't been successful because Lispers didn't want to have anything to do with them; Lispers are just as incapacitated by their techno-religious beliefs as folks from other languages. Religions dislike each other, but no heretic is as damned as someone who starts with your religion and makes a modification to it. Just ask the Albigensians, for instance.

But what's wrong with Common Lisp? Do I really need to say it? Every single non-standard extension, everything not in the spec, is "wrong" with Common Lisp. This includes any support for threads, filesystem access, processes and IPC, operating system interoperability, a GUI, Unicode, and the long list of other features missing from the latest hyperspec.

Effectively, everything that can't be solved from within Lisp is a target. Lisp is really powerful, sure, but some features can only be effective if they're handled by the implementation.

Problem 3: CLOS

Note: this section contains a few factual errors pointed out by Pascal Costanza in a comment below. Some of his corrections are also, of course, opinions, and I've commented on them later in the thread. In any case, while I thank Pascal for his corrections, the errors I've made are utterly irrelevant to my conclusions.

CLOS is icky. I haven't worked with Smalltalk a whole lot, but I've worked with it enough to know that to do OOP right, you have to do it from the ground up. CLOS was bolted on to Common Lisp. Everyone knows it, although not many people want to admit it.

It was bolted on very nicely, and it's not my intention to disparage the efforts of the people who created it. It was an amazing piece of work, and it did a great job of being flexible enough to tie together the conflicting OO systems of existing Lisp implementations.

But let's face it; CLOS has problems. One obvious one is that length isn't a polymorphic function. It's one of the first speed bumps you encounter. You can't create a new kind of measurable object and give it a length method; you have to call it rope-length or foo-length or whatever. That's part of Lisp's endoskeleton showing; you can see the bolt sticking out plain as the nose on your face. It's not seamless; it's not orthogonal, and it's not the Right Thing. But it's not going to change, either.

Another problem is the slot accessor macros. They're insanely clever, but clever isn't what you want. You want first-class function access, so you can pass the getters and setters to map, find-if, etc. You can work around these things, but they're a leaky abstraction, and enough of those will add up to significant mental resistance to getting things done. It's like all those weird little rules in Perl: non-orthogonal rules that add up to forgetting the language every time you leave it alone for more than a week.

What you really want in lieu of CLOS is... complicated. It's a hard problem. Lisp wants to be constructed entirely from macros. It's part of the purity of the idea of LISP: you only need the seven (or is it five?) primitives to build the full machine. Doing CLOS as a bunch of macros was very much in the spirit of Lisp: it was a Lispy thing to do.

But macros are a problem. Yes, they're one of the most important differentiators. But macros are like having these high-powered band-aids, when what you want is not to be wounded in the first place. Having the object system — something pretty fundamental to the language, you'd think — written as a bunch of macros doesn't feel right when all is said and done.

When you work with Ruby or Smalltalk or any suitably "pure" OO language (Python doesn't quite count, unfortunately; its bolts are also showing), you realize there are some distinct advantages to having everything be an object. It's very nice, for instance, to be able to figure out what methods are applicable to a given class (e.g. "foo".methods.sort.grep(/!/) from Ruby), and to be able to extend that list with your own new methods. It's a nice organizational technique.

Of course, that forces you into a single-dispatch model, so it becomes harder to figure out what to do about multi-methods. Some Python folks have implemented multi-methods for Python, and they do it by making them top-level functions, which makes sense (where else would you put them?) I'm not claiming that Smalltalk's object model is going to translate straight to Lisp; you have to decide whether cons cells are "objects", for instance, and that's a decision I wouldn't wish on my worst enemy. I don't envy the person who tackles it.

Regardless of what the solution might be, CLOS remains a problem. It's over-complicated and yet not quite OOP-y enough or expressive enough. The problem of reflecting on an object to see which methods are valid for it is one example, but there are tons of others. Heck, one possibly valid complaint is that it doesn't work very much like the "conventional" OOP systems of C++, Java, Python and Ruby. There's no real reason it shouldn't be more like them. But changing CLOS to be simpler and more seamless essentially means replacing it. And replacing it is probably best done inside the implementation.

In other words, any fix means starting virtually from scratch.

Or maybe you could go the Haskell route and not have OOP at all. That seems to alienate most programmers, though, despite the attractions of not having to create nouns for everything. (Have you ever noticed that turning a non-object-oriented program into an object-oriented one in the same language that does the same thing essentially doubles its size? Try it sometime...) At the risk of predicting future fashion trends, which is rarely a good idea, I'll venture that objects are going to continue to be trendy for at least a few more decades. So I think Lisp needs some form of "seamless" OOP.

Problem 4: Macros

Macros are one of the worst problems with Lisp, or at least they're one of the biggest unsolved problems.

Yes, they're amazingly powerful and critically important and blah Blah BLAH. You can read all about them elsewhere. Paul Graham's On Lisp is the best reference I've found.

But they're fraught with problems. One is that they're not hygienic. You should at least have the option of requesting hygienic macros. Various papers have been published, and implementations implemented, for hygienic defmacro. Yeah, it's hellishly hard to get right, and it's overkill for many situations, but it really does need to be offered as an option. A portable one.

For that matter, you should also have a choice between Scheme-style pattern-matching macros and Lisp-style code-style macros. They're very different, and each kind is better (cleaner) in some situations. People often act as if hygiene is synonymous with define-syntax, but the pattern-template style is orthogonal to the question of hygiene.

Style considerations aside, macros have tool problems. Macros are notoriously hard to debug, and honestly it needn't be that way. If your editor knows all about macros, then you should be able to click to see the expansion, and click again to see its sub-expansions, all the way down to the primitive functions. Some editors can do this, but none of them (that I'm aware of) handle macros as cleanly or seamlessly as they do normal functions.

Syntax in general is a problem. Lisp has a little syntax, and it shows up occasionally as, for instance, '(foo) being expanded as (quote foo), usually when you least expect it. Truth be told, Lisp should probably have a skinnable syntax. That implies a canonical abstract syntax tree, which of course hasn't been defined (and in many implementations isn't even available to you, the way it is in the Io language, say). Once you've got a canonical AST defined, syntax should, in theory, be like CSS chrome. Of course, there are plenty of bodies left in the trail of this particular theory as well. Someday...

In any case, because macros are rarely supported "well enough" by the tools, and because they're not first-class functions, and so on, they wind up being second-class citizens. The rule "you should only use a macro when nothing else will do" implies that they really are a last resort, which (to me) is synonymous with band-aid. Yes, it's wonderful that you have the band-aid — or maybe duct tape is a better analogy — certainly you miss them dearly when you're working in other languages. But you don't want to have to build your entire object system with duct tape.

So macros, like the object system, need to be re-thought from the ground up. There's undoubtedly enough research in the space that someone could throw together a working definition in no time, something just good enough for today's programmers, the ones who expect (and rightfully so, I might add) to be able to name their methods "length" without getting a compiler error.

Problem 4: Type System

See, that's just exactly the problem with type systems. They can make sure you use headings, but they can't ensure you get the numbering right.

Well, it'll take me forever to talk about this one, so I'll have to leave it for another blog. The problem is that the type system has to be extensible and skinnable, and I'm not strictly talking about user-defined types in the sense of OOP or CLOS. Unfortunately it really is a huge open issue, one that'll take longer than this blog to sort through, so I'll have to leave it for today.

Lisp, for all the strengths of its flexible type system, hasn't got this issue right either. Otherwise Haskell and OCaml (and C++, gack) wouldn't be kicking its ass all over the performance map. 'nuff said, at least for now. [And no, they don't quite have it right either.]

I promise I'll talk about type systems soon. But I also promised some friends I'd make my blogs shorter.

There is no acceptable Lisp

This is a problem. It's not a little teeny one, either. The Lisp communities (yeah, there are a bunch) are going to have to realize that if Lisp is ever going to be massively successful, it needs an overhaul. Or maybe a revolution. Contrary to what some might tell you, it doesn't need a committee, and it doesn't need a bunch of money. Linux proved exactly the opposite. Lisp needs a benevolent dictator. Lisp needs to ditch the name "Lisp", since it scares people. And Lisp needs to learn from the lessons of the 45 years of languages that have followed it.

And no, I'm not the guy. You're all far more qualified to tackle this problem than I am. Especially if you're under 26.

63 Comments:

One thing, though -- it's about time to update your understandings of Python's object system. Python's culture doesn't insist that traditional (Smalltalkish) OO is the One True Way to re-use code, but that doesn't mean that everything in Python is not an object (because they are, since 2.2 several years ago) or that Python is less OO because Guido likes the explicit self in method declarations (because really, we both know that's just a syntactic quibble) or Python prefers a protocol-based approach to common functionality like "sorted" or "len" rather than a purely method-based one.

Steve, thanks for another interesting rant. I've watched the lisp community for the last ten years. I've dabbled in scheme and common lisp and although I learned a lot I've never really felt that it was worth using them for serious projects. It always seemed more pragmatic to leverage the existing libraries that exist in the python/ruby/c++ communities, where I know that, at least for a few years, you're going to have user communities that number into double digits.

One language that doesn't get mentiod often these days is Dylan. I had great hopes for it, it had the makings of a better lisp than python or ruby, it had macros if you really needed them, it was deeply OO, it had equivalent infix and prefix syntaxes, it even had native compilers and the potential for C++ level performance. Unfortunately it had multiple abusive parents who each abandoned it before it reached maturity. Now that the Fun-O implementation is open sourced maybe it's worth another look ? From a human perspective we need a DHH marketing effort to do for Dylan what was done for Ruby. There's nothing like a killer app to hype a programming language, theoretical arguments just don't cut it. I don't suppose it's likely to happen but one can always dream ...

Very interesting read. As someone who's been playing with Lisp lately, I certainly identified with the confusion about which Lisp to pick and all the different implementations and CFFI and UFFI and ASDF, etc, etc. Have you seen the list of prerequisites to go through the UnCommon Web Framework tutorial?

It seems to me that Python and Ruby have most of the power of Lisp, but then they have the advantage of a canonical implementation and tons of libraries to let you do interesting things right away.

Your post contains many false premises and fallacious arguments. I suggest you stop disinforming and misinforming, and try to criticize your posts before posting them. There is a severe "begging the question" kind of "reasoning" in this post, which probably results from your world-view, and I suggest you try and figure what these suppositions are, and why they exist.

You show considerable lack of willingness to work with raw materials and mold them into something that is usable, and even elegant. If you perceive a problem, don't stop just there - try and find solutions. You may find them, or you may find that what you have perceived is in fact a non-problem.

* case insensitivity may cause headaches sometimes; * non-generic sequence functions (this can be fixed to some degree though if you really need them).

Not mentioned - the actual problem with macros (difficulties implementing code walkers) and others (you need to learn CL first to properly understand most of them :)

Maybe I've missed something.

Most of other stuff is something you actually have in your head after a year of learning CL. "Lisp is not something like My Previous Favorite Language(s)" kind of stuff etc. It may take a lot of time to explain this properly, I'll just mention one point now:

> Every single non-standard extension, everything not in the spec, is "wrong" with Common Lisp. This includes any support for threads, filesystem access, processes and IPC, operating system interoperability, a GUI, Unicode, and the long list of other features missing from the latest hyperspec.

The spec is actually a copy of language standard (ANSI X3.226-1994). For languages that don't have standard the spec is NIL. So everything not in spec of Ruby (Python) is wrong with Ruby => everything with wrong with Ruby (Python), right? Not just threads, IPC, GUI? Moreover, there's no threads/IPC/GUI/etc. in standards of such successful languages as C and C++.

Here are a number of mistakes made in this blog entry. (Whether they affect the conclusions or not is a different topic.)

* Which Lisp?

- There is no single canonical version of Java. It is an important practical issue whether you can go with the newest version (currently JDK 1.5), or whether you have to stick with an older one (JDK 1.4 or below). I don't know enough about Python or Ruby in this regard, but I would be surprised if this didn't happen in the long run there as well. All languages used in practice have this issue.

- It's not clear that Scheme is better than Common Lisp. There are a considerable number of people who think that the basic design decisions of Common Lisp are better than those of Scheme, for example wrt Lisp-2 vs. Lisp-1. (This is not the place to discuss the details.)

- Scheme is, of course, also used in practice. Once you decide to go for an implementation, you get a similar amount of predefined libraries that you need and that you get with each single Common Lisp implementation, including advanced object systems, etc.

- Paul Graham implements Arc in Common Lisp. That's a typical approach, to implement your own dialect within an existing Lisp framework. If you don't want to embed your own (domain-specific) language in a host language, Lisp doesn't buy you too much.

- Apart from the fact that the spec is ancient, some people actually value the fact that it doesn't change. The advantage is that the language is not a moving target that you have to adapt your software every 1.5 years to.

* CLOS

- Slot accessors are not macros, they are functions. Libraries don't need to be constructed entirely from macros. Use macros only for syntactic abstractions, not for functional abstractions.

- The way to find out the set of all methods for any given object is:

(specializer-methods (class-of object))

This works for any object because all objects have a class in Common Lisp, without any exception (including cons cells).

- If you want to have a single dispatch object system, go ahead. You can build it yourself in Common Lisp - you can even reuse most of the machinery that CLOS already provides. (You can do this by providing your own object model extension via the CLOS Metaobject Protocol.)

- Consider Dylan for a Lisp that integrates OOP more seamlessly. However, the gap between "core" Common Lisp and CLOS is typically below the threshold of recognizability for practicing Lispers. The only exception seems to be a canonical accessor for containers. If this really bothers you, there are libraries that solve this problem.

* Macros

- Macro hygiene is virtually not an issue in Common Lisp (because Common Lisp is a Lisp-2). It's more an issue in Scheme (because Scheme is a Lisp-1). If you want hygienic macros, go for Scheme. There are also pattern matcher libraries for Common Lisp that you can, of course, use in your macro definitions.

- "You should only use macros when nothing else will do." is a simplified expression of the fact that you should use functional abstractions when you can, and use macros for syntactic abstractions. It also refers to the history that macros were used for efficiency in the past, which is not necessary anymore because inlining of functions is supported in most Lisp implementations (and compiler macros are another feature that allows you even more control than macros without losing the advantages of having first-class functions).

* Type System

- Common Lisp's type system is extensible. See deftype. Lisp languages are by default(typically) dynamically typed. This gives you increased expressivity at runtime. (Static type systems give you increased expressivity at compile time.) To a certain degree, dynamic type systems and static type systems are incompatible, so you can't have the cake and eat it at the same time.

* Summary

My impression is that you have use neither Common Lisp nor Scheme on a regular basis, and/or in a considerably sized software project. Most of the issues that you mention are "academic", in the sense that they are theoretical problems which tend not to bite you in practice.

With Python, Ruby, Rebol, PHP, PERL (4, 5, 6), TCL, JavaScript, VBScript, Lua, ... you have confusion, too. A multitude of languages not able to work effectively and each with its own, similar but incompatible, design choices. Each has a slightly different idea of OO, slightly different semantics.

Given that chaos, Common Lisp doesn't even look THAT bad.

Btw., CLOS is not based on macros. CLOS is defined in three levels. The first level is based on objects. There you have objects, classes and methods for CLOS itself. On top of that you have a functional protocol. Macros are then defined on top of that to simplify things for the programmer.

STANDARD-GENERIC-FUNCTION is class and you can do (MAKE-INSTANCE 'STANDARD-GENERIC-FUNCTION) to create one. But the developer can use the functional protocol using (ENSURE-GENERIC-FUNCTION ....). But usually it is more convenient to use (DEFGENERIC ...). DEFGENERIC is a macro. ENSURE-GENERIC-FUNCTION is a function. STANDARD-GENERIC-FUNCTION is a class.

So, CLOS is not a bunch of macros. The macros are just there to make the usage of CLOS for the programmer more convenient.

The best book to read about that kind of stuff is The Art of The Meta Object Protocol. Highly recommended. Alan Kay, likes that book too.

And slot-accessors are not macros. They are functions. You can pass them around. There is no need to work around.

Lisp is unacceptable for one simple reason: it is "controversial".Controversy doesn't come to life by itself. It is created by people who talk and talk and talk about a "contraversial" topic.

For some bizarre reason, everyone who knows the term "programming language" believes himself qualified to talk about Lisp, the same way that most everyone considers himself an expert on economics and psychology, the same way that most everyone who ever saw a TV screen feels himself qualified to talk about the Mideast conflict.

If you're a new user who wants to try Java, you download the latest Java 1.5 SDK from Sun's website.

This is a pretty uncontroversial statement.

We have a similar state of affairs in the Ruby and Python communities (download the latest from ruby-lang.org or python.org).

There is no equivalent of this for Lisp. If you're a new user who wants to try Common Lisp, you download the latest version of X. Where X ∈ {CMUCL, SBCL, CLISP, OpenMCL, Allegro CL, LispWorks}... and this leaves aside the massive set of Scheme implementations.

From a pragmatic perspective, having different (mostly backward-compatible) versions of a canonical language standard in the wild is not in any way equivalent to the mess of varying implementations in the Lisp community. It's not even in the same ballpark.

A lot of Lispers take great pains to point out that technically this shouldn't be a problem, but culturally and politically it's a huge problem. And culture and politics matter a lot in the land of programming languages. There's just no way around it.

There is a wide choice of Common Lisp implementations for the new user, but you will find little difference in practise between them.

Once you have ASDF (an standard library installer) and CFFI (for foreign functions) you have a wide range of common lisp libraries, and an even wider range of libraries in other languages, that you can use without modification on all the major implementations.

I'd be the first to admit there's a big learning curve to become productive in lisp, but I don't agree with you analysis that there is swamp at the end of the road. I think you just got tired of swamp and turned back.

I'm not sure what you mean when you say that Lisp has no canonical AST form. As far as I know, that's precisely what s-expressions are. It's true that occasionally ', #', and read-macros (hell, macros in general) make it harder to see that, but there's no ambiguity about the behavior of quote or the tree form of any expression.

I was a little disoriented trying to interpret such a barrage of opinions going every which way, and then it finally hit me that it simply means you’re a very smart guy. Either you know exactly what you’re talking about, or you’re very convicingly misguided, or possibly even a very good troll. I’m not making a judgement here, despite how that might have sounded so far, just an observing that for someone who is not quite so intimately familiar with your topics, it is not possible to tell.

Really, that alone means your rants are worth reading – even if they are every bit as misguided as some would assert. That is quite a feat, bravo.

laurie cheers asked why macro higiene isn't a problem. I think it's less about Lisp-1 vs. Lisp-2 than about packages. Any serious application will be in its own package, so the temporary variables in its macro expansions will be internal symbols within that package, as long as the macro author doesn't use symbols that happen to be in the COMMON-LISP package (i.e. he should avoid variables like LIST, CAR, SEQUENCE).

Lisp-2 is also a benefit, because a variable named LIST won't impact callers of the function LIST in the body. This isn't even really a macro-specific problem; Scheme programmers have the habit of using the local variable named LST to avoid shadowing the global function LIST.

It's still important to know about the variable capture problem to avoid problems when you use your macro from other functions in the same package. But in practice it's not a major issue.

Have you looked at the R6RS process? It's going to add non-hygienic macros and a standard library interface to Scheme.

I think that the way forward for Lisp (in the shortish-to-medium term) might be a "distribution" of R[67]RS Scheme built from the core language plus a selection of canonical libraries. Effectively, another post-Perl Big Practical Language that just happens to have Lisp as its core. This is of course an old idea: recall Worse is Better from 1991.

I enjoyed your post. Last week I got a copy of Practical Common Lisp to add to my ANSI Common Lisp by paul graham.

One thing that I think lisp is really missing is good introductory documentation. Practical Common Lisp is helping, but other references for new users are lacking.

That said, I'm having alot of fun playing with lisp. Lisp is also (trite as it sounds) impacting the way I think about programing problems in other languages. My C# code began to improve after I started learning lisp last year.

As for the religion... Lisp is fun, but so was C when it was the only language I knew. Now I can pick C, C++, C#, PHP, PLPGSQL, etc... Pick the language based on the problem. Don't solve the problem in the language because it's shiney and cool.

One final thought on PG's comment on the under 26 crowd being his target audience. It seems to be the group that is more willing to chase windmills. I'm 23 and still chasing every so often. That's the fun half of programming for fun and profit.

uh, scheme is clearly an acceptable Lisp for teaching the most important ideas in computer science. lush is more or less an acceptable (much faster) substitute for matlab. common lisp has a lot wrong with it which is why i don't use it. probably a lisp embedded in a language with nice libraries and so on, a la SISC, is the way to go.

i'm sick of people linking to your verbose, whiny, moronic posts. please drop dead, at least with respect to the internet. cheers.

I'm a LISP newbie, but my first act was to download the GNU version. Maybe you guys didn't cut your teeth on gcc and linux, but for anyone that did, I'd imagine the first place to start is the gnu repository for any new tool unless someone can point you to a better solution.

In my opinion the problem isn't that there are a dozen different implementations of lisp. It is that each one of them requires annoying effort to learn. To me this isn't a language issue as much as it is an IDE issue.

Lisp needs an IDE that lets you dive in and learn as you program. I mean I enjoy jumping in and playing computer games with no previous experience. This is because games are designed that way and one reason people like C# because it(and its ide) is designed with the goal of getting people to like it.

As a new user you download the latest JDK, right. Unless you work for a company and your application server does not run with the latest JDK. Companies I worked for used ancient JDKs and ancient application servers, because updating was non-trivial and REALLY expensive.

And for the scripting language you download the latest. But which one? Python? The one with reduce? or the next one without? The C or the JVM version? Perl? 4? 5? or 6? Or better Ruby? But what is the difference? Which version of PHP? The latest? Will it run older code? So you write your code in Ruby. But then you detect that the GC is not good enough. Hmm, but there is basically one implementation and no spec. No choice. No competition.

If I would let somebody develop code for a company, I would demand that it be Common Lisp. I have twenty year old code that still runs and will run for the next twenty years. Just the platform below will change or the Lisp system will get better. My investment in the code is secure.

There! A polymorphic getter called length passed to higher order function as requested. Except that this code is probably hard to read because I've got fed up pissing about trying to come up with a substitute for the PRE tag. I don't think blogger.com is a suitable venue for a technical discussion.

sterra says that 'Lisp needs an IDE that lets you dive in and learn as you program.' Fortunately, it already has one: emacs + SLIME. Emacs is itself an extensible text editor written in a dialect of Lisp, and SLIME is an interface to many different Common Lisp implementations.

A very enjoyable essay/rant. Thanks for the thought-food. I view lisp through a very nostalgic lens; it was the most enjoyable language to code in, but I haven't been able to use it for years. So I mostly only remember the good things.

Disclaimer: I don't know if I'm in your target audience, as I've not developed a large (or even a small) project in CL -- but then maybe that puts me exactly in your target audience. Dunno. (On the other hand, I have been a developer in large C and C++ projects, and moderately sized Pascal and Java projects.) Anyway:

1) Which Lisp?

Never bothered me. I've played with clisp, GNU CL, ECL, and CMUCL. I've not bothered with Scheme, except reading a few chapters of SICP, which I quite enjoyed, and wish I had time to read more of.

By the way, does the multitude of C and C++ compilers bug you, too? Or have you so internalized autoconfig that you don't notice any more?

2) Worthless spec?

Never bothered me. Quite the opposite, I've found the spec quite worthwhile. For GUI I like LTK. Few of the projects I worked on professionally used threads, and the one that did, did so badly. I believe the spec fully defines file system access, but see the disclaimer. As near as I can tell, most of point 2 boils down to "Lisp is neither C nor Unix. Wah!"

By the way, how often do you read the C or C++ specs, and do you miss threading and a GUI there, too?

3) CLOS

Never bothered me. I like CLOS.

By the way, you have the gall to mention C++ in the same paragragh in which you call CLOS "over-complicated"? Stunning.

4) Macros

Never bothered me. I've written a few, and debugged a few, and the alleged issues of "hygiene" seem very much like a "don't do that then" kind of problem. I would much rather deal with the "problem" of macros than not have them.

By the way, how do you define new syntax and control structures in C, Java, or Perl?

4) (That was cute, by the way.) Type system

Never bothered me. But see the disclaimer.

By the way, "skinnable type system"? What does that even mean? (And, could you do it without macros?)

In short, I found several acceptable Lisps. People that use Lisp a lot, like people that use Perl or Python or C or C++ or any other computer language, have learned to deal with its ideosyncracies. Few of us deny that it has them, but we do wonder why people who seem to have few problems with all the other languages' ideosyncracies have so many problems with Lisp's.

Re skinnable type system: There is a relatively new approach in which you can indeed "skin" your type system, i.e., implement your own typing rules. Think domain-specific type systems. That system is called Qi. Guess what language this is implemented in? ;) See http://www.lambdassociates.org/ for more information.

When I find my reasoning leads me to the conclusion "P is not P", I tend to take that as a sign that something has gone awry along the way.

In your case, I think you've conflated "an acceptable Lisp" with "the perfect programming language". There's no such thing. All programming languages suck, each in their own way.

That's partly because programming languages are designed for different purposes, and partly because they're designed by humans.

I don't think the solution is to design a new language, because new languages inherently suck. Nobody uses them, nobody will pay you to use them, nobody's written any books on them, they don't run on your favorite systems, and there's no guarantee that they'll still be around in a few years.

Enjoyed this post, and though I haven't read most of the comments on this post, I enjoyed the next post too. Blahblah.

After first flirting with Common Lisp last July, taking a course at UNI which made quite a bit of use of Scheme, and embarking on an undergrad research project using Common Lisp because I liked Scheme enough to see what doing OO in a Lispy language would be like, I still feel like a Lisp noob, even after messing around with it 3/4 of a year. I find that much of your rant here comports with my experiences. The bit about feeling like you've completely forgotten the language if you're away from it for a week especially hit home and got a laugh out of me. Since I'm working now, I can usually only put significant time in on my project on weeekends, and each time it seems like I'm having to re-learn or re-look-up so much stuff. I have to keep around five tabs of different pages of PCL and CLtL open in Firefox the whole time.

I've also run into such frustrations as no reflection (not even the like of "instanceof"), no way to get a list of the keys in a hash table (or if there is, I can't find documentation in it), not having quite as much polymorphism as I'd like (your mention of "length" is one example, but also having to remember a bunch of different iterators for different composite data types when what you really want is just something like a polymorphic "each" would be another; and I've found "loop" to be of little help -- it's too complicated a syntax -- practically a whole mini-language -- for the new Lisper to grab hold of, and then when you think you've figured out how to get it to do what you mean it turns out to do something else), etc.

I don't even care that much about macros. Maybe it's a function of my newbness, but I've only had need to write maybe two or three macros of my own. But Lispers seem to get so fanatic about them. I appreciate their usefulness, and the times I've needed them I was glad to have them, or at least to have "symbol-macrolet." Obviously Lisp needs macros because so much of it is built from them, but maybe that's the thing -- most of the macros a Lisp programmer needs are probably already there.

But then there are an awful lot of things I like about Lisp, or that I liked enough about Scheme to make me come back. Which is why it's such a bummer about Arc. From what I've read on it, it seems to address a lot of the little annoyances I have about Lisp while keeping lots of what I like, and this makes me anxious to try it. I appreciate that Paul Graham wants to take his time and get it right, but it's frustrating because if there is such a thing as The Moment for it to emerge, it feels to me like it's right about now.

Pascal, now that I've had a chance to read through your comments more carefully, I want to thank you again for them. All are very thoughtful and well-considered.

A few notes:

- I didn't read your very first parenthesized note the first time around, and missed your observation about whether they affect the conclusions. Good catch, my bad for missing it.

- the newest version of each of the languages you mentioned is the canonical version, and they make this possible by growing the language but maintaining backwards compatibility. Of course this results in permanent, un-fixable bugbears in each language, but they at least try to evolve the languages in the directions they can. Haven't seen anything new from CL in a decade or so.

If Allegro were free, this might be a lot smaller issue, because they're definitely innovating at Franz. Making it free might turn it into the defacto spec. It'd require a new way to monetize it, but I don't think that's impossible.

- I didn't say Scheme is better; I said most newcomers conclude that it's better (and of course I lack any real data to back this up, other than the experiences of folks I've known, or folks I've read about like Don Box(?)). However, Common Lisp is obviously better than Scheme in lots of important ways.

- Unclear what PG's long-term intentions for Arc are. Let's hope he has great taste. And that he finishes it.

- C'mon, valuing a non-changing spec? Languages have to grow. Yes, Lisp is growable, but the very strength of CL rests on the fact that it unified all the offshoots for a while. Now we're back into non-unified offshoot territory. The situation is so dire that any competent, respected Lisper (e.g. *you*, Pascal) could simply choose a set of features, policies and fixes, call it PascalLisp, and we'd already be better off than we are today. Well, please don't call it PascalLisp. New Pascal, maybe. :)

- I was totally wrong about CLOS, apparently. I still think it's busted, but I've evidently conjured up mostly imaginary problems in that blog entry. And no, I don't want single-dispatch.

- If Dylan's OO system is "more seamless" than CLOS, then why is everyone else responding that CLOS is seamless? There are either seams or there aren't, right? What's the secret clunkiness of CLOS that I missed that makes Dylan better? Because whatever it is, that's what I meant to say. :-)

- I'm not sure I buy your "virtually not an issue" assertion. Paul Graham spends an awful lot of time in On Lisp (all of Chapter 9, for instance) describing techniques for avoiding this "non-issue". And I sure see a lot of gensyms scattered through all the CL macros I've grabbed and/or written. Are you sure you're not just writing it off because you're used to it?

- Type System: thanks for the link to Qi. I'll be checking it out very soon.

- You're absolutely right: I haven't used Common Lisp much. I've used Scheme a lot, although not in the past year or so.

However, my concerns go pretty far beyond whether the problems are "academic" or not. All language problems are academic in the sense that once you learn to avoid them, they typically impose fairly minimal conceptual and/or syntactic overhead. I think you could estimate the syntactic overhead for each big language out there, and the worst offenders would probably be in the neighborhood of 50%-100%, some of which is solvable by good tools. That's not too bad, really.

I'm not arguing that you can't get work done in these languages. You can get very solid work done in all of them, with enough persistence. My basic impetus for blogging about this stuff stems from the desire to have my cake and eat it at work. I want a language with Python's succinctness, Lisp's extensibility, C's performance, Erlang's parallelism, Java's tools, Io's orthogonality, and apparently Qi's type system. :-) And more besides.

Of course it may not exist as described, but you have to agree that it's possible for us (i.e., everyone) to create something that's much closer to this goal than any existing language out there today. Few people want to try it, though, because we're all contented members of our particular tribes. And if this mystery language ever did appear, I still wouldn't be able to use it at work, because most programmers, even great ones, are too ignorant or entrenched (or whatever they are) to give it a try.

Your blog post shows a profound ignorance of Lisp. It reminds me of FOX news. That so-called news program has been successful by telling a target audience (the people in the red states) just what they want to hear instead of the truth. The weak minded love to be told they are right rather than being challenged with the reality. It makes them very loyal. Likewise there are people who may have dabbled in Lisp but never understood it and maybe this made them feel a bit stupid. Now you come out and tell them that "you're not stupid, Lisp is just badly designed". I'm sure you'll get a loyal following of people who now quote your bogus posting as if it is evidence itself, when it simply garbage.

OP: I guarantee you there isn't one single Lisp programmer out there who uses exclusively Lisp.

For more than a year, my business partner and myself have used nothing but Lisp. CLISP replaced Perl and sh(1) as our server administration language; CMUCL replace PHP and Perl as our web app language; and LispWorks replaced C++ and Java as our fat-client/stand-alone app language.

OP: With Python or Ruby or Java, you've only got one language to choose from.

Only if one of Python, Ruby, or Java address ALL of your needs. On any given day, I'm fixing bugs on a live web app, adding features to a delivered-as-binary desktop app, and tweaking scripted cron jobs on my servers. In light of my previous point, I'll leave it to you to figure out which language let's me do it all with minimal fuss.

OP: But they're [macros] fraught with problems. One is that they're not hygienic...

C'mon! If you're cruising' for flames, stop inserting the punch line. You aren't nearly as much fun as you initially appear.

OP: Some editors can do this, but none of them (that I'm aware of) handle macros as cleanly or seamlessly as they do normal functions.

Please define `handle', 'cleanly', and 'seamlessly'. What you are `aware of' is becoming apparent.

OP: In fact your probability of learning Lisp is decreasing over time, as other languages continue to close the gap in the Lispy areas, and simultaneously increase their lead in non-Lispy areas where Lisp is making little (if any) progress.

In your homepage you have an article talking about the great founding engineers of Amazon that only allowed lisp (I doubted it was CL that we're talking about here, from your article it was more like emacs lisp) and c for development and how you admired their skills.

It is very obvious you never have a first hand experience programming in lisp (in a nontrivial project) and become fluent in it.

There's no 'hobby' lisp projects in your homepage, you never asked any "newbie" questions on cll that one would ask when learning lisp.

You seemed knowledgable about lisp, but only at the surface. You might have skimmed through OnLisp, sicp, cltl2 or whatever, but it is very ob vious that you don't know squad about using lisp.

Please, if you just want to "talk" about things that you don't know, proclaim it at the title of your post (WARNING: I'VE NEVER USED LISP PROFESSIONALLY, THIS IS JUST TROLLING)

Chuck writes: '...no way to get a list of the keys in a hash table (or if there is, I can't find documentation in it)...'

From the Common Lisp HyperSpec, the MAPHASH function looks like it can be used to do what you want: it iterates over a has table, calling a function for each key/value pair. So here's a HASH-KEYS func (it's much prettier with proper indentation, which blogger doesn't allow):

- the newest version of each of the languages you mentioned is the canonical version, and they make this possible by growing the language but maintaining backwards compatibility. Of course this results in permanent, un-fixable bugbears in each language, but they at least try to evolve the languages in the directions they can. Haven't seen anything new from CL in a decade or so.

In CL, you can evolve the language yourself, and don't have to wait for anyone else to do it for you. And you can do this without getting similar compatibility problems.

If Allegro were free, this might be a lot smaller issue, because they're definitely innovating at Franz. Making it free might turn it into the defacto spec. It'd require a new way to monetize it, but I don't think that's impossible.

Franz is in business for several decades now. They are quite successful. I doubt they will change their strategy. Innovation also happens elsewhere. It sounds as if you were looking for a benevolent dictator, but this doesn't fit the Lisp culture.

- I didn't say Scheme is better; I said most newcomers conclude that it's better

If even the newcomers think it's better, it must be, right? ;)

- Unclear what PG's long-term intentions for Arc are. Let's hope he has great taste. And that he finishes it.

I am not holding my breath, and I don't think it's PG's intention to suggest to do so. He clearly says that he is concerned about a language which is supposed to be used in 100 years from now. In the meantime, we can get a lot of things already with the existing Lisp dialects.

- C'mon, valuing a non-changing spec?

Yes, absolutely! This is inherently tied to the fact that you can evolve the language yourself (or use libraries that do so). The need for having some "authority" evolve the language for us is a lot less necessary, if at all.

Languages have to grow.

Most have to, but not all of them.

- If Dylan's OO system is "more seamless" than CLOS, then why is everyone else responding that CLOS is seamless? There are either seams or there aren't, right?

No, it's a gradual thing. CLOS is seamless enough, so to speak. Common Lisp isn't about perfection, it's about pragmatics.

What's the secret clunkiness of CLOS that I missed that makes Dylan better?

I didn't say that Dylan is better, I said it has a more seamless integration of generic functions. (Didn't make me want to switch yet.)

- I'm not sure I buy your "virtually not an issue" assertion. Paul Graham spends an awful lot of time in On Lisp (all of Chapter 9, for instance) describing techniques for avoiding this "non-issue". And I sure see a lot of gensyms scattered through all the CL macros I've grabbed and/or written. Are you sure you're not just writing it off because you're used to it?

I am writing it off because I know that many others got used to it as well. Many useful macros are non-hygienic, as is also described in On Lisp. So it's a matter of what's the default. The default in Common Lisp works pretty well, for several reasons. Finally, you have to understand these issues anyway, no matter whether you use a hygienic macro system or a non-hygienic one.

- You're absolutely right: I haven't used Common Lisp much. I've used Scheme a lot, although not in the past year or so.

Well, then consider the possibility that you should spend a lot more time in actually coding something with Common Lisp before being able to judge it. Common Lisp was not designed for the occasional user, but for expert programmers. It has a steep learning curve that only pays off in the long run.

I'm not arguing that you can't get work done in these languages. You can get very solid work done in all of them, with enough persistence. My basic impetus for blogging about this stuff stems from the desire to have my cake and eat it at work. I want a language with Python's succinctness, Lisp's extensibility, C's performance, Erlang's parallelism, Java's tools, Io's orthogonality, and apparently Qi's type system. :-) And more besides.

Go ahead, noone is stopping you from designing that language. Why haven't you started already? ;)

- The fellow who claimed Lisp has no instanceof operator is incorrect: (typep object class-name)- The title of this post is ridiculously asinine.- "Valuing a non-changing spec" is totally reasonable. You can find code in the CMU AI repositories or in books that were written twenty years ago that still runs. The core parts of the language have little reason to change. Remember that Common Lisp was the distillation of wisdom gained from 30 or so years of use. It is reasonable to expect a language to have to change if you're using one of the newer languages, like Java or Ruby or Python.- CLOS isn't seamless in that some of the functions in the standard are regular functions instead of generic functions. Someone mentioned that LENGTH can't be overridden. This is not a big problem in practice - how often do you need a sequence that can't be built out of lists, strings, arrays, or the general cons structure (which can describe arbitrary trees)? Chopping off method dispatch time for such commonly-used functions isn't such a bad thing either.- Hygiene truly isn't an issue. There is a learning curve involved, but it is not at all difficult to open up a macro with a WITH-UNIQUE-NAMES form to declare the symbols you'll be using in the macro. This does not really jibe with your "hellishly hard to get right" comment. As Pascal (I think) pointed out, hygiene is not as big a problem in CL as it is in Scheme because the variable namespace is the only one 99.99% of macros muck around with and gensyms solve that handily.- Someone in the comments claimed that most of the macros anyone need are already in Lisp and that macros are not terribly useful. This is tremendously false, at least in my experience. Macros can seriously cut down on code duplication and the potential bugs and maintenance issues it can cause. Skim through this for some brilliant, brilliant macrology.

Thanks for the tip, Eadmund. Gotta say though, I find it funny that a language that comes with the kind of high-level functions CL has, would make someone write a function like this to do what lowly PHP has a function built-in for.

- The fellow who claimed Lisp has no instanceof operator is incorrect: (typep object class-name)

I didn't so much claim that it doesn't exist as claim that I still couldn't find it. Thanks for the tip, Bill! And a sarcastic thanks to Pete and the authors of CLtL for making this *so* easy to find in their works instead of coming in here and looking like an idiot! :-/

- Someone in the comments claimed that most of the macros anyone need are already in Lisp and that macros are not terribly useful. This is tremendously false, at least in my experience. Macros can seriously cut down on code duplication and the potential bugs and maintenance issues it can cause. Skim through this for some brilliant, brilliant macrology.

You've misread me, I never said that macros weren't terribly useful, in fact I said that I appreciate their usefulness, but haven't personally had to write very many. Being already quite used to reducing code duplication in other ways with boring old things like functions, I tend to go that route first.

I stand behind my statement that most general-purpose macros you could need are already there, the product of decades of people writing macros. Any macro I can forsee having to write myself would have to be for something pretty specialized, and chances are the functionality I'd be after isn't too long a jump from what can be done with a couple of functions and macros that already came with CL.

Basically, macros have won, having proved their usefulness by becoming almost invisible. A CL programmer uses existing macros constantly without really having to think very hard about that fact that they're macros.

I don't think of writing one's own macros as an essential core necessity for someone to be able to program effectively in CL, but more of an advanced technique to learn once you're already quite adept with the language. In fact I think it's necessary to be pretty skilled with Lisp in the first place to be able to start writing good macros. But seemingly it's the first thing Lisp fanatics want to point out to would-be converts about why the language is so special, when I think there are plenty other valid and more persuasive things to focus on that won't confuse the hell out of the newbies you want to attract.

Like I said, my interest in Lisp programming was solidified in a Programming Languages course I took where we used a fair amount of Scheme and by the end of the semester we had all built an interpreter for a fairly simple made-up language called Babel, and none of us needed to write a single macro to do it. I came away with a favorable impression of the Lisp family, and a desire to learn them more in-depth, without needing macros to convince me.

It's my impression that a lot of what Steve is getting at with this post is the ways that Lisp hurts its future by scaring off newbies. Macros tend to involve hairy-looking syntax, and all the potential hygiene pitfalls being bickered about right here that you have to tread carefully around, and in the end, Macros are a luxury more than a necessity. Yet the Lisp community seems to feel it necessary to fanatically push them on new Lisp users very early (as in PCL), and I think it freaks people out.

"All day I endured challenge, tough questions, attacks to my credibility and tough stares. If I didn't fancy myself a tough martial artist, I would have cried in the restroom. But I held it together. From that day forward, I made it my mission to try to understand the mind of a highly technical person since it was so far away from my own."

As we can see, this attitude is not unique to Lisp programmers. In fact, the comment about people unable to learn lisp being "stupid" and the one about "smug lisp weenies" are essentially of the same class.

Onto more substantive topics, we can easily see that Lisp does not have the advantage of having popularity as an explicit goal (or of "low learning curve", etc). My understanding is that Python and Ruby do (please correct me if wrong), so it is not surprising they are successful at gaining a decent number of users, keeping in mind their lack of funding relative to Sun/Microsoft.

So it's really tempting at this point to do some sort of biased consolidation of the comments so far into a new blog post, and introduce it as a topic for discussion. But then I'd just be a new newsgroup, one in which my posts appear in a bigger font (metaphorically speaking) than anyone else's, which is lame.

Let's get a few important details out of the way here.

First, it's true that I haven't done enough Common Lisp programming. But I did stay in a Holiday Inn Express last night.

True fact, though: I have *tried* to use Common Lisp many, Many, MANY times over the past 2 years or so. There are certainly many things I instantly love about it. Pascal's recent comment about how it's growable hits right at the heart. I feel suffocated by other languages because of their obvious non-extensibility, relative to Common Lisp.

The basis for my blog entry (and c'mon, really, is my title and thesis *that* much of a stretch?) is a simple question: could Lisp be better?

I mean, people ask this question of Python all the time. They do it in a ritualistic form called a PEP, and despite its flaws, it at least exists. Java has JSRs. Ruby has, ummmm, something probably. The point is that there's a governing body, elected or otherwise, that you can appeal to in order to get new features added.

And let's face it: in spite of Lisp's incredible growability vis a vis any other language out there, including Scheme, there are certain directions in which it cannot grow. You can't fake threads with macros; not in any way that matters. You can't fake Unicode. Lisp has to talk to the external world, and historically (ancient history, that is), that's been something Lisp has had difficulty deigning to do.

I mean, you can pick on my technical choices all day long, but you *know* there are problems with Lisp, things that you wish were better. Its package system, for instance. That's a no-brainer. Or its extralanguage facilities like the asdf installer, which by all rights should be as ubiquitous and universal as CPAN.

And cross-platform is a joke. I run Windows, which means I can't run SBCL, which is by all accounts superior to CLISP. And CLISP is nice, but also quirky (according to Siebel and my own experience) and slow.

But like I said in my follow-up article, my concern isn't so much about whether Lisp has any reasonable implementations. It's whether the Lisp community will admit that Lisp could (and should) in fact be improved.

The basis for my blog entry (and c'mon, really, is my title and thesis *that* much of a stretch?) is a simple question: could Lisp be better?

Yes, of course Lisp could be better. Since there is no "perfect" programming language, there is always room for improvement by definition. The question is: What's the right approach to "organizing" change?

I mean, people ask this question of Python all the time. They do it in a ritualistic form called a PEP, and despite its flaws, it at least exists. Java has JSRs. Ruby has, ummmm, something probably. The point is that there's a governing body, elected or otherwise, that you can appeal to in order to get new features added.

Here is an important difference between the Lisp culture and the culture of many other programming language (not all of them): In Lisp, you don't have to wait for some authority to have your language changes approved. You can just change your language and continue to focus on your real problems. The need for some authority is a lot lower than elsewhere.

Besides that, I find the notion of a "benevolent" dictator highly questionable. The underlying assumption is that other people know better what's good for me than myself (or my group of programmers, for that matter). I think that's a dangerous attitude and couters the notion of enlightenment that I find very important. Check out "The Guru Papers: Masks of Authoritarian Power" by Joel Kramer and Diana Alstad for a thorough treatment of what I think the problem is.

Here is a quote from that book: "Sadly, society and parents insidiously put out messages from childhood on that others know what's best. Many people are deeply conditioned to expect and hope some outside agency, power or person will solve their problems. Letting go of expectations or even wanting this is difficult, partially because what one is left with is oneself and all of one's limitations."

Lisp indeed leaves you with your own ideas and your own limitations. It doesn't pose any artificial restrictions on your programs that you have to work against, and it doesn't provide any "color by numbers" examples that make you feel like you have achieved something. Lisp requires you to be creative.

And let's face it: in spite of Lisp's incredible growability vis a vis any other language out there, including Scheme, there are certain directions in which it cannot grow. You can't fake threads with macros; not in any way that matters. You can't fake Unicode. Lisp has to talk to the external world, and historically (ancient history, that is), that's been something Lisp has had difficulty deigning to do.

If you really want these things, you will find ways to do them, even conveniently so. Multithreading is part of many Lisp implementations, Unicode also, as well as foreign function interfaces. Lispers use these things all the time, and even use compatibility layers when necessary in order to create code that is portable across several implementations. It requires more initial effort, as is the case for the whole language anyway, but it isn't a black art either.

I mean, you can pick on my technical choices all day long, but you *know* there are problems with Lisp, things that you wish were better.

Of course. I would like to see a Squeak-style development environment for Common Lisp.

Its package system, for instance. That's a no-brainer.

I think Common Lisp's package system is among the best things on earth since sliced bread. Really. ;)

That's supposedly again just a detail, and not related to your general claim. However, it's indeed important that this is really just a detail. The package system (including its precursors) has been in use since the 70's, and people have successfully managed to develop code using it. Changing it (which I think is largely impossible without breaking its basic concepts, but that's another discussion) would just result in unnecessarily breaking legacy code, and you would probably not have achieved a substantial improvement in return. Many of the suggestions for improvement of Common Lisp have this flavor: little, mostly aesthetic improvements but breaking legacy code at the same time, which is uncalled for. I mean, you can really download code from software archive that are 20 years old or older, and the code just works! What's the point in throwing this accumulated knowledge away? Just that the language caters for some dubious notion of modernism or popularity, which will probably change again in a few years?

Or its extralanguage facilities like the asdf installer, which by all rights should be as ubiquitous and universal as CPAN.

Is CPAN an officially sanctioned mechanism? Does it give any guarantees wrt the quality of code that you can get there? If not, then we're not really worse off.

It may sound surprising that for a language that's nearly half a century old (i.e., Lisp) that such mechanisms aren't as mature as for other languages, but recall that the "Common Lisp new wave" is only a few years old, and is indeed in a relative early stage compared to other communities. Many of the things that you say are necessary will most certainly come into existence in the next few years, I am pretty sure about that!

And cross-platform is a joke. I run Windows, which means I can't run SBCL, which is by all accounts superior to CLISP. And CLISP is nice, but also quirky (according to Siebel and my own experience) and slow.

Windows is a comparatively neglected platform in the Common Lisp community, at least in the open-source part of it. If you want good Windows support, you probably have to pay for it. And there is another technical error: Clisp can be a much better choice than other Common Lisp implementations, depending on the technical requirements. Especially if your application depends on generating and compiling a lot of code at runtime, the fact that its compiler is very straightforward and fast, and only has to produce bytecode, can be a big advantage. In comparison, SBCL's compile produce highly efficient machine code, but the compiler is very slow (that's why it produces very efficient code ;). In scenarios with runtime code generation, this can be a serious disadvantage. Finally, Paul Graham had chosen to use Clisp vor Viaweb because it had the most efficient stream implementation back at that time. So it's not good to pick an implementation based on low-level efficiency considerations, because they tend not to matter much in practice. The overall structure and implementation strategy, as well as the choice of algorithms, is typically much more important.

But like I said in my follow-up article, my concern isn't so much about whether Lisp has any reasonable implementations. It's whether the Lisp community will admit that Lisp could (and should) in fact be improved.

Of course, but IMHO just not in ways that people typically think of. And it's certainly not a reason to write off Common Lisp, or to appear to make that suggestion.

Let's not beat around the bush anymore. Smug Lisp Weenies is putting it nicely.

In reality, you've got a bunch of people with pathological personality disorders.

Circle the wagons, torch the infidelhttp://groups.google.com/group/comp.lang.lisp/browse_thread/thread/18cc7a823112559c/e86335fb4cc6c367?hl=en&

And if that doesn't show up on the post, it's the comp.lang.lisp "How Lisp Sucks" thread.

But the Smug Lisp Weenies like it the way it is. They don't want Lisp to become mainstream. They want something they can dork around with and not have to produce with.

If Lisp became mainstream, then they would have to go learn Haskell or something. Haskell is a far superior language anyway, and doesn't suffer from a bunch of sniveling weenies running around producing nothing.

> You choose a language based on your perception whether their users are "weenies" or not? That's weird. > I thought it should be based on whether the language is good or not...

Well, yes and no. When one runs into problems with a language, one does well to consult the experts. If those experts are a bunch of knuckleheads, whether or not you get the answer, you're going to come out of the experience crushed. So after that, either you'll avoid asking for answers to questions, and thus end up doing much more poorly than you could otherwise, or you'll continue to ask questions, hat in hand, knowing that you're in for a (metaphorical) beating. This doesn't seem likely to engender enthusiasm for a language :).

> Here is an important difference between the Lisp culture and the culture of many other programming language (not all of them): In Lisp, you don't have to wait for some authority to have your language changes approved. You can just change your language and continue to focus on your real problems. The need for some authority is a lot lower than elsewhere.

This is great, if and only if you're running the company or organization in charge of the code, or if you're going to work at one place. What do you do when you've spent a year plugging in various external-system support bits, and a year later you change jobs, and you have to hope that your new employer will give you a year to do the same thing all over again? Not, I expect, that there is a huge pool of Lisp-programmer jobs out there, so this may be a moot point....

In any case, if you use Java, or C++, or C#, or Ruby, or whatever, you have a bunch of library bits, and maybe a few locally-developed libraries, and you're coding on your actual product every day. If you change jobs, you go to your new job and start coding on the new product, threads, GUI, and all. Yeah, you're not a Lisp God, but...you're shipping product, and that counts for something.

Steve: I think you would do better to learn Lisp before disparaging it. I've been known to critisize Lisp myself- in fact, since I program in CL every day, you can usually find me cursing John McCarthy's ancestors and descendants, 17 generations in each direction (and more importantly bitching about the authors of the spec, and impugning the implemeters of my implementation). But I have reasons, and I could actually justify my tirades, if I had to.

You, on the other hand, are clearly not capable of making cogent criticisms of a programming language. You mix legitimate criticism with complete nonsense, and you are as apt to be wrong about factual matterrs as to be right about them. The only conclusion I can come to is that you do not understand programming language design very well, and are not capable of programming outside a fixed template.

"You, on the other hand, are clearly not capable of making cogent criticisms of a programming language. You mix legitimate criticism with complete nonsense, and you are as apt to be wrong about factual matterrs as to be right about them. The only conclusion I can come to is that you do not understand programming language design very well, and are not capable of programming outside a fixed template."

That's an interesting description of Steve Yegge, and from what I can tell, it may even have some merit. Apart from that, I would like to know why Lisp isn't becoming more widespread. Are all those problems with Lisp really only imaginary? If Lisp really is such a great language, why doesn't anybody notice? Just asking.

Many people have noticed it, have used Lisp and have done well. I suspect that you won't find many of the really interesting people spending their time answering comments on some obscure blog, though (this applies to any language).

I completely identify with your post about the one true language. I was one of the kids washed upon the Lisp shores by the Paul Graham hype. From a CS education in Java and C++ to writing macros in Lisp... it blew my mind.

Then I moved on to Scheme... and with it SICP. The video lectures blew my mind yet again, and I came to truly respect the power of first class procedures, lexical scope, and programming with combinators. The section of the course on interpreters gave me an entirely new appreciation for programming languages.

But as grateful as I am to Common Lisp for its introduction, and as happy as I am about Scheme teaching me so much, I now find that I cannot use them. This frustrates me to no end, and has begun this hunt for the "perfect" language that will probably never end.

I looked at Haskell, but I don't have time to learn its radical ideas yet. I simply want to convert on the lessons I have already learned from the Lisps. Ruby looked promising, but it aint it either. Now I'm looking at OCaml. It looks pretty good but I guess I'm going to have to wait for that one true language that is elegant but also has mindshare. This is just one of my many interests and I am torn between them all. I go back and forth between being convinced that I must write my own language and searching for someone else's language that I can get behind and really push. Then some other times I just want to get things done, and it seems like as sad as this is, Java--with its utter lack of first-class lexical closures and type-inference and every other feature I am convinced should be sweet--is emerging as the tool most frequently useful for the applications I am needing / wanting to build.

But one thing is for sure... the field of programming and programming languages is incredibly young and we have a lot of learning to do. Paul Graham did a service by shining light on a functional programming language and launching people on these myriad quests for beauty and truth. It's going to take a long time for the lessons of Lisp and ML and Haskell and Smalltalk to coallesce into something that is practical with a lot of mindshare. I'm starting to wonder if it won't take a major software company's backing to really get something of true quality off the ground. You can say what you will about the cathedral, but the organization of the Java community is pretty amazing and that has to have something to do with Sun and IBM and perhaps the design of the language? I can't put my finger on it. Perhaps I'm not smart enough. But good post... Lisp is awesome, but we need to be thinking about how to bring it into the light, whether or not the result is named Lisp. Maybe I should shut up and use something that is good enough and make something with it... I guess I am, and that's called Java. But no. I'm a perfectionist. I want something that feels as pragmatic as Java and as cool as Lisp. Lets keep working.

Your comments about CL's package system are interesting. While attending this years JavaOne I noted that the proposed super packages for Java 7 look very muck like CL packages, at least concerning functionality.

As a LISP newbie (I did Scheme in my undergrad) this is a fascinating article and comment thread. It seems like LISP stalled out in the early years because of performance issues, and then never was able to pick up enough steam to avoid fragmentation.

Now we've got a situation where a bunch of programmers who are used to being the smartest programmers in the room, all have decades of experience doing things a certain way, and everyone's too damn opinionated to agree on what forward progress means.

It's kind of sad really, that a language like Java has so many libraries and can do so many things out-of-the-box simply because the majority of programmers can't or won't think of a better way to do things. They just take whatever Sun gives them and then write some Frankenstein APIs to ship their product.

Anyone with a little more propensity for abstraction will say Lisp is a better language, but the agreement ends there, and any kind of unified community and roadmap is stalled out by a hailstorm of informed religious opinion. It's a hell of a lot harder to convince a lisper of anything than it is a java-head who may be religious, but will change his mind based solely on peer pressure.

The fact is that different approaches have their pros and cons, but there's also a benefit to be had from a critical mass of developers.