Paul Donnelly wrote:I don't know about that. For every outspoken Lisp advocate, I think there are many more Lispers who hesitate to speak up for fear of overselling it.

OK, but let's think about that for a second. Who gets heard?

Nobody. Yeah, some people hear Paul Graham if they hang out on the right kind of web site. So non-Lispers are either hearing the occasional praise from some super-hacker they've never met, or nothing at all. Where's the common man in the Common Lisp world? Keeping his head down, or nonexistent, one assumes.

Paul Donnelly wrote:Nobody. Yeah, some people hear Paul Graham if they hang out on the right kind of web site. So non-Lispers are either hearing the occasional praise from some super-hacker they've never met, or nothing at all. Where's the common man in the Common Lisp world? Keeping his head down, or nonexistent, one assumes.

Sorry to take so long to respond to you- it has been a hectic week. Well- you know I wonder how many people worldwide actually program a fair bit in CL. It's hard to know, but I'm pretty sure it's a small number. On the other hand, I think a _lot_ of people have read Graham's take on things. "Beating the Averages" still pops up everywhere. And while I think that "Beating the Averages" is a fine piece of writing, if a bit dated, a lot of people seem to get the wrong idea from it. The thing is that Graham was already a "super-hacker" when he sat down to write ViaWeb (he was also operating in an environment where you _had_ to be a pretty decent programmer to write a non-toy webapp, and where you really had to do a lot of the heavy lifting yourself. In a lot of ways it was a perfect time to be using CL, because there weren't really a lot of reasonable web application frameworks out there at the time so most people were hand-rolling almost everything.)

Anyway, that aside, what I'm getting at is that I think a lot of people miss a couple of things that are implicit in "Beating the Averages." CL has some real advantages as a programming language, but it's not magic. It takes a real investment of time to learn to use CL at a level where it pays the kinds of dividends Graham is talking about. I know I don't get as much from it as I could- I've been programming in CL intermittently for years, but I'm no Paul Graham .

Beyond that, the problem of the "little webapp" is one that's been pretty convincingly solved at this point. I mean, if you want to write a digg or a reddit or some other app that's just enough not like them to be fresh but play in the same space, CL is likely a bad choice unless you're already a pretty good CL programmer who has already worked out how to do that kind of thing in CL without much muss or fuss. Because, honestly, you can knock out a first draft of a site like that in a weekend in rails, or whatever, and how successful it becomes is going to have a lot more to do with things like how influential the early adopters you manage to get are, and whether or not you design the interface so that people don't hate it, and.. well, luck. I mean- CL is actually a perfectly reasonable choice for that kind of thing, but so is *gasp* -choke- PHP.

I loathe PHP, btw. But actually I got paid a princely sum to learn it (or its basics, at least), by someone who had promised more than they could deliver, spent a month messing around, and then called me asking me to write a site from scratch in time for a client meeting the next day. This was on a Thursday night, so I had him reschedule the meeting to Monday and spent the weekend in a caffeine fueled haze at double my hourly rate. Interestingly, this was not the first time this guy had paid me to learn a programming language, or at least to do my first real programming in one. A year or so earlier he had asked me to write a subsection of a site for well-known college, and I had written it using mod_python, having never programmed in Python before, beyond a few exercises from the O'Reilly book on the subject.

That might seem like a digression (well, I suppose it is, really), but I do have a point or two waiting in the wings, I promise. The first is that I could pick up Python or PHP and write a web application in it, at fairly frightening hourly rates (more on that in a sec, god help us) without anyone feeling like I was stealing from the company, or the client, or whatever. In fact the Python app I wrote for him was used for 6 or 7 years, and had a _lot_ of page-views over that time period, but I was only asked to revise it once- they wanted to downgrade their python version (because of the old RH stupidity about Python versions they wanted me to remove a call to a function in the string library that had been added to Python 1.6 but was missing from 1.5.) The only real overhead of using python was looking up some API stuff- I mean, I knew C and Java and Perl and.. well, anyway, Python wasn't much of a challenge.

Now there were two really significant factors involved here. One is that if I had tried to do this in CL, without knowing any CL, I would have been stealing from the company, and rather obviously so- I wouldn;t have gotten things done on time or on budget. Or maybe I would have, actually- I was always good at that. But I wouldn't have been writing code that used the things about CL that make it a good language. Because you can't pick that up in a weekend.

The other factor is that this is all ancient history. I think I wrote the python app in 2000, and the PHP app in 2002. In 2000 there still weren't _that many_ people who could actually deliver working web applications reliably. People were starting to get it, but I had already been doing it for a few years, so I was pretty fast at it, and I was a known quantity to the people hiring me (if I said it would be done on time for the client meeting on Monday, it probably would be done.) If I spent a few extra hours picking up a new language, no biggie to anyone. I never asked what the guy who was pimping me out in these cases was making (well, I never pressed him on the subject ), but I guarantee you that it was a lot more than I was making. That wouldn't fly now, of course, since there are a lot of 13 year olds who can do that kind of stuff in Rails easily enough (thank god they can't be trusted to deliver anything or everyone in the web industry would be working for comic-book money.)

So my point(s) (at last.) First off CL isn't going to help anyone beat the averages in a domain where the programming problems are not particularly significant. In fact CL might be a real drawback to someone not already used to its arcane ways. I mean- if you can write a first working draft of a site in a weekend in Blub, but it takes you two weeks to figure out how to make it work in CL... basically, when the average is within a certain epsilon of the optimal, in terms of how useful a language is to you, you can't beat the averages by using a better language. In fact, you can't beat the averages by being a better programmer, in that case. It just stops mattering. Anyone with a pulse can get a reddit clone up.

Second, the advantages of making the jump from Blub to CL are not like the advantages of going from C to Python or Ruby for things better written in Python than in C. I mean- I started out as a C programmer, and C is pretty much indispensable for certain kinds of systems programming, but it was a great weight off my shoulders when I started programming in garbage-collected languages. Right at the end of the time I was doing c++ I started to figure out how to do a style of c++ programming that requires very little manual memory management, but even so... c++ is a lot of work for what I consider to be a disproportionately small return (that said, c++ is actually a pretty damned interesting and powerful language in the hands of the three Jedi c++ programmers who survived the Java wars.)

So when you go from C to Java, it is a bit like magic.. you get to stop worrying about memory, to some extent. Or- well, at least you get to stop worrying about manually freeing it... it's not that hard to leak memory in Java, actually. And then when you go from Java to Python you get released from all that B+D stupid static typing nonsense (I'll leave smarter static typing alone here.. another day). So up until this point it really is kind of like magic. Without having to learn much you've multiplied your productivity by a lot.

But if you go from Python to Ruby or from Ruby to Smalltalk... well, it's not magic anymore. The low-hanging fruit is already gone (to mix a metaphor). You're still climbing Blub mountain, but it's no longer free. The sunny meadows wherein you frolicked so recently, celebrating your releas from "malloc" and "int x;" have given way to a more difficult landscape. Until this point you've been able to just walk up Blub mountain, and spread your picnic wherever you choose, but now you are going to have to do a bit of climbing. But the thing is- you know how to walk, because you've been doing it forever. So there's a qualitative change here.

But you don't have to climb too much to get at least some of the benefits of Ruby and Smalltalk. This is at least partially because other climbers have done a lot of the work for you. But then you look up at CL and.. it is actually a pretty daunting climb. You're going to need the pitons and the ropes for this one. There's no obvious place, on the way up, to lay out your picnic blanket and have a snack. In fact, you're going to have to learn a lot about climbing, while climbing. That's the nature of the more rarefied reaches of Blub mountain (whether or not CL is the summit of Blub mountain is left as an exercise for the climber.)

So, to drop the metaphor for a second, people get annoyed because 1) Paul Graham told them they would be rich if they started programming Lisp, and 2) Lisp is hard, so they can't be immediately rich, as they would certainly have been if they had just used Ruby in the first place.

This isn't Paul Graham's fault, because what people _ought_ to take away from "Beating the Averages" is this: If you are a really good CL programmer, with years of experience, including all those hard bits that involve math, algorithms (but I repeat myself), etc., and you can get in on something just at the point that it's deemed new and important, but at the point that it's still really hard.. well, you too can make ~20 million from CL, just by being a good programmer. Or you can make money by just being a really godly programmer- call that the ITA model, if you will.

In short, CL offers very few advantages for easy programming tasks, (unless you already know it well, in which case it's actually kind of nice for trivial stuff), and it offers very few advantages for people determined to make a lot of money from easy programming tasks. Unfortunately a lot of people who fit that description get really enthusiastic about CL just because Graham is a good painter, with words (I have no opinion on his canvases) and go rushing up Mt Blub (you knew it was coming back) in a T-shirt and jeans. Unfortunately, people who fail to climb Mt. Blub in a T-shirt don't die of exposure up there- they just get bitter about the whole thing and troll c.l.l for a while. Now I have nothing against trolling c.l.l, but...

Anyway, another problem with this whole thing is that it's basically impossible to bitch about any aspect of CL without being mistaken for one of these crippled former "climbers"/entrepreneurs. I mean- CL has a boatload of issues (though they actually fill an amazingly small boat when you consider that CL was designed by committee more than 20 years ago) but none of them are fatal. So basically, even if you love CL you can't vent about its broken bits without someone giving you a lecture about the ANSI process.. Well, I'm guilty here- I've given some lectures on that subject myself, and will again. Call that a meta-defect of CL .

I've never expected CL to become a dominant language again. I take a Fitzgeraldian view of programming languages. I guess I think about CL in the same way that many Jews have tended to think about Judaism. I'm not all that interested in gaining converts (CL is not for everyone), but I can see the value in educating those who are drawn to the faith. Not that I am very knowledgeable myself- I am so lacking in understanding that I might benefit from explaining some things.

One thing I've considered is thoroughly annotating the sources of important libraries. Actually, more than annotating.. doing something like literate programming after the fact. I mean- it's easy to say to someone who asks about what code they should read: "read cl-ppcre." But it's hard to get a toe-hold in even a small (but rather complex for its size) library like that. It's doubly hard if you are still learning the language.

I've been a critic of undergraduate CS education since- well, actually I started bitching on the first day and I haven't stopped since. I've been asked to do better by people who wanted to score some rhetorical points. It occurs to me that undergraduate education would be a lot better if it involved more exposure to real code. So I've been looking at things like cl-ppcre to see if they could be written up in this fashion. I don't know where I would post such a thing- maybe here if it were not too large.

Anyway- I know that this post is long and digressive. It took me hours to write it. I suppose that given a few more hours I could make it half the length and four times as persuasive. I hope you can follow the thread of logic that I meant to wind through the whole thing.

Paul Donnelly wrote:Good for you with the headlocks.

That was actually based on a true story, wherein I put a small German VB programmer in a headlock.But I embellished it a bit. It wasn't over him misunderstanding "let." It was more about him misunderstanding the nature of the stack, and trolling me in IRL. That's really the wonderful thing about spreading the word in person. You can troll back, and be sure your headlock has met your mark. Of course this was a buddy of mine, and had we not been talking about programming I would have been putting him in a headlock for other reasons. I don't recommend just going around putting every German you see in a headlock- less than half would be about right, in my experience.

Last edited by duncan on Mon Jun 08, 2009 4:54 pm, edited 1 time in total.

This is all good stuff you've written, and I agree with almost all of it (one thing I wouldn't say is that Lisp is hard per se. Learning to take advantage of everything Lisp offers is very hard though).

duncan wrote:It occurs to me that undergraduate education would be a lot better if it involved more exposure to real code. So I've been looking at things like cl-ppcre to see if they could be written up in this fashion. I don't know where I would post such a thing- maybe here if it were not too large.

Yes, I think that's very true. You don't learn to write prose or poetry without reading it. Undergrad programs seem to focus largely on writing code, either together in class or for assignments, meaning students are mostly exposed to their own shaky code. An annotated "Lisp Reader" with problem sets related to working on the programs therein would be very cool. Ditto for other languages, of course. PAIP is pretty close to that, but it's not beginner-level.

duncan wrote:Anyway- I know that this post is long and digressive. It took me hours to write it. I suppose that given a few more hours I could make it half the length and four times as persuasive. I hope you can follow the thread of logic that I meant to wind through the whole thing.

jwm-art wrote:The analogy with mountain climbing captures pretty well how I feel at the moment about CL. In fact, I've taken to picnicking on the meadows again, first on bash scripts and now PHP...

Hmm- well, look. PHP will give you VD, in the long run. I mean- if people give you money to do PHP that's all well and good, but you shouldn't take it seriously. A pimp can't stand those broken reference semantics. Anyway, the meadows are reserved for people who do Python or Ruby or some-such . Climbing Blub hill begins with the first step, IMHO. [EDIT: but being good at bash is sort of perversely useful so...]

While I believe the central message of the essay is still relevant and certainly entertaining, the story no longer holds up as a reason to use Lisp without providing suitable context. My hope is that someday we'll have a new essay along similar lines but discussing the use of Lisp in an entirely different problem domain. I still believe that CL occupies something of a niche where high levels of abstraction and decent runtime performance are available in one language (Haskell and OCaml come close but you have to be happy with the type system to get along with them). For my current work, I don't think I could find a more suitable language than Lisp so perhaps one day I should document my experiences, even if they don't end quite as profitably as Graham's

Well said, Duncan. I agree with you wholeheartedly about both the difference between CL and other languages like Ruby (smaller than the distance between C and Java) and the fact that you can't criticize CL without somebody thinking you're being an ingrate. In fact, I agree with almost everything you wrote. Well said.

Personally, the biggest thing that I take away from CL is an overall philosophy of Lisp, not a specific impression that CL is the One True Programming Language. Lisp is bigger than CL. CL made a lot of compromises at the time of standardization to keep already-running code running. It also broke a lot of new and interesting ground (CLOS and the MOP). It didn't get everything right (e.g. streams), IMO, but no programming language does. I think "new Lisps" like Clojure are great in terms of getting the Lisp innovation machine moving again, though I also see some warts there (lack of tail recursion because of JVM limitations). It had stalled in the early 1990s and never really restarted. IMO, Scheme never really delivered much beyond CL (the hygenic macro quest seems like folly now, IMO, but at least we know that through experience), though its foundational concepts of lexical scoping and tail recursion (optional in CL) were clearly influential to CL.