Friday, December 09, 2005

I went away on a family vacation at the first part of this week and just got back last night. During that time, I completely missed the whole Redditscandal. It seems like the guys at Reddit ended up choosing to rewrite Reddit in Python because Lisp just wasn't working for them. To some, this seems to be a slap in the face of Lisp. However, if you look at spez's blog entry, you can see that the rationale for choosing Python was pretty sane.

Emacs and SLIME are a killer combination, but I develop on a Mac, and reddit.com is a FreeBSD box. On my Mac, my choices of threaded Lisp implementations was limited to OpenMCL, and in FreeBSD it's CMUCL. Because of the low-level socket and threading code we had to write, reddit would not run on my Mac, and I was always tethered to our FreeBSD development server. Not being able to program offline is a pain.

The biggest trouble that plagued us was that we could never quite get Lisp reddit stable enough to sleep at night. There were weird threading issues that would bring the site to its knees a couple times a day and required constant monitoring.

Now, here's where the Lisp psychosis comes in. Rather than suggesting that the guys at Reddit were dopes, or that they didn't try hard enough, or that they should have done such and such a workaround, blah, blah, I wish Lispers would step up and internalize that the Reddit experience was a great case study and that the community should work to solve the issues it raised. These guys did not have a bad Lisp experience. In fact, they are quite complimentary of Lisp. For example:

Lisp is an amazing language. After spending the entire summer working entirely in Lisp, it's nearly impossible to work in another language and not say to myself, "If only this were Lisp..." Lisp has many goodies that make programming in it a joy: the REPL, macros and the lack of syntax are some. I won't go into the details, but rest assured, it's cool. People become Lisp zealots for a reason.

So here's what I took away from the Reddit feedback:

Lisp is a great language. Keep this point in mind. The Reddit developers gave Lisp compliments; they didn't 'diss it.

Lisp has a balkanized feature set. Some (necessary!) things to build modern applications are not cross-platform. When you have to work on multiple machines and environments, which is more and more the norm these days, there are no open source implementations of Lisp that run across the dominant environments without differences. (CLISP comes the closest, but you may or may not be able to tolerate its GPL license terms.) Because these features are not standardized, you're left writing compatibility layers if you want things to work across platforms.

In particular, networking and threading are problem areas.

Lisp libraries are scant. This is a well known problem in the Lisp community. The standard, basically valid, response is "Jump in and help us write some more libraries." (Kudos to Kenny Tilton for driving this line hard. He's right, but there's also a bit more to it than that.)

The library problems are compounded by the balkanization of feature set. In some cases you can find something that sort of works, but it may work on another implementation, not yours. If it has any dependencies on the problem areas of threads and networking, you've got a long road of tweaking ahead to get it to work. This time is better spent getting on with your real task.

Edi Weitz (yet again) wins the Lisp Superhero award for creating the best libraries out there, bar none. Seriously, if anybody aspires to create libraries that are well-used, go take some cues from Edi. His code is always high quality, his APIs and implementations are always complete, he's absolutely responsive to problem reports, he provides great documentation, and he often tests his code across multiple implementations, trying to make them as cross-platform as possible. In short, you couldn't expect better service and support from a commercial vendor, and Edi releases his code as open source.

First, stop grumbling and suggesting workarounds to the Reddit folks. They did the Lisp community a great service by documenting their experiences. Rather than harrassing them, sit down and talk with them to get more info.

Next, focus on the foundational balkanization problems. In my opinion, the biggest issues with Lisp are the lack of standard (defacto or otherwise) networking and threading APIs. If those were in place, it would be a lot easier to get libraries that worked all over the place. Would this solve everything? No, but it would go a long way and would enable lots of other innovation on top of that foundation instead of having everybody spending time creating compatibility libraries and generally re-inventing the wheel.

Work on getting a good cross-platform, open source Lisp implementation with a liberal license. Like I said previously, I really like CLISP, and I use it for developing on Windows, but the license is not suitable for all code since it all-but-forces your code to be released as GPL. I'd really love to see SBCL or CMUCL ported to Windows, with the full set of functionality. I'd also love to see SBCL's baseline functionality present on all platforms (currently threading only works on Linux, for instance, not BSD). By the way, this is no knock on any other Lisp implementation. I just happen to use SBCL on Linux and think it rocks. If OpenMCL can make the jump off of Mac onto other platforms, that would be great. Or maybe GCL can do the job. I don't know the exact route, but the fact is the lack of a baseline functionality across all platforms is hurting us.

Can I do all the above? Nope. I've tried grokking the internals of SBCL and I'm only qualified to kibitz around the margins. I stand in awe of guys like William, Christophe, Dan, Nikodemus, Gabor, Juho, Alexey, and many others (Update: I added to this list twice already because I felt so bad for leaving out somebody's name--sorry William and Nikodemus. If you're a significant SBCL contributor--or even and insignificant one--believe me when I say I stand in awe of you too ;-). I do try to help out SBCL by reducing the friction for newbies to get started with Lisp, compiling the RPM binaries that are available on SourceForge and maintaining the FedoraLisp.org Yum repository.

That said, I'm going to start working on the networking API issues. You'll see a document from me posted over the next couple of weeks. It's been in process for a few months, but I finally got a hankering to finish it with this Reddit bru-ha-ha. If you're interested in networking APIs and have a desire to help in such an endeavor, please drop me a line (dave at findinglisp). I'd love to spread the work around. If somebody else wants to grab the threading stuff by the throat that would be great.

Above all, stay productive. Use the Reddit feedback to motivate you to make Lisp better. Whatever you do, stop suggesting workarounds to the Reddit guys or making them feel bad for the choices they made. They seem like smart guys, so let's assume that they did what they did knowing all the options (they do have Paul Graham on the board, so I'm guessing that they talked about this before they did the rewrite). They have moved on and our job is to make any such workaround unnecessary for the next crew that tries to use Lisp.

And before anybody says it, yes I know that the commercial Lisp vendors have solved some of these problems. I have discussed before some of my thoughts about the various license terms the vendors use. It isn't that those terms are bad in the abstract, they just don't allow those implementations to solve the problem I'm interested in. In short, I think it would help drive Lisp adoption if a common, cross-platform, free, open source version existed, in the same way that GCC has helped democratize programming in C. The next-best alternative would be a $99 "Turbo Lisp" environment ala Borland's language products in the 1980s/1990s. Perhaps the Lispworks guys will offer such a thing (hint, hint ;-).

I think another good thing would be to come up with a "unified" web site for this rebranded open source lisp (say SBCL).

The web site would contain all the working libs for that distribution (asdf-installable), the source and binaries for the implementation, and lispdoc (or edi weitz or kmr-like) documentation for each library. (example)

And the hyperspec.

Everything on cliki and common-lisp.net

I'd love to see SBCL seem on first glance to be just like the python or ruby web sites, a one stop shop as if it were the only implementation of lisp possible.

What do you think? If nothing else, such a site would indicate where the gaps are and what's being done to fill them in.

Can someone please explain to me why using clisp "all-but-forces" you to use the GPL? In the COPYRIGHT file it says that if you don't use the internals of clisp, your program does not fall under the GPL. Is there something here that I've been missing?http://cvs.sourceforge.net/viewcvs.py/*checkout*/clisp/clisp/COPYRIGHT

Finally someone other than John Wiseman gets it. The goal shouldn't be to build a better Reddit, it should be to make a case for the Reddit guys to move back to Lisp (whether this would happen in reality or not).

Andy, you should probably read the CLISP FAQ, specifically A3.2. It has never quite been clear to me whether the SOCKETS or FFI packages are covered by the exclusions or not. Specifically, FFI looks murky since it isn't explicitly covered by the exclusions and the FAQ mentions FFI as being one of the things that will taint your code. That said, it does say "programs which need to access non-portable CLISP internal symbols (in the packages 'SYSTEM', 'CLOS', 'FFI', etc), must be covered by GNU GPL as well." I have always taken that to mean that if you need to use FFI (and what modern Lisp program doesn't at some point), then you're tainted. I'll ask Bruno for an explicit clarification, however. I don't want to misrepresent the CLISP license. It's a good system and it wasn't my aim to throw mud on it or the authors.

Tolstoy, let me be clear that I wasn't calling for any rebranding. I'm just suggesting we fix problems and keep making things easier for people to use. I do like your idea for putting together a unified distribution, however. This is sort of the Lisp-in-a-box concept, but taken to the next level with libraries and the rest, to make a larger, more full distribution. Anything which gets us closer to a "one-click install" is better, IMO.

I had an exchange with the CLisp crowd (Sam or Bruno) and they said they would clarify the license to make clear that using the FFI and CLOS would not infect an application. I did not follow-up to see if the language changed. This was many months ago. FWIW. DemonKenny (since I cannot persuade Reddit to show me as anything other than anonymous).

please keep in mind that creating good API isn't just converting your "great idea" into code.The trick is to create really good API that would be versatile and flexible and functional. Bad examples here are fu*ked up crazy gtk gnome, and other libraries which change their APIs constantly every sub-sub-sub-sub-version gets released.You know what I mean? API should be designed in such way that it would survive without modification *at*least* a year or maybe even more. APIs should be as stable as it's possible.

Great post, Dave. Perhaps the best thing for the lisp community might be to take a cue from the C++ community, and look at what they are doing with the boost libraries, or rather, the process around them.

These are a set of libraries which are open source and community developed, but subject to rigorous peer review to common standards. This process reflects the reality that formal ISO standardization is a tedious process that takes too long to benefit the community. However, it is a step beyond one random person publishing their independently-developed APIs. To my knowledge, there is nothing like this in the common lisp world.

I think this pseudo-standardization process is even more important for lisp, where there are are so many different ways to do things, that mixing and matching code from different places can be tricky -- e.g. Use CLOS or not?

Perhaps this flies in the face of the "lone hacker" mentality of many lispers, but I do think it would be a good thing overall for the community.

The relatively new CLRFI process is attempting to be a lightweight method of giving Common Lisp extensions status above mere memos and yet lighter weight than anything like a formal standards committee. It's my intention that anything I come up with eventually get proposed as a CLRFI. There's a lot of work before that point, however.

The problem is we need networking libs NOW. We don't have time for a commitee. But after the lib is available it would be helpful if we had a group to review it. I just don't think a design by commitee approach would get any further than the annual "we have update lisp" meeting does.

Please excuse the absolute ignorance out of which this question is asked. I've just begun programing and have only written a few thousand lines in Korn Shell and Perl.

I don't understand the problem with the libraries. Can't you just “link to” or “embed” existing compiled libraries? Why wouldn't programmers in general work on improving a set of “universal” compiled libraries, which could be “imported” by any language? I thought making this possible was the point of Swig, Perl's XS, etc. Why duplicate libraries for all these different languages?

2] Create a CPAN style central for distributing libraries. This is super important because there is exists an insane amount of high quality Lisp code spread across various repositories which people don't know about. ASDF is cute but not practical.

I understand why you chose to switch but I would like to know why you chose pyton (over ruby) and what frameworks you decided to use. Did you write your own or did you use something off the shelf such as django or cherry py? Why not zope?

I think an article on your thought process would be very helpful to others evaluating the vast array of choices available for building web applications.

Thanks for the CLRFI reference. However, I don't think their process is quite right -- it is still stuck in the mindset that it is good to have perfectly defined interfaces, with multiple, competing implementations. This is maybe an offshoot of "Worse is Better". The multiple implementation problem is what got Common Lisp into trouble in the first place. CLRFI should not be a process mainly for defining interfaces and "reference implementations", it should be a process for building up a coherent, useful, peer-reviewed, best-practices code repository.

And, this need not be a long, drawn out process. Dave can write a good set of networking or threading libraries today, and people can start using them immediately. After a few months of outside assessment, they can be peer-reviewed and submitted to this repository.

The idea of multiple implementations being bad comes from observing the compatibility problems that arise from multiple versions. C++ has had many such problems, as does Lisp. Java, Perl, Ruby, and Python have comparatively few. This improves the library situation and tends to make the language more popular with application developers who want to get things done rather than study issues of compatibility.

Babar, when you say "ASDF is cute but not practical," to what are you referring? ASDF seems pretty workable to me. When you say you want something like CPAN, do you mean something that's a larger, centralized but replicated repo, as opposed to ASDF's dependence on CLiki with distributed storage of the files? Both models have some strengths and weaknesses: for ASDF distribution scales well, but the long-term reliability of any individual package is only as good as the individual author wants to make it. For a CPAN-like architecture, there needs to be a more formal centralized structure, but if done right that structure is probably more stable long term. Anyway, this question is sort of outside the topic I was trying to drive, but your comment caught my eye.

This is a follow-up to the questions on CLISP and the GPL. I shot an email to Bruno Haible and asked him to clarify the CLISP license. He said that calling a function other than the public functions in the packages specifically mentioned in the CLISP COPYRIGHT file (COMMON-LISP, COMMON-LISP-USER, KEYWORD, CLOS, GRAY, and EXT) forces your code to be released under GPL. I specifically asked about the FFI and SOCKETS packages and he said any use of those packages forces your code to be GPL. Ergo, I stand by my original statement that CLISP all but forces your code to be GPL. If you write nothing other than code which calls ANSI or near-ANSI functions (e.g. GRAY streams), your code doesn't have to be GPL.

IMO, living without FFI or SOCKETS gets pretty uninteresting fairly quickly. For instance, imagine if GCC said that any program which called anything other than the standard C library had to be released under GPL. We wouldn't have nearly the adoption and innovation of GCC that we do today. I'm all for GPL as one of the license options and I think it's a great choice for CLISP itself, but the requirements on user-written code make it less desirable for me to use as my standard Lisp implementation.

Of course, if your program is GPL anyway, then CLISP is a fine choice. While it doesn't have a native-code compiler, the implementation is quite good overall and it's one of the most cross-platform Common Lisp implementations around.

Wasn't there a reaction from the Python community when Reddit arrived on the scene, holding a Lisp flag up high? And after the initial excitement upon Reddit's arrival faded, there has been much talk about community building for these types of services as an area to focus on, perhaps in response to perceived flat growth. You tell me. A little controversy such as a switch to Python can therefore be considered a publicity stunt. Conveniently, the switch reinfoces Paul Grahams point that Lisp is particularly well suited for startups, as the language allows you to stay flexible and rapidly develop into unknown areas. As you may know, PG's web service years later after being sold to Yahoo was eventually rewritten in another language too.

I remember working on a LISP interperter in college circa (1987) written in assembly of all things!

This last project of the semester was so relatively sraight forward that I ended up absolutely hating my compiler class that I had to take the next year (that project took the entire semester to complete).

When I look at the way industry has gone I can't help but shake my head in disgust. Over all, I am quite happy that I eventually decided to do something other than program computer for a living; it's obvious to me that if I had, I would hate my job, my coworkers and even the people who made my tools.

I remember working on a LISP interperter in college circa (1987) written in assembly of all things!

This last project of the semester was so relatively sraight forward that I ended up absolutely hating my compiler class that I had to take the next year (that project took the entire semester to complete).

When I look at the way industry has gone I can't help but shake my head in disgust. Over all, I am quite happy that I eventually decided to do something other than program computer for a living; it's obvious to me that if I had, I would hate my job, my coworkers and even the people who made my tools.

It would seem that you should really consider your last point about CLISP & GPL distribution carefully. The CLISP site and others have commented that distributing software without requiring GPL is possible. Perhaps there really are ways to do this? With the massive amount of work it must be to port an existing lisp to another platform, clarifying this point certainly seems like the quickest way to your goal.

I think the switch to Python is ok. It is their site and they know the guy doing their web framework.

But what I think is not so fortunate is that poo throwning and clumsy wordings. Was it really necessary to use such a strong word as "psychosis"? That was more fuel on the fire.

Me, I like the Lisp community. I think they are a bunch of nice people mostly. If not perhaps a bit impatient about things that always get talked about by newcomers to the irc channel and usenet group.

So while obviously I see the advantages of home-grown LISP network libraries---they'll fit in better with the language paradigm and the capabilities of the language---writing an API from scratch that will do everything you want, be cross-platform, and not change is a long and arduous task. This simply does not fit with the desires for "networking now!".

I would implement something like Perl's XS (but hopefully leaving behind some of its problems...) this would allow applications to immediately use stdlib functions for these things, which although not 100% cross-platform, certainly are very close. Then you'd have A: a list of import functions in the beginning of the app and B: a bunch of calls to the imported functions in the rest of the app.

In the near term, this would give you a cross-platform working set of functions all in C's stdlib. In the long term, you could work on developing native socket functions etc. And older applications would not break with the introduction of new, intelligently designed native functions since the import stuff would go on working merrily as before. Further, a reimplementation of the import stuff (in case something was missing in the first implementation) would only require the rewrite of an application's import functions.

This would give cross-platform compatibility, code stability, a whole huge set of libraries, etc. I'm new to this conversation, so good suggestion or am I missing something here? Wouldn't this fit the laundry list of things LISP lacked for reddit?

Hello Dave, Nice article, but I have a question about 'forces your code' to be GPL. From my reading and understanding of the discussion about this on comp.lang.lisp and clisp mailing lists, the clisp GPL forces the clisp binaries to be GPL'ed.

If one only distributes the lisp source code, they can be under any license. Is my interpretation correct, what do you think?

I'm just a Lisp spectator, but why not use ABCL (Armed Bear Common Lisp), a Java-based Common Lisp implementation? That would seem to solve the networking and threading issues, and the result would be as portable as Java is (i.e., highly portable).

D. Robert said { I specifically asked about the FFI and SOCKETS packages and he said any use of those packages forces your code to be GPL. Ergo, I stand by my original statement that CLISP all but forces your code to be GPL. }

I was just wondering if someone were to package up the part containing the FFI or the Sockets part of their code, as say a GPL'd library, if they could then call make calls to that library in a program that they were writing and not "taint" there program... Seems like the best compromise.. as the library so made would follow the intentions of the FSF by being passed back and usable by all, but still any use of that library without modification to the library itself would or could be non-BPl. Any thoughts on that?Fairly-anonymous-gene

With respect to CLISP and the GPL:The CLISP requirements are entirely justified to my mind. If you want to write non-GPL code in CLISP, simply provide a full wrapper API to the CLISP internals. While you're at it, port that wrapper API to CMUCL, SBCL, whatever. The glue code is a GPL'd compatibility layer bringing together multiple Lisp environments. Which is one the the exact things that we need anyhow, and it enables creation of proprietary code in CLISP. Yay.

But who is actually using CLISP for proprietary code? Even on Windows? Exactly.

W.r.t. CLISP, GPL and FFI: I think this discussion is an excellent argument in favour of "standardization" of APIs, instead of settling on specific libraries (evolved next month).See how GRAY is specifically excluded from the CLISP GPL contagion: if there were a FFI "standard", CLISP would likely strive to adhere to it and then could also exclude it from the contagion clause.Note that GRAY streams are no official standard in any way. It's just a paper.