Today, we deployed the first version of Quickref, a new global documentation project for Common Lisp.

The purpose of Quickref is to provide a centralized collection of reference manuals for the whole Quicklisp world. This means around 1500 libraries, for a total of around 3000 ASDF systems. The reference manuals are generated by Declt, which is probably the most complete documentation system for Common Lisp currently available, and delivered in HTML (PDF versions could easily be made available as well).

A lot of things can still be improved, but I'm pretty satisfied with the result so far. 3000 ASDF systems is a hell of a test suite for Declt, and I'm happy to report that it passes on practically all of them. Only a couple of issues remain, not even due to Declt itself, and less than a dozen or so libraries still pose problems (mostly technical difficulties due to foreign dependencies).

Quickref was made by Antoine Martin, as part of an internship with me. Many thanks to him! We still have some cleanup and packaging to do, but we expect to open-source the infrastructure soon. I also want to thank Mark Evenson, Erik Huelsmann and the Common Lisp Foundation for hosting the project on common-lisp.net (it was only natural)!

Finally, let me restate this again (and again): reference manuals are not user manuals. They are... reference manuals. Although automatically generated, there are some things you can do, as a library author, to improve the output (this is an area of Declt which I intend to work on in the future). Please refer to the Declt user manual (notably section 3.2 Coding Style) for more information.

I was investigating a bug in Declt where some floating point numbers were printed with exponent markers (e.g.0.5f0 instead of just 0.5) in the Texinfo file, which broke the parsing of the file by Perl.

Eventually, I found out a double infringement of the robustness principle. First of all, Declt failed to comply with part 1 of the robustness principle: be lenient with the others. The Texinfo file generation routine should have been wrapped into a call to WITH-STANDARD-IO-SYNTAX and it wasn't. Always do that to be on the safe side. Lesson learnt.

This failure on my part, however, had the interesting consequence of exhibiting what I consider a serious infringement of part 2 of the robustness principle: be strict with yourself. It would have remained unnocited otherwise. The culprit here is not Declt. This time, it's the common-lisp-stat library. Problem: the simple fact of loading this library globally changes the value of *READ-DEFAULT-FLOAT-FORMAT* from SINGLE-FLOAT (the default) to DOUBLE-FLOAT. This is bad, and it can break your code in all sorts of nasty ways.

Explanation

*READ-DEFAULT-FLOAT-FORMAT* tells the reader how to read floats when no exponent marker is provided. By default, 0.5 will be read as a SINGLE-FLOAT. But this variable also influences the printer (out of a concern for printing readably I guess): when printing a float of a different format than the current default, then the appropriate exponent marker is also printed. So here is precisely what happened. Declt had been compiled with some floats (e.g.0.5) read as SINGLE-FLOATs. Later on, those floats were supposed to be printed aesthetically as such. But right after loading common-lisp-stat, the default format changed to DOUBLE-FLOAT and all of a sudden 0.5 started to be printed as 0.5f0.

Consequences

This is bad enough already, but consider that messing with the standard IO syntax globally like this can break your code in all other sorts of even nastier ways. Imagine for instance that common-lisp-stat had been loaded before Declt, and Declt needed to be recompiled. All of a sudden, Declt would be using double floats and the bug would be gone. That is, until the next start of the REPL, after which all floats would be printed like 0.5d0!

I'm happy to announce the release of Declt 2.3. Declt is my reference manual generator for Common Lisp libraries.

The improvements and bug fixes in the last two releases are the result of running Declt against the whole Quicklisp world (around 3000 ASDF systems for 1500 libraries).
See this post for more information.

New in this release:

Advertise file extensions in references.

Advertise the type of foreign definitions.

More robust display and indexing of, and with, lambda-lists.

Use UTF8 special characters to denote invisble ones.

More robust support for Texinfo brace escaping.

Handle modules sharing the same location.

Ensure output is done with standard IO syntax.

Fix potential duplication of some (non-lisp) files and document all static files.

Fix potential duplication of packages documentation.

From the 2.2 "Christopher Pike" release (not previously advertised):

Require a UTF-8 environment.

Understand ASDF's notion of inferred system, and also be more protective against ASDF extensions.

Support for improper lambda lists (e.g. destructuring ones).

Improve contact defaulting code.

Update support for SBCL's setf expanders introspection.

Accept ASDF system designators.

Various bug fixes in the areas of method combinations, accessor definition merging and setf expanders.

Declt 2.0.1 "Benjamin Sisko" is out. This is a bugfix release with one internal change (a special variable was not following the earmuffs convention) and one actual bugfix (the same Texinfo anchor was generated for symbols with the same name but in different packages).

I've just released version 2.0 of ASDF-FLV, my ASDF extension for supporting file-local variables (ala *PACKAGE*). The code hasn't changed, but as for my other libraries, the system and package names are now prefixed with net.didierverna. ASDF-FLV is also available on GitHub now.

The reason I'm doing this now is that at least two of my other libraries are going to use it in a mandatory way, either directly or indirectly (and in turn, that's because no implementation has bothered to implement CDR #9 yet ;-).

I was watching the discussion between Gilad Bracha and Matthias Felleisen on gradual typing this afternoon (it's available on YouTube). This was the last event at the STOP workshop, part of ECOOP 2015 in Prague. I couldn't attend it because I was somewhere else (Curry On) at the time. The discussion is interesting, but if you go all the way, in the last 10 minutes or so, you will notice that Matthias seems to be completely obsessed with what he calls the "Return of the SegFaults".

Basically, the point is the following. Mathias dislikes the optional types in Common Lisp because it's opening the door to unsafety. Initially, any dynamic language has a sound type system (all type errors are caught; at run-time, yes, but they are caught). As soon as you introduce an optional type system ala Lisp, the compiler implementors are "pushed" to use the provided type information for optimization, hence weakening the type system and breaking soundness. It's the "return of segfauts".

Of course, I agree with that, at least on the principle. Yes, Common Lisp's weak, optional type system is an antiquated thing. However, it seems to me that Matthias is forgetting two important things on the practical level:

by default in many implementations that I know, if not all of them, introducing type declarations doesn't actually break the soundness of the type system but leads to even more type checking. For example, (defun plus (a b) (+ a b)) works on every possible numerical value, but add (declare (type fixnum a b)) in there and it will suddenly stop working on anything else but integers. It's only if you require the compiler to optimize for speed at the expense of safety that you effectively weaken your type system.

In practice, the risk of re-introducing segfaults in your application may be mitigated by the interactive aspect of the development (TDD made easier, simultaneous write / compile / run / test / debug phases etc.).

So my impression is that Matthias is largely exaggerating the problem, but I'm not really qualified to tell. That's why I would like to know from you guys, working with Lisp in the industry, writing large applications, lots of type annotations, and (declaim (optimize (speed 3) (safety 0) (debug 0)) (EDIT: that's exaggerated of course, I really mean breaking safety for performance reasons): how much of a problem the "return of the segfaults" really is in practice ?

As a side note, this reminds me of the dynamic vs. lexical scoping debate. Many people were and still are strongly opinionated against dynamic scoping by default. Of course, I too, at least in principle. But how dangerous dynamic scoping really is in practice (EDIT: I'm not talking about expressiveness, e.g. closures, here. Only unsafety.)? What I can tell you is that in the 15 years I was actively maintaining the XEmacs codebase, I may have run into name clashes due to dynamic scoping... twice.

Declt 2.0 "Kathryn Janeway" is out. This release doesn't contain any change in functionality, yet deserves a major version upgrade since it contains 3 important changes: an infrastructure revamp (along the lines of what Clon endured not so long ago), a license switch from the GNU GPL to a BSD one, and finally a system / package name change. The prefix is now net.didierverna instead of com.dvlsoft. Do I need to apologize for this again? :-)

Reported to me by Craig Norvell: Haystax appears to have some interesting Lisp projects in the works. Combining Lisp, Prolog, RDF (AllegroGraph), and a BBN for insider threat detection solutions.

From their website:

Haystax's predictive models are continuously updated based on the prevailing threat environment making it highly suitable for both detection and continuous evaluation of threats. These unique models go beyond traditional web and business intelligence to enable organizations to achieve contextual real-time situational awareness by fusing all operationally relevant information - private, public, video and live feeds - into consolidated views to show patterns and identify threats that are usually buried in too much noise or not placed in proper context.

After 15 betas, I'm happy enough with the current state of Declt to finally make a 1.0 release.

A lot of things have changed since the previous version, sometimes in a backward-incompatible way. Many more items are documented (including, as you have recently seen, method combinations). In addition to the reference manual, generated by Declt itself, there is now a real user manual.

Declt is still SBCL-only, requires ASDF 3 and Texinfo 4 but generates code that is compatible with Texinfo 5. Also, beware, I've deleted the old repository and moved the project to GitHub. Below is a more precise description of what Declt currently does.

Declt (pronounce "dec'let") is a reference manual generator for
Common Lisp libraries. It works by loading an ASDF system and introspecting
its contents. The generated documentation contains the description for
the system itself and its components (modules and files), the packages
defined in that system and the definitions found in those packages.

Exported and internal definitions are listed separately. This allows the
reader to have a quick view on the library's public API. Within each
section, definitions are sorted lexicographically.

Reference manuals are generated in Texinfo format (compatible, but not
requiring Texinfo 5). From there it is possible to produce readable /
printable output in info, HTML, PDF, DVI and PostScript with tools such
as makeinfo, texi2dvi or texi2pdf.

The Declt reference manual is the primary example of documentation generated by Declt itself.

In the process of writing Declt, I had to deepen my knowledge of some Lisp corner cases, notably in the area of introspection. As you know, Lisp has in fact more than 2 namespaces. Sometimes, introspecting a potential symbol definition in one namespace is trivial. COMPILER-MACRO-FUNCTION is one example. The "functional" namespace is heterogeneous but you can still make your way out of macros, regular or generic functions very easily. In the same vein, distinguishing constants, special variables and symbol macros in the "variables" namespace is more complicated because there is no standard way to access that information, but with the help of some vendor-specific machinery (e.g. SB-INT:INFO), it's still doable.

At some point, I tackled the case of method combinations, and to my greatest surprise, found out that introspecting a method combination by name is not simple. In fact, it's not even doable. The reason is that method combinations don't have a namespace proper. Let me explain why.

You define a new method combination of some NAME with DEFINE-METHOD-COMBINATION, and you use it with the :METHOD-COMBINATION option to DEFGENERIC. But how do you introspect a NAME for a potential method combination definition? Well, you can't do exactly that. First, there is no standard way to do so. Next, let's look at what the MOP provides. One would expect something along the lines of FIND-CLASS...

There is indeed something called FIND-METHOD-COMBINATION, but either I'm missing something, or it is really, and I mean really badly designed. The arguments to this function are: a generic function meta-object, a method combination name and some method combination options. So if this function is supposed to be the equivalent of FIND-CLASS for method combinations, what in the hell are the 1st and 3rd arguments for? In fact, it's not supposed to do what its name suggests. According to the AMOP, p.191, the purpose of this function is to "determine the method combination object used by a generic function". In practice however (at least in SBCL), it's not doing that either (see below), and anyway, determining the method combination object used by a generic function is better achieved by using the GENERIC-FUNCTION-METHOD-COMBINATION accessor.

So this protocol doesn't seem to make any sense. In order to understand what to make of all this, I looked at how SBCL handles method combinations (I don't know what other vendors do) and here is what I found. When you call DEFINE-METHOD-COMBINATION for some NAME, SBCL creates a new method for FIND-METHOD-COMBINATION, eql-specialized on that NAME. This method is encapsulated in a closure containing the method combination's definition, ignores its first argument (the generic function meta-object; that's why I said that it's not really doing what the AMOP says it does), recreates and returns a new method combination object on the fly every time it is called.

This has several consequences. First, the notion of "method combination named NAME" doesn't really make sense. Method combinations don't have a proper namespace. Every time you create a new generic function, the method combination essentially becomes local to that function. Redefining a method combination has no effect on existing generic functions using a method combination of the same NAME. To put it differently, you can end up with several generic functions using different yet equally named method combinations.

In Declt, I'm now assuming that programmers behave and only define method combinations once. Which brings us to the second consequence. With that assumption in mind, the "proper" way to introspect a NAME for a method combination definition (at least in SBCL) is to first look for the existence of a FIND-METHOD-COMBINATION method eql-specialized on that NAME, and then call it to retrieve the actual definition.

I haven't thought about it a lot yet, but it would be interesting to investigate the idea of cleaning this up a bit. I mean, establishing a real global namespace for method combinations, creating an alternate FIND-METHOD-COMBINATION protocol more along the lines of FIND-CLASS. And then, it could also be interesting to investigate on the potential applications of allowing method combination redefinition to affect live generic functions, just like class redefinition affects live instances...

This morning, I came up with a nice little trick which made my day. Perhaps this is already well known or even idiomatic, but in case it is not, it goes like this.

Sometimes, I have to read a whole Lisp file containing more than just one toplevel form and get the contents as an unprocessed list of expressions. This happens for instance when loading a DSL program that needs post-treatment.

But for some reason, this has always felt clunky to me. I mean, instead of ogling the family jewels of this poor shy file, I should be able to read it as a whole (hint: read-delimited-list) and preserve its intimacy.

And then I realized that the only thing that prevents me from using read-delimited-list is the lack of an ending parenthesis. But as always, Common Lisp immediately comes to the rescue, this time with its rich streams API. The idea is that you can create a string stream that just contains the closing parenthesis, and concatenate this stream at the end of the file one. I can now do something like this instead:

I've just released a new version of Declt, my reference manual generator for ASDF systems.

This release containts some improvements based on Sabra Crolleton's feedback. The most notable improvements are support for two new license types (MIT and LGPL), a new :DECLT-NOTICE keyword argument that gives you control on the "automatically generated by Declt" notice that appears in the reference manuals, and a bug fix (missing support for empty lists in lambda-lists).

Programming languages are tools. Just like hammers. There's nothing personal about them. They just help you build stuff. Yet, there are many religious wars about programming languages out there. Wars which despite being all about science, are much more related to emotions, beliefs and personal aggressions than about objective arguments.

This is funny because do you know of any religious wars about hammers? So what's the difference?

Here's a recent personal example. A guy explaining how the static typing of Common Lisp works (type declarations) and what kind of performance-oriented optimization can be achieved with it (C-like, static but weak typing in SBCL for instance). And then, there's inevitably the troll (whom I know knows better) in the audience who goes:

So, yeah, what you're doing is just C code, and you have to type manually, and it's ugly. So if it's just for writing C code, I don't see the point in using another language.

Hmmm. Let's see. So, yes indeed, static typing in Common Lisp is ugly. And yes, it's not even strong typing. And yes, it would be nicer to have run-time hotspot detection and automatic type-dependant optimization like what's found in some other languages or virtual machines, rather than having to do it by hand (BTW, that would make for a nice Ph.D. I think). But what does that really tell you? That no language is perfect? Wow, thank you very much, that's new. For as much as I think that Lisp The Idea™ is perfect, I don't think anyone ever pretended that Common Lisp (or any Lisp for that matter) was. But is that a reason for not using it at all and sticking to C? Any sane computer scientist knows that the choice of a language doesn't boil down to only one parameter. Any computer scientist who tells otherwise is a troll.

In spite of all its defects, I still have a gazillon reasons to prefer Common Lisp over C, C++, or any language that I know of currently (and this may very well change in the future). The flexibility of lambda lists, the macros, the MOP or more generally its structural and behavioral reflexivity. Its run-time compilation, debugging, introspection and intersession capabilities. These are just a few examples. Still, I don't deny anyone the right to prefer another language for whatever purpose and whatever reasons they may feel legitimate.

So, I normally just ignore those purposedly trollesque and completely idiotic remarks. Yet, sometimes like yesterday, I snap. It gets on my nerves and I become all upset and angry. Why? I never get angry when someone tells me that my hammer is a piece of crap (it's not). I do enough Aikido to know how to control my temper, but for some reason, it doesn't always work when it comes to programming languages. So what is it about them that in spite of all your efforts, you can't help from getting personally and emotionally involved once in a while?

I think the answer becomes apparent when you consider the artistic aspect in programming. When an artist creates a piece (music, theater, dance, painting, architecture, whatever you like), (s)he exposes a very intimate part of himself through his creation. The art "is" the artist, and the artist "is" his art. In doing so, he puts himself in danger. It's like yelling out to the whole world Hey, look, I'm vulnerable right here!!. It's a well know fact that many artists are very fragile, in the sense that they suffer from their creation not being liked. Because a piece of art is intrinsically a piece of the artist himself, when you say I don't like this piece of art, you're actually saying I don't like the artist. Then, it's up to artist to handle the fact of not being liked.

And that's the whole problem, which, as a musician, I know all too well. Where does the artistic fiber come from? It's an urge to express yourself. To express something that you can't express in any other way. A very deep and perpetual wound of some sort, a feeling of not really belonging. More importantly, it's a calling. Sometimes, the simple fact of creating is enough to heal you a bit, but more often, you create in order for your creation to be seen or heard. So yes, it's a calling to the Others. You expect them to answer your call by telling you that they like you (your art, but that is the same). Artists often have this urge to be liked by the Others. So when you dislike some artwork, you're also not liking the artist himself (the part of him that lives in his creation) and you're actually giving him the exact opposite of what he was looking for. And that hurts.

Back to programming languages. Why do we get all emotional about them, and not about hammers? The answer is in fact quite simple. Look at an architectural masterpiece. Do you see the hammer that was used to build it? Now look at a software masterpiece. Do you see the language that was used to write it? That's the crucial difference. You cannot decouple the language from the software, even once it has been written (the art is not in the executable; it's in the source code). The language itself will always be here for you to contemplate.

All in all, I think that's why there will always be language wars. Languages are not just tools, actually. They're not just like hammers. As soon as you care about the code you write, your software becomes artwork, you become an artist, and you start to be personally and emotionally involved. Your software becomes part of you. And contrary to the hammer, your sticky programming language, being intrinsically bound to the artwork, also becomes part of you. That's when the battle for objectivity is lost. By criticizing the language, the troll also criticizes your artwork, and in doing so, he tells you that he doesn't like you. That may hurt.

It's good to consider programming as art. Unfortunately, this also means that there will always be language wars.

I've just released a new version of Declt, my reference manual generator for ASDF systems.
This release includes some uninteresting internals update, plus an important bug fix: there were two calls to FIND-METHOD missing an ERRORP flag set to nil, leading to Declt throwing an error where it shouldn't have.

The most important change in this release is the support for LispWorks, which brings the number of supported implementations to 8. One left to go, and I may eventually switch to RC status. Thanks to Martin Simmons for providing a fully functionnal version of LW 6.1. As for CLISP and Allegro, there is an optional dependency on CFFI for LispWorks.

Two backward incompatible changes that may affect you:

Variables renamings: *current-context* has been renamed *context*, and *default-synopsis* has been renamed *synopsis*. This should remain transparent unless you're using Clon in a somewhat advanced way.

clon:exit has been upgraded to SBCL's new quitting protocol. If you use this function (or if you want to compile the demo programs), please upgrade to SBCL 1.0.57.

Finally, support for terminal autodetection and stream handling in general has been improved for all implementations.

To my greatest disappointment, I discovered today that it is not possible to replace Lisp parenthesis by, say, ... curly braces. What a shame. Hell, it's not even possible to freely mix the two. Very naively, I had expected that:

would suffice, but no. All implementations that I have tested seem to agree on this, although the error messages may differ. For instance, trying to evaluate {and} gives you an "unmatched close parenthesis error" except for CMU-CL which chooses to ignore it, but then report an end-of-file error. The unmatched close parenthesis, of course, is the closing curly brace! So what is going on here?

When an opening curly brace is read, the original left paren macro function is called. In SBCL for instance, this is SB-IMPL::READ-LIST, which looks for a hardwired right paren on the stream. Yuck. It doesn't find one, but it finds my closing brace which triggers the "standalone" right paren behavior (spurious paren alert). In passing, it also surprised me that SB-IMPL::READ-LIST is not implemented in terms of READ-DELIMITED-LIST.

EDIT: as mentioned in several comments, we could use read-delimited-list to look for a closing curly brace, but even this won't work completely. The problem is with dotted lists (see Pascal's comment). SBCL hard-wires #\) in its dotted lists parsing procedures.

So it appears that dispatching macro characters are only shaky. What we miss is a true concept of syntactic categories (Common Lisp character syntax types are close, but not quite there yet). In fact, TeX, with its notion of catcodes (category codes), seems to be the only language that gets this right. Ideally, any character with associated status LIST TERMINATOR should do as good as a right paren (the problem is only with closing, not opening).

Instead of hard-wiring the right paren in the Lisp parser, a quick workaround would be to check whether the next character on the stream is a dispatching one, and in such a case, whether its macro function is the one originally associated with the right paren. If so, it should then simply stand as a list terminator. This is actually an interesting idea I think: could the built-in macro functions become equivalent to actual category codes, and could we completely remove hard-wired characters in Lisp parsers?

Anyway, this whole story is a true scandal because it ruined an otherwise cool live demo of mine. So much for syntax extensibility. I will immediately complain to the concerned authorities.

I'm happy to announce that my contribution to TUG 2012, the next TeX Users Group International conference, has been accepted. Please find the title and abstract below.

Star TeX, the Next Generation

In 2010, I asked Donald Knuth why he chose to design and implement TeX as a macro-expansion system (as opposed to more traditional procedure calls). His answer was that:

he wanted something relatively simple for his secretary who was not a computer scientist,

the very limited computing resources at that time practically mandated the use of something much lighter than a true programming language.

The first part of the answer left me with a slight feeling of skepticism. It remains to be seen that TeX is simple to use, and when or where it is, its underlying implementation has hardly anything to do with it.

The second part of the answer, on the other hand, was both very convincing and arguably now obsolete as well. Time has passed and the situation today is very different from what it was 50 years ago. The available computing power has grown exponentially, and so has our overall skills in language design and implementation.

Several ideas on how to modernize TeX already exist. Some have been actually implemented. In this talk, I will present mine. Interestingly enough, it seems to me that modernizing TeX can start with grounding it in an old yet very modern programming language: Common Lisp. I will present the key features that make this language particularly well suited to the task, emphasizing on points such as extensibility, scriptability and multi-paradigm programming. The presentation will include reflections about the software engineering aspects (internals), as well as about the surface layer of TeX itself. Most notably, I will explore the possibilities of providing a more consistent syntax to the TeX API, while maintaining backward compatibility with the existing code base.