Fight the System

Installing and Using Common Lisp Libraries

Henrik Motakef

Redistribution and use in source (XML Docbook) and
'compiled' forms (SGML, HTML, PDF, PostScript, RTF and so forth)
with or without modification, are permitted provided that the
following conditions are met:

Redistributions of source code (XML Docbook) must
retain the above copyright notice, this list of conditions
and the following disclaimer as the first lines of this file
unmodified.

Redistributions in compiled form (transformed to other
DTDs, converted to PDF, PostScript, RTF and other formats)
must reproduce the above copyright notice, this list of
conditions and the following disclaimer in the documentation
and/or other materials provided with the
distribution.

Important

THIS DOCUMENTATION IS PROVIDED "AS IS" AND ANY
EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
DOCUMENTATION, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH
DAMAGE.

Understanding Systems

While there are quite a few libraries for Common Lisp
downloadable on the Web, Newbies often have a hard time figuring
out how to use them. Neither clicking on
setup.exe nor configure; make;
make install usually help. Instead, Lispers use
system definitions that are kind of like
a mixture of Makefiles and pkg-config
tools (only different), together with a library called
mk:defsystem to handle them.

In Lispspeak, a system is a
collection of files that together make up a library, an
application or something similar. Usually, most if not all files
are Lisp source files, but sometimes other file types are
included as well, for example data and configuration files, C
sources or documentation.

Some of the files may depend on others – for example
because they use macros defined in some other file – so
they have to be loaded/compiled in the right order to work
properly. Doing so by hand, using LOAD and
COMPILE-FILE, may by possible, but is surely not fun
– especially when you have a system containing dozens of
files, or want to use several systems at once.

In addition to the problem of compiling systems properly,
there also has to be a convenient way to find them and load them
into a Lisp image. The ANSI Common Lisp standard includes
rudimentary support for that, by means of the the functions
PROVIDE and REQUIRE, that manage a collection of
so-called "modules". Unfortunately, it doesn't really say much
about how a Common Lisp implementation should actually handle
modules (for example where it should look for installed
modules), which makes it awkward to use them in practice.

While there are alternatives[1] , mk:defsystem is the most common way to
manage systems today: Most libraries come with a
mk:defsystem-compatible system definition, and mk:defsystem itself
works with most popular (and even some not-so-popular) Lisp
implementations.

Setting up defsystem

mk:defsystem is maintained as a part of CLOCC,
the Common Lisp Open Code Collection, which happens to include
lots of other useful CL code, too. You can find CLOCC at http://clocc.sourceforge.net,
or download mk:defsystem only from its Sourceforge project
site.

For obvious reasons, you cannot use the convenience
mk:defsystem provides to load itself, you have to make your Lisp
system know about it the hard way. Fortunately (and for exactly
that reason), it is only a single source file
(defsystem.lisp), so that you can easily
LOAD it, like this:

(load "/path/to/your/defsystem.lisp")

Since you are likely to use it in almost every
Lisp session, it is best to do so in your Lisp startup
file. The actual file name of your init file depends on the Lisp
implementation and operating system you use. For example, for
CLISP on Unix, it
would be ~/.clisprc.lisp, CMUCL uses
~/.cmucl-init.lisp. See the documentation
of your Lisp of choice when you don't know where to put startup
code.

You can also build a new Lisp image that contains
mk:defsystem, so you don't have to load it at all. Again, your
Lisps documentation will tell you how to do so.

mk:defsystem has the nice feature that it can find system
definitions on its own; you only have to give it the name of a
system to load or compile, not the physical location where its
definition or source files live. This depends on two things:
First, the file containing the system definition should be
called foo.system (where "foo" is the name
of the system). That's not usually something you have to care
about, since library authors tend to know and respect this
convention. Second, mk:defsystem has to know where to look for
these files. It maintains a list of directories called the
central registry (in a variable called
mk:*central-registry*) where it expects
system definitions to be.

Generally, it's a good idea to keep all your Lisp libs in a
central directory, for example
~/cl-library/. Additionally, quite a few
libraries by default expect their sources to live on a logical
host[2]
called cl-library:, so we set this one
up in our lisp init file, too.

Installing your first system

As an example, we'll install the CL-PPCRE regular
expression library using defsystem.

The first step is obviously to download the sources from
the CL-PPCRE
homepage. Unpack them in your
~/cl-library directory. This should create
a subdirectory
~/cl-library/cl-ppcre-${VERSION} containing
both Lisp source files and system definitions.

CL-PPCREs system definition is written in a way that it
will find the source files based on its truename, that is,
the "real" name of the system definition file, even when loaded
via a symbolic link. So the easiest way to make everything work
is creating a symbolic link to
cl-ppcre.system in a directory in your
mk:*central-registry*, for example like
this:

(In case you wonder, mk:oos is just a
short nickname of the function
mk:operate-on-system.)

Once you have compiled CL-PPCRE, you can load it in
subsequent Lisp sessions by running

(mk:oos :cl-ppcre :load)

Finding the Sources

Not all libraries are as nice as CL-PPCRE, using the
truename of the system definition file to find the directory
containing the sources. As stated above, others expect a logical
host to set up properly, still others simply hard-code the
paths that happen to work on the developers machine.