lichtblau's blog

2011-06-19

Several years ago, I started working on the XML parser that Gilbert
Baumann had written as part of his web browser called
"Closure".
Since then, "Closure XML"
(or cxml for short)
has developed into a set of little libraries, one main goal being
completeness and correctness with regard to the various standards they
are following.

And standards abound in XML land, which is nice for implementors
(thanks to the good test suites!) and nice for users (because the
specs partially serve as documentation, and make it easy to transition
between different languages implementing them). But I've always tried
to release cxml with enough documentation to get users started for all
the parts that are implementation-specific. And not all areas are
covered by standards: Of course, the document format itself is
specified strictly; the same goes for XPath, XSLT, schemas, etc.

But little is standardized in terms of API support, and that sort of
choice is generally good: After all, a Lisp XML parser should fit into
the Lisp world and not mimick (say) JavaScript too much. But many
good ideas can be borrowed from other languages. Examples inspired by
Java are
STP, motivated
heavily by XOM (and tweaked for added
lispiness) -- and SAX:

SAX is a
classic Java API. It defines a protocol of methods that get called by
an XML parser, and each method call signifies an event
(e.g. that the parser saw a XML tag). In cxml, SAX is one of two
fundamental APIs offered (the other being a StAX-like pull-based
interface), and it's essential to its inner workings. Yet I had never
bothered to document it fully. For one thing, everyone seemed to know
SAX from Java anyway. It's also hidden from view for most users. And
ultimately, it's just a list of generic functions, right?

Technically it is just that, and yet it's central to communication
between cxml's libraries, and it makes parsing and serialization in
cxml modular and reusable.
Hence some users had long
suggested to me that I should explain SAX in full.

2011-06-18

SBCL moved to git last week, and with the main repository hosted on sf.net, several users have asked for an SBCL mirror on github.

Automatically synced mirrors are available on github and gitorious (where SBCL branches had, of course, also been uploaded by contributors long before the official switch to git), and it will now be easier than ever to make your own repository and branches.

2010-04-07

For context, see Nathan's blog.
[Edit: I've just learned on IRC from Maciej Katafiasz that iterate also has a feature built in which renders this hack unnecessary. Even more iterate awesomeness! I'll leave this post up though, because I still think it' s pretty cute that SBCL optimizes this case so nicely.]

Note that the COLLECT in the outer loop is ITERATE's collecting feature. The trick is to use it in a FLET before starting the inner loop, thereby making it available further down.
Declarations added only to get a nicer disassembly.

2009-08-12

For several years, the best (and only) option for portable access to relational databases from Common Lisp has been the CLSQL library, a free reimplementation of LispWorks' CommonSQL interface with support for many backend databases, a lispy SQL abstraction as well as simple object-relational mapping support.

Recently, several alternatives have emerged: Postmodern provides particularly good support tailored to PostgreSQL only, other libraries abstract away from relational databases entirely, and finally, there are two new stars on the library horizon: CL-RDBMS and CL-PEREC.

Having used CLSQL for several years, I am currently investigating a switch to CL-RDBMS and PEREC for a code base of non-trivial size, aiming to improve scalability, portability across databases, and last not least readability of the code base.

And I am very happy with everything I have found so far:

In the CL-RDBMS/PEREC ecosystem, work is split into two layers.

CL-RDBMS forms the lower layer. It abstracts over database access libraries (currently with support for PostgreSQL, Oracle, and SQLite) and defines an optional, Lispy, extensible syntax for SQL for data definition and query statements.

The optional PEREC layer is the one which really shines. It is an object-relational mapping (ORM) seamlessly integrating CLOS classes with database tables.

Highlights:

Sensible caching model: Configurable caching of slots within a transaction, but not between transactions.

Object identity preserved (within a transaction).

Protection against accidental cross-transaction access.

Lispy SELECT statements using code that looks like ordinary CLOS use, but which usually compiles down to efficient database-side SQL statements.

Overall, an architecture that is ideal for robust, scalable setups with multiple threads and processes without risk of the "oversold concert tickets" situation reported for a popular Lisp web site based on older ORM software a few years ago...

The only fly in the ointment for me was the lack of introductory material. PEREC comes with an extensive test suite and nice examples, but I would have wished for a few more hints to get me started.

Thanks to fellow blogger pinterface, my wish has been granted. He has written a blog series about PEREC with many details about getting set up, and including various customization tips.

2009-04-14

His approach is extraordinarily simple and uses a small Lisp program to parse clbuild's dependencies file and write out a .dot file. Of course, using Graphviz is a well-known approach for this kind of dependency diagram. Bill Clementson mentioned Richard Newman's graphs a few years ago.

But Ben's graph is a fresh take on this because of the way it uses clbuild's recorded dependency information, which shows dependencies of projects rather than dependencies of systems. What's the difference between projects and systems? Lisp projects often contain multiple .asd files.

Before thinking about the details of that distinction, let's look at the picture. Here's his graph, showing Hunchentoot, Drakma, cxml, and others:
Observations:

Yes, there's a lot of Babel in there. (It deserves it!)

The project dependency graph is not transitive. Note how CXML depends on Babel, which in turn needs Stefil. But CXML itself does not depend on Stefil. How can that be? It's because the systemCXML depends only on the systembabel. And that's not where the use of Stefil comes from. It's actually just babel-tests that needs Stefil, and that isn't needed when you're compiling CXML.

Where there are indirect dependencies, the graph shows them in full. Note that CL+SSL has an arrow to babel here, although it doesn't use it directly. This indirect dependency is due to its use of CFFI.

While probably not suitable for all dependency graphs, I think that this explicit display of indirect dependencies gives very nice results in this case, because it highlights commonly used libraries like Babel more than an ordinary graph would have done.

You can download his code at github. Note that you don't even have to download the projects that you want to compute a graph for, if you replace the call to directory with any other test of your choice.

2009-04-04

Newly
released: Xuriella
XSLT, the first implementation of XSLT written entirely in Common
Lisp.

Based
on Plexippus
XPath, Xuriella is an implementation of XSLT 1.0, written by Ivan
Shvedunov and me.

Xuriella is quite complete and correct -- we run the official
testsuite, with more than 95% of tests currently passing.

Extensions

One advantage of a pure-Lisp implementation is that extension
elements (as well as XPath extensions) can be defined easily.

That's a huge plus because XSLT itself is a very specialized
programming language -- it excels at XML/HTML generation and
transformation only. Being able to write custom extensions in Lisp
helps with any non-XML-ish parts of the task which XSLT itself might
not handle conveniently.

Documentation

If you just want to try applying stylesheets, there are only two
functions you need to know about: parse-stylesheet
and apply-stylesheet.

For details about these functions (and all others, including those for
extensions), refer to the
API documentation.

Example

The example
uses Hunchentoot and Xuriella with XSLT as a template language in a
simple directory listing request.

2008-12-07

The first version of Ivan
Shvedunov's Plexippus
XPath has been released, an implementation of XPath 1.0, written
entirely in Common Lisp.

Plexippus has been available from darcs for a few months now, and we
have had some early users (thanks for testing the code at that early
stage), but this is the first tarball release. New tarballs for cxml
and its other add-on projects are also available now, so that you don't
have to juggle with git, darcs, and CVS anymore.

Beta version: SBCL-only

Plexippus is well-tested on SBCL, but has not yet been ported to any
other Lisp implementation. That's why this release is declared as a
beta version.

(Ports to other Lisps mainly involve support code for non-trapping
floating-point arithmetic. Patches are welcome.)

The XPath protocol

For extensibility, Plexippus defines a protocol of generic functions
to implement the XPath data model, so any object model for XML can be
queried using Plexippus if it implements these generic functions.
Out-of-the-box, there is support
for cxml-STP (a DOM
alternative) and cxml's DOM.