lichtblau's blog

or: How I got sucked back into SBCL hacking on Christmas.

: david@radon:~; ps -o pid,vsz,rss,comm -p `pidof sbcl`
 PID VSZ RSS COMMAND
 1019 570424 4424 sbcl
: david@radon:~; ps -o pid,vsz,rss,comm -p `pidof sbcl`
 PID VSZ RSS COMMAND
 1019 963320 404204 sbcl
: david@radon:~; ps -o pid,vsz,rss,comm -p `pidof sbcl`
 PID VSZ RSS COMMAND
 1019 3988328 825804 sbcl

More than a decade ago, Gilbert Baumann started writing the Closure web browser. It includes a great HTML parser, written all in Lisp.

Released today, Closure HTML is a stand-alone version of the parser.

It supports HTML 4, understands malformed HTML, and can (optionally) be used in conjunction with Closure XML and its data structures.

An easy way to get started with Closure HTML itself is with its LHTML builder, which represents HTML elements as simple lisp lists.

Together, the two parsers can be used to turn HTML into XHTML or vice versa, and in particular to parse HTML into DOM or STP. Even for users who only parse and work with XHTML internally, the new code can be useful to emit normal HTML 4 as the last step of processing.

http://repo.or.cz/ hosts a git repository for SBCL and makes it easy to publish your own fork of SBCL.

Here is a quick step-by-step guide for anyone planning to have his repository hosted there. (All of this will be painfully obvious to the git experts.)

• Register a user account. All you need is an SSH public key, no questions asked.
• Create the fork. Find the SBCL project and go to "fork". Enter a project name for the fork and an admin password.

Done. Now you have a fork, but you need to initialize it first.

• Go to your "Project Settings" page and add yourself as a user.

Otherwise you cannot push to your own project.

• Push into the fork. One way to do this is to clone the normal SBCL repository, then use

  git push --all ssh://yourusername@repo.or.cz/srv/git/sbcl/yourprojectname.git

Don't forget the --all, which instructs git to push all refs. Whatever a ref is, anyway.

It is unfinished, slow, buggy, unmaintained, in need of a rewrite -- and now you can hack it yourself!

Doesn't that sound exciting?  Of course it does.

Prerequisites. Only Linux/x86 is supported¹. You will need several hours of spare CPU time, about 1 GB of RAM, and lots of disk space. Compilation involves building SBCL and classpath first, so make sure to install all required dependencies first. Debian users can run

# apt-get install sbcl svn cvs wget jikes
# apt-get build-dep classpath

to do so.

Build script. Grab cloak using git [edit: needs git 1.5, no idea why]:

$ git clone http://www.lichteblau.com/git/cloakbuild.git 

and compile it using clbuild-like commands:

$ ./build update
$ ./build world

Usage. The bin directory contains scripts called java, javac (courtesy of ecj), javap, and javah that run Lisp with the right arguments.

$ ./bin/java -version
CLOAK Virtual Machine, running on SBCL 0.9.8.6 (Linux 2.6.22 X86)
Copyright (C) 2003-2007 David Lichteblau

Technically it is a precompiler, and to avoid unpleasant surprises at run time, you might want to run

./bin/precompile foo.jar

before starting anything non-trivial.

Finally, read cloak/TODO and start hacking.

What's new? Compared to the big binary tarball available previously, this one comes with sources only, has been updated for current SBCL, and for Classpath 0.91 (which is still ancient, but a little step forward). The scripts in bin/ are also new.

¹ No AMD 64 support yet. For now, use an x86 chroot instead.

As mentioned earlier, I set out to define an alternative to the W3C's Document Object Model, inspired heavily by Java's XOM, but made for Common Lisp.

The result is STP, a data structure for XML that is full-featured and uses CLOS, but is more natural than DOM and gets namespaces right. Its implementation cxml-stp is available as an add-on library for Closure XML.

(For most purposes, it should be preferable to other alternatives, but DOM fans -- in case there are any -- can be assured that DOM support in cxml will not go away either.)

Read more about STP in the tutorial.

Gary King is confused because [XSLT] seems so ridiculously verbose. Others have already suggested mad higher-order function tricks using XSLT 2.0.

My solution: Macros. If XSLT lacks an element to do what you want (creating a text node with a newline, in Gary's case), just invent the feature you need and send your XSLT stylesheet through another XSLT stylesheet to implement it.

BR

Say you have demo.xsl which wants to use

 <x:br/>

to emit a newline. (In this example, `x' is simply the namespace for our extensions.) Write an additional stylesheet macros.xsl and send the original demo.xsl through the macro stylesheet to generate the actual XSLT source code. A macro template for <x:br> would be as simple as:

 <xsl:template match="x:br">
 <_xsl:text><xsl:text>&#10;</xsl:text></_xsl:text>
 </xsl:template>

In the macro stylesheet, xsl is the namespace of the "macro definition" and _xsl is the namespace of the "macro expansion". (If you care about details, the trick is to use xsl:namespace-alias to make the XSLT processor believe they are different namespaces.)

DOTIMES

For a more interesting example of macro use, suppose we want to repeat our code count times. Doing this kind of iteration involves a recursive template call, which we want to hide. We will define a macro <x:dotimes> that can be used like this:

 <x:dotimes var="i" count="3">
 <xsl:value-of select="$i"/>
 </x:dotimes>

Our macro stylesheet replaces each use of <x:dotimes> with a template call, and adds a recursive template as a top-level element:

 <xsl:template match="xsl:stylesheet">
 <xsl:copy>
 <xsl:apply-templates select="@*|node()"/>
 <xsl:for-each select="//x:dotimes">
 <_xsl:template name="x:dotimes_{generate-id()}">
 ... recursive template definition here ...
 </_xsl:template>
 </xsl:for-each>
 </xsl:copy>
 </xsl:template>
 
 <xsl:template match="x:dotimes">
 <_xsl:call-template name="x:dotimes_{generate-id()}">
 ... parameters elided for brevity ...
 </_xsl:call-template>
 </xsl:template>

Download the full macros.xsl and demo.xsl to try the example. To run it with xsltproc, use the Makefile in the same directory.

XOM is a DOM alternative written in Java and for Java -- in contrast to DOM, which feels wrong in almost every language.

Key phrases:

• "Comatose lists"
• This is a cathedral, not a bazaar
• There's exactly one way to do it
• The Wrong Side of 80/20

Lots of good ideas waiting to be stolen. Stay tuned for a Common Lisp adaptation.