1 Introduction

Cheat-JS - macros for JavaScript. Kinda.

This document assumes you already know Common Lisp and JavaScript.

About Cheat-JS

Lisp macros are powerful and easy to implement because Lisp programs
are made of s-expressions.

Lisp-style macros are difficult to add to other languages because most
languages have very non-uniform syntax compared to Lisp. Source
transformations (and most importantly macros) would be easier in, say,
JavaScript, if it were possible to convert the JavaScript code into
s-expressions, transform it, and convert it back into JavaScript code.

Turns out that we can transform JavaScript code into an AST made of
s-expressions using
parse-js. Converting back
into JavaScript code can be done with
cl-uglify-js (ironically,
cl-uglify-js:ast-gen-code is a capable pretty printer). All that
remains to be done to have macros (well, defined in another language)
is define transformations to be applied on the output of
parse-js. This is what Cheat-JS does: get the parse-js AST, apply
the transformations, convert back to JavaScript code.

The idea is rather obvious - the main reason Cheat-JS exists is that I
could not find something similar on the net. There are probably many
people who privately do similar things with tools like parse-js and
the pretty-printer part of cl-uglify-js - that, or my Google skills
failed me :)

Important Note

Cheat-JS includes a modified version of parse-js, written by Marijn
Haverbeke. This is necessary because I (Miron Brezuleanu) needed to
modify parse-js a little. The license of parse-js is in the
LICENSE-parse-js.txt file. The modified files from parse-js
included in Cheat-JS are parse.lisp, tokenize.lisp and
util.lisp. The modifications were permitted by the parse-js
license. This is not an official copy of parse-js and is not
supported by Marijn Haverbeke. If the modified parsing code in
Cheat-JS breaks, it's exclusively my fault - I messed up the code.

Cheat-JS also uses cl-uglify-js unmodified, via
Quicklisp. These two libraries do most of
the work, Cheat-JS is mostly 'glue code'.

BIG WARNING

I haven't used Cheat-JS on any large projects. I don't have enough
imagination to compensate for this lack of experience, so it may have
a lot of problems I haven't thought about. Right now it's just a proof
of concept.

This assumes that we have defined a @defclass macro which does the
above expansion - we'll define two such macros in this document.

One of the parse-js modifications necessary for this to work is
allow @ as a character in identifiers (the Cheat-JS recommended
convention for naming macros is @ followed by the macro
name). Currently macro names can't be nested
(i.e. some.namespace.@iife is not valid syntax).

I also had to modify parse-js to convince it to parse macro
invocations that look like function calls, but have a list of
statements instead of a list of parameters (see the invocation of
@iife above).

This macro is similar to
alexandria:when-let. @whenLet
has the most complicated interface possible for a Cheat-JS macro: its
argument list has both expressions (testResult, someTest()) and
statements (the console.log call). When invoking such macros,
separate the expressions and the statements with a semicolon, as in
the example above.

It is of course possible to define the anaphoric version of
@whenLet, @awhen (from
On Lisp, page 190).

The guide on how to write Cheat-JS macros (below in this document) is
based on defining @defclass (and even a safer version of
@defclass), @iife, @whenLet and @awhen.

Getting started

You can get Cheat-JS at http://github.com/mbrezu/cheat-js. It's
probably best to git clone it inside the local-projects directory
of your Quicklisp install, so you can load it with (ql:quickload :cheat-js) in your REPL.

Note: I've only tested it with SBCL, so I recommend you use SBCL
too. It should work with other CL implementations, though (all the
code required is standard CL).

Running the tests with:

(cheat-js:run-tests)

gives some confidence that things are not obviously broken (they are
most likely broken, but in ways that are subtle enough to fool the
tests).

The next section will provide you with some pointers on how to define
your Cheat-JS macros. If you run into problems, look at the
tests.lisp file for example code - the code there matches the text
in the next section.

how the macro invocation call looks like (what you want to write in
the JavaScript source code); this is the macro's "API"; it is
JavaScript code (or almost);

how the macro expansion looks like in JavaScript; this is the
macro's "result";

how to transform the AST of the invocation into the AST of the
expansion; this is the macro's "implementation", written in Common
Lisp.

Let's define @defclass.

Defining @defclass

We need to see how the macro invocation looks like in JavaScript:

var Person = @defclass(name, shoeSize)

We can tell that the macro is an 'args only' macro (i.e. the
invocation looks like a normal JavaScript function invocation, it does
not contains statements). We can inform Cheat-JS about this:

> (cheat-js:register-args-macro "@defclass")

We also want to see the AST for the invocation (we use Cheat-JS's
parsing function because the above snippet is not parsable by
parse-js without tweaks; in particular, the parsing won't work as
expected if we don't call register-args-macro as above, so don't
skip that step):

The part that starts with :MACRO-CALL is the interesting part; this
is the AST representation of our macro invocation; this is what we
need to transform into the expansion (don't worry about the apparently
missing ( in front of :MACRO-CALL above, it's because the list
following :VAR is made of conses, not lists).

We only need the macro arguments to perform the expansion. Since we
told Cheat-JS this is an 'args only' macro, it knows we're only
interested in the arguments (not the entire :MACRO-CALL tree), so
that's what it will pass to our expander function:

The parameter args contains the list of arguments extracted from
(:ARGS... above (in our case ((:NAME "name") (:NAME "shoeSize"))). The value returned by the function should be the
expansion shown above (s-expression starting with (:FUNCTION...).

This is a 'body' macro - its only argument is a list of JavaScript
statements. Let's tell Cheat-JS:

> (cheat-js:register-body-macro "@iife")

We can now ask the parser for the invocation AST. We'll work with a
simplified invocation, though - the example above will generate a
large AST, and we can just as well manage with a smaller one. It's
also better if our invocation has more than one statement, so let's
try this:

Defining a safer @defclass

This code has a well-known problem. To create a Person object, one
should use a call like new Person('John', 42);. If we forget the
new keyword, we are in trouble, because this inside the function
no longer refers to a newly created object, but to window, the
global object.

The :MACRO-CALL node has subnodes for both :ARGS and :BODY, they
will be both passed to our expander function.

What about the expansion? As the
when-let documentation
says, @whenLet should run the statements in its body if all the
bound variables are true. And it would be nice to have a new scope for
our variables, so this is a suitable expansion:

More macros

Conclusion

To define a Cheat-JS macro, you need to know the three things required
for any macro: the invocation, the expansion, the transformation.

You also need to call one of the cheat-js:register-*-macro functions
to declare the type of your macro and
cheat-js:register-macro-expander to install the expander
function. It's best to call cheat-js:clear-macros before your macro
definitions to start clean.

Use cheat-js:explode to macroexpand JavaScript code after you
defined the macros.

Troubleshooting

If things break, right now the best course is to isolate the problem
(it's a problem in the expander? is the expected AST for the expansion
incorrect? etc.). The examples above should provide some information
about how Cheat-JS works. Reading cheat-js.lisp (rather small right
now, less than a hundred lines) and the tests.lisp files may provide
more clues about what Cheat-JS expects from a macro definition.

Closing thoughts

Still reading? Wow!

One thing is obvious: Cheat-JS makes it possible to write macro-like
transformations on JavaScript code, but it's not nearly as easy as
writing Common Lisp macros. Maybe this isn't a bad thing - we should
be writing macros only when there's no other way to avoid code
duplication.

There are plenty of quirks. There's only so many transformations you
can do (function calls are not as frequent in JavaScript as they are
in Common Lisp, and macro invocations are 'hooked' to function
calls). Maybe parse-js could be tweaked harder to make it possible
to insert macros at other points. For now, the transformations
possible with 'function call' macros are enough for me.

You need to be able to 'pattern match' ASTs and figure out how to
transform macro invocations ASTs into macro expansions ASTs. This is a
basic macro writing skill, but with an indirection (in Common Lisp the
source code is the AST, not so with JavaScript).

You also need to know Common Lisp. In theory, a Cheat-JS based
preprocessor could be distributed and used by people who are only
'consuming' macros produced by someone else (a 'macro
producer'). Hmmm, people will certainly be amused if a 'macro
producer' starts distributing a 40MB executable (this is about the
minimum size for SBCL standalone executables) that explodes constructs
in the source code into larger code :-)

With a JavaScript parser written in JavaScript it would be possible to
do what Cheat-JS does without Common Lisp (though without backquotes
the generation of macro expansions is probably a pain, and the AST
would have to be uniform - nested arrays, no classes, maybe? - to make
it easier to 'pattern match' and analyze).

Right now, the audience of Cheat-JS is probably the audience of
ParenScript (mostly because you need to be a lisper to fully use
Cheat-JS). I (Miron Brezuleanu) wrote a few thousands of lines of
ParenScript code and found that there is some 'impedance mismatch'
between ParenScript and JavaScript (especially around the '.' operator
in JavaScript and modules in JavaScript). This was most likely my
fault: instead of writing Lisp to be compiled to JavaScript, I was
trying to write JavaScript with s-expressions. I found it harder to
write code in ParenScript than in JavaScript, and the presence of
macros didn't compensate for this extra effort. I tried to find a way
to have macros while writing something closer to JavaScript. Cheat-JS
is what I came up with.

Thanks for taking the time to read about Cheat-JS; I hope you'll find
it useful (or at least amusing - or both)!

Please use the Github
issues page to report any
bugs or to post feature requests.