I think a better way to phrase this would be: Can a language have Lisp's powerful macros without the code being written as a bunch of datastructures and offering the the same uniform api to manipulate those datastructures as offered in "regular" code?
–
dnolenApr 27 '10 at 12:42

Programming languages now have at least three choices to make such calculations possible:

base the source code transformations on string transformations

use a similar primitive data structure like Lisp. A more complex variant of this is a syntax based on XML. One could then transform XML expressions. There are other possible external formats combined with internalized data.

use a real syntax description format and represent the source code internalized as a syntax tree using data structures that represent syntactic categories.

For all these approaches you will find programming languages. Lisp is more or less in camp 2. The consequence: it is theoretically not really satisfying and makes it impossible to statically parse source code (if the code transformations are based on arbitrary Lisp functions). The Lisp community struggles with this for decades (see for example the myriad of approaches that the Scheme community has tried). Fortunately it is relatively easy to use, compared to some of the alternatives and quite powerful. Variant 1 is plain ugly. Variant 3 leads to a lot complexity in simple AND complex transformations.

Another problem is HOW to transform the code. One approach would be based on transformation rules (like in some Scheme macro variants). Another approach would be a special transformation language (like a template language which can do arbitrary computations). The Lisp approach is to use Lisp itself. That makes it possible to write arbitrary transformations using the full Lisp language. In Lisp there is not a separate parsing stage, but at any time expressions can be read, transformed and evaluated.

Lisp is kind of a local maximum of simplicity for code transformations.

Also note that READ reads s-expressions to internal data. In Lisp one could either use a different READer for a different external syntax or reuse the Lisp built-in reader and reprogram it using the read macro mechanism. There are examples for both approaches to provide a different external syntax in Lisp.

The current Lisp syntax is popular among Lisp programmers for two reasons:

1) the data is code is data idea makes it easy to write all kinds of code transformations based on the internalized data. There is also a relatively direct way from reading code, over manipulating code to printing code.

2) the text editor can be programmed in a straight forward way to manipulate s-expressions. That makes code and data transformations in the editor relatively easy.

Hi Rainer. Can you elaborate on The consequence: it is theoretically not really satisfying and makes it impossible to statically parse source code (if the code transformations are based on arbitrary Lisp functions). If this is theoretically not really satisfying, what would be satisfying then?
–
Faheem MithaMar 19 '12 at 9:12

Absolutely. It's just a couple orders of magnitude more complex, if you have to deal with a complex grammar. As Peter Norvig noted:

Python does have access to the
abstract syntax tree of programs, but
this is not for the faint of heart. On
the plus side, the modules are easy to
understand, and with five minutes and
five lines of code I was able to get
this:

This was rather a disapointment to me. The Lisp parse of the equivalent expression is (+ 2 2). It seems that only a real expert would want to manipulate Python parse trees, whereas Lisp parse trees are simple for anyone to use. It is still possible to create something similar to macros in Python by concatenating strings, but it is not integrated with the rest of the language, and so in practice is not done.

Since I'm not a super-genius (or even a Peter Norvig), I'll stick with (+ 2 2).

In order to have lisp-style macros, you need a way of representing source-code in data structures. In most languages, the only "source code data structure" is a string, which doesn't have nearly enough structure to allow you to do real macros on. Some languages offer a real data structure, but it's too complex, like Python, so that writing real macros is stupidly complicated and not really worth it.

Lisp's lists and parentheses hit the sweet spot in the middle. Just enough structure to make it easy to handle, but not too much so you drown in complexity. As a bonus, when you nest lists you get a tree, which happens to be precisely the structure that programming languages naturally adopt (nearly all programming languages are first parsed into an "abstract syntax tree", or AST, before being actually interpreted/compiled).

Basically, programming Lisp is writing an AST directly, rather than writing some other language that then gets turned into an AST by the computer. You could possibly forgo the parens, but you'd just need some other way to group things into a list/tree. You probably wouldn't gain much from doing so.

chrisdone.com/z is an experiment showing string-based macros are not horrific, given the minimum structure of knowing where the parameter ends. TCL is similar - with a bit of support for {...} nested delimiting and some escaping, it makes it easy to write higher-order commands which are really string-based macros. TCL is also quite homoiconic - it uses same nested braces structure for data. Still, it's not exactly pretty (or fast) compared to Lisp's real trees - it's been abserved that "TCL is Lisp on drugs".
–
Beni Cherniavsky-PaskinApr 4 '13 at 16:37

Parentheses are irrelevant to macros. It's just Lisp's way of doing things.

For example, Prolog has a very powerful macros mechanism called "term expansion". Basically, whenever Prolog reads a term T, if tries a special rule term_expansion(T, R). If it is successful, the content of R is interpreted instead of T.

I'm curious does this mechanism have access to the entire Prolog language to do the expansion? Wikipedia makes it sounds like something that happens at some preprocessor step making it a lot less useful than Lisp macros if that is true.
–
dnolenApr 27 '10 at 12:33

1

Term expansion, at least in SWI-Prolog, has access to the entire Prolog language and to every preceding term. It's not a preprocessor, it is a part of the interpreter/compiler itself.
–
Little Bobby TablesApr 27 '10 at 13:33

Looks a lot more complicated than Common Lisp style macros. I suppose that's the thing about CL style macros- they hit a sweet spot of simplicity (in being able to understand how to write them) and power. Perhaps one day more programmers will understand CL style macros and we'll get to the next level.
–
dnolenApr 27 '10 at 12:37

1

@dnolen: Common Lisp isn't the only Lisp. Dylan's macros are based on Scheme's hygienic macros, which have a slightly more complcated syntax but carry other benefits.
–
ChuckApr 27 '10 at 17:57

Have a look at Sweet-expressions. Wheeler makes a very good case that the reason things like infix notation have not worked before is that typical notation also tries to add precedence, which then adds complexity, which causes difficulties in writing macros.

For this reason, he proposes infix syntax like {1 + 2 + 3} and {1 + {2 * 3}} (note the spaces between symbols), that are translated to (+ 1 2) and (+ 1 (* 2 3)) respectively. He adds that if someone writes {1 + 2 * 3}, it should become (nfx 1 + 2 * 3), which could be captured, if you really want to provide precedence, but would, as a default, be an error.

He also suggests that indentation should be significant, proposes that functions could be called as fn(A B C) as well as (fn A B C), would like data[A] to translate to (bracketaccess data A), and that the entire system should be compatible with s-expressions.

Overall, it's an interesting set of proposals I'd like to experiment with extensively. (But don't tell anyone at comp.lang.lisp: they'll burn you at the stake for your curiosity :-).

Erlang's parse transforms are similar in power to Lisp macros, though they are much trickier to write and use (they are applied to the entire source file, rather than being invoked on demand).

Lisp itself had a brief dalliance with non-parenthesised syntax in the form of M-expressions. It didn't take with the community, though variants of the idea found their way into modern Lisps, so you get Lisp's powerful macros without the parentheses ... in Lisp!

Code rewriting in Tcl in a manner recognizably similar to Lisp macros is a common technique. For example, this is (trivial) code that makes it easier to write procedures that always import a certain set of global variables:

With that, all procedures declared with gproc xyz rather than proc xyz will have access to the foo, bar and boo globals. The whole key is that uplevel takes a command and evaluates it in the caller's context, and list is (among other things) an ideal constructor for substitution-safe code fragments.

Take a look at "sweet-expressions", which provides a set of additional abbreviations for traditional s-expressions. They add indentation, a way to do infix, and traditional function calls like f(x), but in a way that is backwards-compatible (you can freely mix well-formatted s-expressions and sweet-expressions), generic, and homoiconic.

No, it's not necessary. Anything that gives you some sort of access to a parse tree would be enough to allow you to manipulate the macro body in hte same way as is done in Common Lisp. However, as the manipulation of the AST in lisp is identical to the manipulation of lists (something that is bordering on easy in the lisp family), it's possibly not nearly as natural without having the "parse tree" and "written form" be essentially the same.

Boo has a nice "quoted" macro syntax that uses [| |] as delimiters, and has certain substitutions which are actually verified syntactically by the compiler pipeline using $variables. While simple and relatively painless to use, it's much more complicated to implement on the compiler side than s-expressions. Boo's solution may have a few limitations that haven't affected my own code. There's also an alternate syntax that reads more like ordinary OO code, but that falls into the "not for the faint of heart" category like dealing with Ruby or Python parse trees.