Reading Paul Graham's essays on programming languages one would think that lisp macros are the only way to go. As a busy developer working on other platforms, I have not had the privilege of using lisp macros. As someone who wants to understand the buzz, please explain what makes this feature so powerful.

Please also relate this to something I would understand from the world of python, java, c#, c development.

13 Answers
13

The short answer: macros are used for defining language syntax extension to Common Lisp or Domain Specific Languages (DSLs). These languages are embedded right within your existing Lisp code. Now, the DSLs can have syntax similar to Lisp (like Peter Norvig's Prolog Interpreter for Common Lisp) or completely different (e.g. Infix Notation Math for Clojure).

As a more concrete example: Python has list comprehensions built-in the language. This gives a simple syntax for a common case divisibleByTwo = [x for x in range(10) if x % 2 == 0] gives the numbers from 0 to 9 that are divisible by two. Back in the Python 1.5 days there was no such syntax, and that would have expressed as:

These are both functionally equivalent. Let's invoke our suspension of disbelief and pretend Lisp has a very limited loop macro that just does iteration and no easy way to do the equivalent of list comprehensions.

In Lisp you could write the following. I should note this contrived example is picked to be identical to the Python code not a good example of Lisp code.

Before I go further: I should explain a little bit of what a macro is. It is a transform of code by code to code. That is a piece of code read by the interpreter (or compiler) which takes in code as an argument does a manipulation and the returns the result which is then run in place.

Of course that's a lot of typing and programmers are lazy. So we could define DSL for doing list comprehensions. In fact, we already using one macro already (the loop macro).

Lisp defines a couple of special syntax forms. The quote (') indicates the next token is a literal. The quasiquote or backtick (`) indicates the next token is a literal with escapes. Escapes are indicated by the comma operator. The literal '(1 2 3) is the equivalent of Python's [1, 2, 3]. You can assign it to another variable or use it in place. You can think of `(1 2 ,x) as the equivalent of Python's [1, 2, x] where x is a variable previously defined. This list notation is part of the magic that goes into macros. The second part is the Lisp reader which smartly subtitutes macros for code but that is best illustrated below:

So we can define a macro called lcomp (short for list comprehension). It's syntax will be exactly like the python that we used in the example [x for x in range(10) if x % 2 == 0] - (lcomp x for x in (range 10) if (= (% x 2) 0))

(defmacro lcomp (expression for var in list conditional conditional-test)
;; create a unique variable name for the result
(let ((result (gensym)))
;; the arguments are really code so we can substitute them
;; store nil in the unique variable name generated above
`(let ((,result nil))
;; var is a variable name
;; list is the list literal we are suppose to iterate over
(loop for ,var in ,list
;; conditional is if or unless
;; conditioanl-test is (= (mod x 2) 0) in our examples
,conditional ,conditional-test
;; and this is the action from the earlier lisp example
;; result = result + [x] in python
do (setq ,result (append ,result (list ,expression))))
;; return the result
,result)))

Now we can execute at the command line:

CL-USER> (lcomp x for x in (range 10) if (= (mod x 2) 0))
(0 2 4 6 8)

Pretty neat, huh? Now it doesn't stop there. You have a mechanism - a paintbrush. You can have any syntax you could possibly want. Like Python or C#'s with syntax. Or .NET's LINQ syntax. In end, this is what attracts people to Lisp - ultimate flexibility.

I know it's pretty unrelated, but I'm wondering about syntax and how the parsing actually works... Say I call lcomp that way ( changing the thirs item from "for" to "azertyuiop" ) : (lcomp x azertyuiop x in (range 10) if (= (% x 2) 0)) will the macro still work as expected ? Or is the "for" parameter used in the loop so that it must be the string "for" when called ?
–
daderMay 16 '13 at 2:05

@dader So, loop is actually another macro and for is a special symbol used in that macro. I could have easily defined the macro without the loop macro. However, if I had, the post would've been much longer. The loop macro and all its syntax is defined in CLtL.
–
gte525uJun 22 '14 at 22:09

In most programming languages, syntax is complex. Macros have to take apart program syntax, analyze it, and reassemble it. They do not have access to the program's parser, so they have to depend on heuristics and best-guesses. Sometimes their cut-rate analysis is wrong, and then they break.

But Lisp is different. Lisp macros do have access to the parser, and it is a really simple parser. A Lisp macro is not handed a string, but a preparsed piece of source code in the form of a list, because the source of a Lisp program is not a string; it is a list. And Lisp programs are really good at taking apart lists and putting them back together. They do this reliably, every day.

Here is an extended example. Lisp has a macro, called "setf", that performs assignment. The simplest form of setf is

(setf x whatever)

which sets the value of the symbol "x" to the value of the expression "whatever".

Lisp also has lists; you can use the "car" and "cdr" functions to get the first element of a list or the rest of the list, respectively.

Now what if you want to replace the first element of a list with a new value? There is a standard function for doing that, and incredibly, its name is even worse than "car". It is "rplaca". But you do not have to remember "rplaca", because you can write

(setf (car somelist) whatever)

to set the car of somelist.

What is really happening here is that "setf" is a macro. At compile time, it examines its arguments, and it sees that the first one has the form (car SOMETHING). It says to itself "Oh, the programmer is trying to set the car of somthing. The function to use for that is 'rplaca'." And it quietly rewrites the code in place to:

For example, in C, the switch/case construct only works with integral types and if you want to use it for floats or strings, you are left with nested if statements and explicit comparisons. There's also no way you can write a C macro to do the job for you.

But, since a lisp macro is (essentially) a lisp program that takes snippets of code as input and returns code to replace the "invocation" of the macro, you can extend your "primitives" repertoire as far as you want, usually ending up with a more readable program.

To do the same in C, you would have to write a custom pre-processor that eats your initial (not-quite-C) source and spits out something that a C compiler can understand. It's not a wrong way to go about it, but it's not necessarily the easiest.

Other things you can do with macros are creating new keywords and/or mini-languages (check out the (loop ...) macro for an example), integrating other languages into lisp, for example, you could write a macro that lets you say something like:

I think it should be: "(apply && ,@exprs) ; this and might be wrong!"
–
SvanteMar 16 '09 at 1:55

1

@svante - on two counts: first, && is a macro, not a function; apply only works on functions. second, apply take a list of arguments to pass, so you want one of "(funcall fn ,@exprs)", "(apply fn (list ,@exprs)" or "(apply fn ,@exprs nil)", not "(apply fn ,@exprs)".
–
AaronMar 26 '09 at 17:41

(and ... will evaluate expressions until one evaluates to false, note that side-effects generated by the false evaluation will take place, only the subsequent expressions will be skipped.
–
SlomojoJan 4 '13 at 0:15

Think of what you can do in C or C++ with macros and templates. They're very useful tools for managing repetitive code, but they're limited in quite severe ways.

Limited macro/template syntax restricts their use. For example, you can't write a template which expands to something other than a class or a function. Macros and templates can't easily maintain internal data.

The complex, very irregular syntax of C and C++ makes it difficult to write very general macros.

Lisp and Lisp macros solve these problems.

Lisp macros are written in Lisp. You have the full power of Lisp to write the macro.

Lisp has a very regular syntax.

Talk to anyone that's mastered C++ and ask them how long they spent learning all the template fudgery they need to do template metaprogramming. Or all the crazy tricks in (excellent) books like Modern C++ Design, which are still tough to debug and (in practice) non-portable between real-world compilers even though the language has been standardised for a decade. All of that melts away if the langauge you use for metaprogramming is the same language you use for programming!

Well, to be fair, the problem with C++ template metaprogramming is not that the metaprogramming language is different, but that it's horrid -- it was not so much designed as discovered in what was intended to be a much-simpler template functionality.
–
Brooks MosesJan 7 '11 at 1:55

@Brooks Sure. Emergent features are not always bad. Unfortunately, in a slow moving committee driven language, it's difficult to fix them when they are. It's a shame so much of C++'s modern useful new features are written in a language few can even read, and there is a huge gap between the average programmer and the "high priest".
–
Matt CurtisJan 7 '11 at 3:13

2

@downvoter: if there's something wrong with my answer, please leave a comment so we can all share the knowledge.
–
Matt CurtisJan 7 '11 at 8:22

A lisp macro takes a program fragment as input. This program fragment is represented a data structure which can be manipulated and transformed any way you like. In the end the macro outputs another program fragment, and this fragment is what is executed at runtime.

C# does not have a macro facility, however an equivalent would be if the compiler parsed the code into a CodeDOM-tree, and passed that to a method, which transformed this into another CodeDOM, which is then compiled into IL.

This could be used to implement "sugar" syntax like the for each-statement using-clause, linq select-expressions and so on, as macros that transforms into the underlying code.

If Java had macros, you could implement Linq syntax in Java, without needing Sun to change the base language.

Here is pseudo-code for how a lisp-style macro in C# for implementing using could look:

I'm not sure I can add some insight to everyone's (excellent) posts, but...

Lisp macros work great because of the Lisp syntax nature.

Lisp is an extremely regular language (think of everything is a list); macros enables you to treat data and code as the same (no string parsing or other hacks are needed to modify lisp expressions). You combine these two features and you have a very clean way to modify code.

Edit: What I was trying to say is that Lisp is homoiconic, which means that the data structure for a lisp program is written in lisp itself.

So, you end up with a way of creating your own code generator on top of the language using the language itself with all its power (eg. in Java you have to hack your way with bytecode weaving, although some frameworks like AspectJ allows you to do this using a different approach, it's fundamentally a hack).

In practice, with macros you end up building your own mini-language on top of lisp, without the need to learn additional languages or tooling, and with using the full power of the language itself.

Since the existing answers give good concrete examples explaining what macros achieve and how, perhaps it'd help to collect together some of the thoughts on why the macro facility is a significant gain in relation to other languages; first from these answers, then a great one from elsewhere:

DOLIST is similar to Perl's foreach or Python's for. Java added a similar kind of loop construct with the "enhanced" for loop in Java 1.5, as part of JSR-201. Notice what a difference macros make. A Lisp programmer who notices a common pattern in their code can write a macro to give themselves a source-level abstraction of that pattern. A Java programmer who notices the same pattern has to convince Sun that this particular abstraction is worth adding to the language. Then Sun has to publish a JSR and convene an industry-wide "expert group" to hash everything out. That process--according to Sun--takes an average of 18 months. After that, the compiler writers all have to go upgrade their compilers to support the new feature. And even once the Java programmer's favorite compiler supports the new version of Java, they probably ''still'' can't use the new feature until they're allowed to break source compatibility with older versions of Java. So an annoyance that Common Lisp programmers can resolve for themselves within five minutes plagues Java programmers for years.

Lisp macros represents a pattern that occurs in almost any sizeable programming project. Eventually in a large program you have a certain section of code where you realize it would be simpler and less error prone for you to write a program that outputs source code as text which you can then just paste in.

In Python objects have two methods __repr__ and __str__. __str__ is simply the human readable representation. __repr__ returns a representation that is valid Python code, which is to say, something that can be entered into the interpreter as valid Python. This way you can create little snippets of Python that generate valid code that can be pasted into your actually source.

In Lisp this whole process has been formalized by the macro system. Sure it enables you to create extensions to the syntax and do all sorts of fancy things, but it's actual usefulness is summed up by the above. Of course it helps that the Lisp macro system allows you to manipulate these "snippets" with the full power of the entire language.

In short, macros are transformations of code. They allow to introduce many new syntax constructs. E.g., consider LINQ in C#. In lisp, there are similar language extensions that are implemented by macros (e.g., built-in loop construct, iterate). Macros significantly decrease code duplication. Macros allow embedding «little languages» (e.g., where in c#/java one would use xml to configure, in lisp the same thing can be achieved with macros). Macros may hide difficulties of using libraries usage.

In python you have decorators, you basically have a function that takes another function as input. You can do what ever you want: call the function, do something else, wrap the function call in a resource acquire release, etc. but you don't get to peek inside that function. Say we wanted to make it more powerful, say your decorator received the code of the function as a list then you could not only execute the function as is but you can now execute parts of it, reorder lines of the function etc.

I got this from The common lisp cookbook but I think it explained why lisp macros are good in a nice way.

"A macro is an ordinary piece of Lisp code that operates on another piece of putative Lisp code, translating it into (a version closer to) executable Lisp. That may sound a bit complicated, so let's give a simple example. Suppose you want a version of setq that sets two variables to the same value. So if you write

(setq2 x y (+ z 3))

when z=8 both x and y are set to 11. (I can't think of any use for this, but it's just an example.)

It should be obvious that we can't define setq2 as a function. If x=50 and y=-5, this function would receive the values 50, -5, and 11; it would have no knowledge of what variables were supposed to be set. What we really want to say is, When you (the Lisp system) see (setq2 v1 v2 e), treat it as equivalent to (progn (setq v1 e) (setq v2 e)). Actually, this isn't quite right, but it will do for now. A macro allows us to do precisely this, by specifying a program for transforming the input pattern (setq2 v1 v2 e)" into the output pattern (progn ...)."