Your second link seems pretty clear, what is your understanding of the two concepts, to see if there is some fundamental confusion on your part, otherwise your question is too broad and subjective IMO.
– James BlackAug 12 '10 at 13:47

1

My question is a question of practicality - why is metaprogramming better than, say, writing a few pieces of parametrized SQL queries and gluing them together based on certain conditions? Or is that metaprogramming? (I don't think so, but that's why I'm asking the question - Is it different and why is it better?).
– Wayne WernerAug 12 '10 at 16:44

1

Because sometimes you don't even know what table you're going to query or what columns you'll return ahead of time (perhaps it depends on a combination of user inputs, too difficult to precalculate all possibilites ahead of time), so you use dynamic SQL (which maybe could be thought of as a form of metaprogramming).
– FrustratedWithFormsDesignerAug 12 '10 at 18:40

11 Answers
11

Imagine a guy who builds cars. Say it's the same thing as using a computer.
At some point he realizes he's always doing the same thing, more or less.
So he builds factories to build cars, and it's much better. He's now programming !
Nevertheless, once again, at some point, he realizes he's always doing the same thing, to some extent.
Now he decides to build factories that build factories that build cars. That's metaprogramming.

Metaprogramming is immensely powerful, but one glitch in the system makes all advantages turn into monster difficulties. So master it and use it... Or stay away !

This is a good answer, but it shows the thing I find most annoying about metaprogramming. When does it end? What is stopping us from having factories that build factories that build factories that build sheep that build factories that build factories the build cars?
– XQLRSZApr 23 '15 at 18:36

1

@XQLRSZ The same way you know when to stop writing any code: when it stops making your life easier. Unless you're doing something over and over again, there's no reason to extract it into a piece of code.
– Andrew PiliserFeb 23 '16 at 18:34

Entirely agree with first paragraph. Disagreee with the second part. Metaprogramming, in my experience, allows more bugs than programming itself. If there's a bug, who cares, we can just fix it as it's only been distributed to devs. If there's a bug in our actual code though, that gets released and it's a headache to fix. So my advice would be to metaprogram freely and in a carefree manner. Don't worry about TDD or perfection. Just get it to work.
– Colm BhandalApr 15 at 11:18

I think of metaprogamming as "programs that write (or modify) other programs".
(Another answer said "factories that make factories", nice analogy).

People find all sorts of uses for this: customizing applications, generating boilerplate code,
optimizing a program for special circumstances, implementing DSLs, inserting code to handle
orthogonal design issues ("aspects") ...

What's remarkable is how many different mechanisms have been invented to do this piecemeal:
text-templates, macros, preprocessor conditionals, generics, C++-templates, aspects, reflection,...
And usually some of these mechanisms are built into some languages, and other mechanisms
into other languages, and most languages have no metaprogramming support at all.
This scatter-shot distribution of capabilities means that you might be able to do some
kinds of metaprogramming in one language, with limitations, and yet not be able to do those kinds in another. That's aggravating :-}

An observation that I have been following to the hilt is that one can build generic
metaprogramming machinery that works with any language in the form of
program transformations.
A program transformation is a parameterized pattern: "if you see this syntax, replace it by that syntax".

One transformation by itself generally isn't impressive, but dozens or hundreds can make
spectacular changes to code. Because (sophisticated) program transformations can in
effect simulate a Turing machine, they can carry out arbitrary code changes, including
all those point-wise techniques you find scatter-shotted about.

A tool that accepts language definitions. language-specific transformations and generates
another to apply those transformations is a meta-metaprogramming tool:
a program to write "programs that write programs".

The value is that you can apply such tool to carry out wide varieties of changes
to arbitrary code. And, you don't need the language design committee to realize that you
want a particular kind of metaprogramming support, and hurry up to provide it
so you can get on with your job today.

An interesting lesson is that such machinery needs strong program analysis (symbol
tables, control and data flow analysis, etc.)
support to help it focus on where problems are in the code, so that metaprogramming
machinery can do something at that point (a very weak kind of example of this are
point-cut specifications in aspects, that say "make changes at places that look like this").

The OP asked for specific examples of where metaprogramming was applied.
We've used our "meta"-metaprogramming tool (DMS Software Reengineering Toolkit) to carry out the following activities on large code bases automatically:

Language Migration

Implementing Test Coverage and Profilers

Implementing Clone Detection

Massive architecture reengineering

Code generation for factory control

SOAization of embedded network controllers

Architecture extraction for mainframe software

Generation of vector SIMD instructions from array computations

Reverse engineering of code back to concepts

across many languages, including Java, C#, C++, PHP, ...

The OP also asked, "Why was this better than the alternative?"
The answer has to do with scale, time, and accuracy.

For large applications, the sheer size of the code base means you don't have the resources
or the time to make such analyses or changes by hand.

For code generation or optimization tasks, you might be able to do
it by hand, but the tools can do it much faster and more accurately.

In essence, these tools do what human beings simply cannot.

It is worth noting that the tools have no creativity; you still
need humans to determine what to have them do, e.g., to decide
what the task is (see above list for examples) and determine
how to define the analyses/transformations to achieve the effect.
You still need meta-programmers. However, when a meta programmer
arms such a tool with the right knowledge, the resulting code can
appear to be built by an incredibly fast, creative, expert coder.

I've gotten the most use out of metaprogramming for bridging between different APIs.

A working example would be FireBreaths JSAPIAuto1 that eases writing C++ classes that are exposed to JavaScript. By providing a registering facility for the functions that are to be exposed, the argument types can be inspected and from that fitting code generated at compile-time that converts from the script-API-types to native C++ types and back, even directly supporting map, vector, etc.

As a simple example, consider an exposed add(a, b) function that uses some scripting API types:

I have an SQL Plus script that generates and then executes other SQL Plus scripts. The generates script runs queries against some tables that have time-stamp fields, and when I designed the script, it was impossible to know what time window to select. So, the main script does its work, and figures out what time ranges need to be in the sub scripts. Then it generates the subscripts by writing their code to file (and substituting placeholders for the actual start and end times). Finally it executes the subscript(s). I've used this trick for a few situations now (though often more complicated than this one) where the structure of the substeps depends on results of earlier steps.

I once got a spreadsheet mapping elements from an XSD to table columns in a database. It was possible to generate XSL snippets and complete queries from the spreadsheet using macros and VBA. These snippets and queries were copied and pasted (mostly as-is with no neede changes) into the system that executed them and processed the results. Not a pretty solution but it certainly made a very tedious job a lot less tedious, and the code that resulted was probably a lot more consistent-looking than if I had spent a week or two writing it all by hand.

I can give my own specific example: I am developing ABSE, which is a meta-programming approach. With ABSE you create a model (actually, a tree) where each item is an "Atom". This Atom represents a "concept" and contains the necessary meta-data for its definition.

In ABSE, the implementation of a concept is actually a "mini-program".

Then, the host modeler (AtomWeaver, developed alongside ABSE) takes the model and "weaves" a generator program out of all its Atoms. That program is then run, generating the desired artifacts (source code, data, etc).

So, the ABSE workflow is:

Create a discrete concept (a fraction of the meta-metaprogram)

Reuse that concept in a model (effectively building the metaprogram)

Host modeler weaves and runs the metaprogram

The metaprogram generates your final program

At first sight this looks like a lot of redundant, complex work, but it is actually quite straightforward if you grasp the concept.

Advantages of meta-programming (not exclusive to ABSE)?:

Changing the model and regenerating a complete system (Imagine refactoring features instead of source lines).

Changing a few definitions in the model can result in distinct programs (a Software Product Family).

By reusing templates, you can change the template's code, regenerate and get your code changed in dozens, hundreds of places.

Many others, really

Metaprogramming, code generation, program transformation are new exciting worlds in software development, IMHO. However, metaprogramming requires a new skill: meta-thinking.

We can define meta-thinking as "thinking about how you think about your own development". A kind of class reflection, applied on yourself. In practice, you must find out your own development patterns, isolate them, make them generic, and then turn them into metaprograms using your favorite technique, being it ABSE, DSL's, DSM, etc.

I´m not a tech-savvy, but let me ask you this, did you just say that you have create a 4 level system as if they were building blocks, capable of future reconfiguration depending on the needs of programming, so that with those building block you can weave a fix, change or build old or new nets ?
– Nader BelalApr 23 at 14:26

Metaprogramming based libraries/code help write directly explicit and simple code that will generate implementation details code for you, depending on parameters used.

Boost is full of (C++) libraries that demonstrate what can be achieved with metaprogramming. Some good (and maybe hard to understand) examples are Proto that allow implementation of DSL, Spirit that allow to write a compiler using EBNF grammar directly inside the code, and many other blow-minding libraries.

The corresponding code generation example is the GNU flex and bison tools. Like Spirit, it is used to create lexical analyzers and parsers. However, they have their own language that is not C or C++, must run as a separate step, and their output must be compiled by a C or C++ compiler. This illustrates the fundamental difference between code generators and metaprogramming: Code generators do their job as a separate "preprocessing" step, whose output is then compiled, while metaprogramming is simply compiled to object code in one step just like the rest of the code.
– Mike DeSimoneAug 12 '10 at 14:11

I don't think that "metaprogramming" vs. "code generation" is "does the source-to-object-code compiler do it" or "is it done inside my primary programming language". All that matters is that the code I write manually gets converted into something that runs. The "compiler" can usefully do metaprogramming (that's following metaprogramming directives I wrote) at a variety of stages, early, middle, late. What matters is that metaprogramming is about manipulating the code I wrote, using additional code I wrote, to produce the final code that executes. Yes, that includes code generators IMHO.
– Ira BaxterJul 4 '13 at 15:29

I've created a program tool which will generate ASP.NET web page source code from any MS Access data entry form. The technique that I used was to create my own ASP.NET text templates for each type of form control. I simply plugged in the values such as TOP, LEFT, HEIGHT, WIDTH, CONTROLSOURCE from the MS Access form objects meta data.
For example, my template for an ASP.NET text box looks like this:

My program generates the entire web page source code for one MS Access form in 2-3 seconds.The alternative is to code by hand the ASP.NET web page from scratch; a task that could potentially take hours or even days.

Imagine an MS Access database with 24-35 forms. To hand code each and every form as an ASP.NET web page source code could take weeks if not months.
Using a conversion tool with meta programming techniques , in this case, reduces development time for the web pages from weeks and months to hours .

You have a set of third-party classes, to which you want to add generic behaviour - for example some kind of security/access control, mapping out objects as JSON, etc.

You could write or generate sub-classes for everything, adding wrapper methods to add in access control and call the superclass. With meta-programming, you can do that at runtime, and also your changes will be automatically applied to any additional / changed third party classes.

With the JSON example, by using introspection of the class you should be able to generate the code to serialise an object, and then add this as a method to the class. The other extremes would be generating or writing the code upfront (before compilation) and impacting every time the class changes, or a completely generic approach that used introspection on each individual object, each time you wanted to map it.

Depending on the language and runtime in question, a metaprogamming approach is likely to be faster than the wholly generic/introspective one, but slower that upfront code, as you have reduced a lot of data lookups into code.

Where meta-programming doesn't exist directly in a language, it also seems to me that it is often re-invented through frameworks (i.e. IoC style containers like Spring).

You could look at Common Lisp's macros or C++'s templates and see how they're used. Both are metaprogramming in the sense you're using. You'll find that both are used heavily in a lot of code.

Lisp macros are often used to redefine the language. As an example, the last chapter of Paul Graham's On Lisp creates an working object-oriented extension for Common Lisp. Another example is the now-defunct Garnet.

The old Standard Template Library for C++ (mostly incorporated in the standard library) was a way of introducing a large number of containers and algorithms that worked as if they were built into the language, at least in terms of integration and efficiency (not syntactically).

We use meta-programming a lot to create properties in VBA. We have various Excel spreadsheets with many headers on them and we want to define getter/setter properties for each header, allowing us to manipulate cells under that header. Manually doing this would be a nightmare.

The meta programming framework of choice for us was Notepad++ and its find/replace regular expressions capabilities. Here is how we meta-programmed our properties:

Copy a list of headers from Excel to Notepad++

Record a Notepad++ macro to clean up the data (remove whitespaces and special characters). At the end of this we have a list of newline separated strings.

Manually copy the list to another .CSV file and use Excel to generate a list of line numbers. Then copy back to Notepad++.

Write a regex to convert a property name into a property definition, adding all the whitespace, keywords etc. Use the line number as the column number in our property definition.

At the end of this we have a process that's a mixture of manual steps, recorded macros and a regex that we can re-apply every time we want properties for a sheet. And we did! To great effect.

That's the power of meta-programming. When to use it is a matter of experience/intuition. But I recommend answering this question:

Will if be quicker for me to just code this directly, or can I automate some/all of the process, and speed up my process?

That gives you a line to draw beyond which meta-programming is no longer useful. If you can just code it quicker, even if it's 10 repetitions, just do it! Only if it's hundreds of repetitions, or it's something you expect to reuse many times in future then meta program it.

Another point is that there are degrees here. I once wrote a Java program to create a bunch of files for adding a new IntelliJ inspection to an inspections coding project. That was a fair bit of overhead: creating the Java project and compiling it etc. On the other hand, Notepad++ find/replace is just a tiny step above manually typing stuff yourself. The advice here is to start doing things manually and then automate as you see a need, only up to the point where it makes sense. No need for a Java program when Notepad++ will do. No need for Notepad++ when manually typing it will do.