I, as a Haskeller, envy Lispers as well. Metaprogramming is so incredibly easy in Lisp. Oftentimes I can find good solutions in Haskell for things that people would normally do using metaprogramming, but still. And almost always the types really help me, but every now and then they get in the way.

All I really want in a language is something similar to Haskell, with Lisp-like metaprogramming, easy support for Erlang-style concurrency, and syntax that doesn't immediately terrify the Blub programmers. Is that so much to ask?

Metaprogramming and syntax don't go that well together. If you're going to manipulate programs effectively, you're going to be operating on source trees, and it's easiest to do that when source code and source tree are of the same form.

Thank you for picking up on that half of my joke. Everyone else went for the type system part...

Not that it isn't easy to make a candied Lisp dialect with "friendly" syntax more appealing to non-Lisp programmers, but anyone who actually learns a Lisp quickly realizes that it just gets in the way...

If I were to personally try creating an "ideal language" to match my tongue-in-cheek plea, I'd probably start with a terse Lisp-like syntax and a HM-like type system and go from there.

Yes I think that is too much to ask. Metaprogramming, I think, is by definition incompatible with compile time type checking.

The "meta" in Metaprogramming means that the program is able to manipulate itself (recursively) at runtime. I believe it's theoretically impossible to combine that with AOT type checking and Haskell without its type system would not be Haskell at all.

[edit] Of course you could argue that the sort of thing C++ does with templates is metaprogramming as well, only at compile time. In that case I suggest that we need another term for that because it's totally unlikle what Lisp is so good at.

If the compiler is part of the runtime (i.e. an interpreter that compiles expressions in order to evaluate them), and then the type checker is part of the compiler, even eval should be able to throw type errors. What, then, is the problem?

It depends on what you mean by "runtime." If you're creating a new datatype, is that really "at runtime," or are you just talking to the compiler interactively?

To put it another way: metaprogramming might have effects at runtime, but not the kind of effects that change depending on what the runtime does. Metaprogramming should be deterministic for your program to be considered to be "in production."

Thank you so much! I've had informal ideas quite like this from working on my "HTML rewriting system" part of my (unreleased) Common Lisp web framework. I now have a very specific search term to acquire more formalized knowledge which should speed up some future developments a lot because I won't have to independently come up with solutions to as many of the problems now.

My definition of metaprogramming is writing a program that manipulates itself. I'm not sure I would call any arbitrary code generator metaprogramming as this would cause the "self" in my definition suffer a severe identity crisis :-)

But yes, C++ templates as well as Lisp macros are compile time metaprogramming. All languages that have eval and/or allow function/method bindings to be replaced at runtime allow runtime metaprogramming.

Since static typing guarantees certain invariants it cannot be compatible with a program that violates those invariants at runtime. You could still decide to consider it metaprogramming when the program manipulates itself only within the limits of those guarantees, but when you look at what real world runtime metaprogramming is being used for (for instance in Rails) you will realise that these things (e.g method_missing) would not be possible within the limits of a statically typed language.

[edit] Lisp makes both compile time and runtime metaprogramming exceptionally easy due to its homoiconic nature.

It's certainly possible for a typed language like ML or Haskell to support metaprogramming. As other have noted, Haskell has Template Haskell. However, systems like MetaML and MetaOCaml support metaprogramming and give much stronger typing guarantees than Template Haskell. See http://www.metaocaml.org/examples/ for inspiration.

Sure, but there are theoretical limits to what runtime metprogramming can do whilst upholding the guarantees that a static type system provides.

For instance, in order to check that a particular function call conforms to the function signature, the function signature must at least exist. Type checking a call to a function for which not even the signature exists anwhere within the system is impossible.

One of Haskell's defining features is a very expressive and powerful static type system, so yeah, compile-time type inference/checking was pretty heavily implied by "similar to Haskell".

And yes, the fact that I was essentially asking for a language with both static typing and runtime self-modification was the reason for the tongue-in-cheek "is that so much to ask". Might as well ask for a program that can compile any legal perl program [0].

That said, I suspect there's a lot that could be done to allow certain subsets of metaprogramming techniques in a static-typed language; some sort of crazy meta-type system that lets the compiler prove that something will only produce code with a particular polymorphic type, maybe? I don't know.

Yes. A lot of the meta-programming is actually programming over the structure. For example, given a Java class, you could generate database code. In Java, this relies heavily on introspection, but you if you're interested in doing this in a type-safe way then you could look at Generic Programming (in Haskell).

Also, you can generate code (at runtime) which is type-safe, compile the code, etc. Everything you ask for is possible in Haskell, however, it is definitely more complicated than in Lisp.

Compiling code at runtime may improve performance, but it doesn't give the kind of guarantees that static typing gives you.

Static type checking proves that certain invariants hold at runtime and hence that certain defects are not possible. If a program can manipulate itself at runtime in arbitrary ways (not just reflect on itself in a read-only fashion), those proofs become invalid.

Lisp-like metaprogramming is really tricky, because a lot of the language that you interact with is defined by macros. The Haskell that you see is a thick layer of syntactic sugar over a much smaller Haskell core.

Whoa. That doesn't match my experience at all. I'd say it's simple and practically effortless (compared to what you'd have to do elsewhere). It would be surprising if it weren't, given that the whole language is organized around code=data.

You should check out the Qi language, though the developer who has historically contributed the most to it has departed some months ago. It has the ideas you are looking for (sans message-passing concurrency, I think.)

Does Template Haskell affect this discussion at all? (That's not a rhetorical question - I haven't tried any Template Haskell, so I'm genuinely interested to know what it's capable of and how it compares to Lisp macros.)

TH does a lot of the same stuff. On a practical level, it isn't as good, though. The implementation leaves a lot to be desired. For example, I once tried to use TH to check XMonad configurations at compile-times (specifically, that their keymaps were sensible), but turns out TH cannot operate on anything defined in the same module!

Can you give an example of a situation where Haskell type checking gets in your way? Not to take issue, but because I'm a collector. I've been told that it's not that easy to make a list of functions in Haskell unless they all have the same input and output types, which seems restrictive to me.

It's not "easy" to make a list, but instead maybe you would be able to work with a tuple (which is easy).

More practically, Haskell is going to ask you implicitly why you want a heterogenous list of functions. In all likelihood they're going to have a common interface at some point and you abstract around that so that the list is still "homogenous" (which is harder, but still pretty easy).

Well, polymorphic lists are one thing that are somewhat awkward. Other thing is that a small change in some part of program (say introducing an additional parameter, or having to use IO or state) can require massive changes in rest of the program, or a PhD in category-theory.