Best way to insert a method into a class with a bound variable

I'm working for the first time intensively with Scala macros and I still lack the more basic concepts. For example I'd like to build the definition of a method that access a free variable and then insert that method into a classdef and bind the free variable to some of the fields of the class. What's is the best way of doing that?

There's no easy solution to robust symbol management in the current macro system, so binding manipulations is something that might prove challenging. If you could elaborate on your use case, I will try to help.

What I'm trying to build apart is the method "apply(tr:scala.util.Try)". This method can call the promise "result$async" to give the result of the computation of the async block.

I'd like to be able to build that method outside of the construction of the state machine class to then add it to the class and make sure that the "result$async" variable refers to the field of the class.

By the way, I'm using Scala 2.11.0-M7 so there's no problem in having a solution that uses quasiquotes. My only fear with quasiquotes is that If I've understood well I have to take care of hygiene of the macros when using them.

You might find it useful to read through the entire discussion, but in short if we want to force binding of a given identifier (which can be Ident, Select or SelectFromTypeTree, i.e. subtypes of RefTree) to a given definition (which can be PackageDef, ClassDef, ModuleDef, ValDef, DefDef or TypeDef, i.e. subtypes of MemberDef), then we make sure that symbols of both trees are the same.

This is currently the only way of working with bindings (we plan to introduce something new and shiny in the future, but it's too early to tell at the moment).

=== 2. Problems with using symbols ===

The thing about symbols as they are implemented in our compiler is that they are very capricious.

First of all, unlike with trees or types, you can't create symbols on a whim. If your MemberDef is already attributed ("attributed" = "assigned with symbols and types") by the typechecker, then you can bind to it using its symbol. That's all good. However, if you've just created a MemberDef from scratch (either using reify or quasquotes), then it doesn't have a symbol yet. A natural reaction to this is to just go and create a symbol for that MemberDef, but it's quite complicated (we'll probably cover this a bit, later in the post).

Secondly, oftentimes you can't take a symbol created somewhere and transplant it somewhere else. E.g. if you already have `def sum(num: Int) = freeVar + num` typechecked (and typechecking among other things creates symbols for MemberDefs) then you can't just put this tree inside MyClass.

There are no good reasons for this to be complicated - it's just a very unfortunate artifact of scalac's implementation strategy. We have plans to make this dead simple (as it should rightfully be), but it's going to take time to get there, and at the moment we have to deal with what we have.

=== 3. Two ways of dealing with symbol problems ===

In situations like async's, sbt's and yours, you have two options: a simple but not very robust one, and a really complicated but almost bulletproof one. Let's see what we have.

1) The first way is to abandon symbols altogether and rely on a gensym-like approach to binding. To make sure that you bind to `result$async`, you give it a sufficiently unique name (e.g. result$async looks awkward enough to hope that noone's going to use it) and then hope that there won't be inadvertent name collisions. To make sure that there aren't any problems with transplanting symbols, you unattribute all trees that you plan to transplant using `c.resetLocalAttrs`, destroying all symbols and types that have been assigned to them.

This is more or less simple, but is brittle in several ways (however the alternative requires noticeable effort, so people typically go for it anyway - even SBT until very recently used resetAttrs).

The first problem is that, when names are handled on such a low level, there's never such thing as a sufficiently unique name. Even with obscure names, clashes do happen, and most likely not because of someone's ill intent, but because of unfortunate interaction between your macros. E.g. if you have an async block inside an async block, you might inadvertently bind to the wrong result$async. Such situations are quite rare, but they do happen.

The second problem is much more noticeable. There's a nasty thing about scala's typechecker. In some cases it's not idempotent, which means that if you take a tree of a certain shape, typecheck it, and then reset attrs - you might end up with a tree that's not going to typecheck anymore: https://issues.scala-lang.org/browse/SI-5464. Affected tree shapes are sufficiently rare in order to not become a tragedy, but they are also sufficiently frequent to become a nuisance. So far there's nothing to be done here - it's something that you have to accept if you go for this.

2) The second way is to nurture those symbols manually, which means that you're going to cater for those symbols' needs yourself, without relying on the typechecker. This implies two things: manual symbol creation and owner chain repairs.

Creating a symbol manually requires knowing/creating its owner and flags. The owner comes from the enclosing lexical context (e.g. a class method's owner is the class's symbol), but I don't have an exact recipe for this at the moment - you'd have to consult scalac's sources to see what owner is appropriate in this or that situation. Flags are also context-dependent, and scalac's sources would also be the best documentation here. This covers the first problem with symbols.

All in all, we've seen that there are two ways of addressing the challenge that you're facing. Using unique names and resetAttrs is more or less simple to execute, but isn't robust enough to support 100% of Scala features. Manual creation of symbols and symbol repairs work just fine with everything that the users might throw at you, but that's nontrivial to implement.

I'll go for the first approach since I'm working in a really simple implementation that doesn't need to be bullet proof. However I'll be really happy to get a deeper understanding of the compiler so I may try to play a bit with manual creation of symbols just for the fun.

Any pointers for where to look to learn some of the details about how to create symbols?

That's perfect, now I understand better what I need to do to be able to bind variables.

Now I'm fighting with the manipulation of the symbols and I've seen the following code:

caseclassSymLookup(stateMachineClass:Symbol,applyTrParam:Symbol){

defstateMachineMember(name:TermName):Symbol=

stateMachineClass.info.member(name)

defmemberRef(name:TermName):Tree=

gen.mkAttributedRef(stateMachineMember(name))

}

That is used in async to get a symbol and generate a reference for a term name. However this is used by doing quite a complex setup based on Global that seems like a global context for doing transformations. Is there a nicer way to do this or should I resort to the set up that the guys in async have used?

Global is the cake of the compiler itself. As opposed to c.universe, which exposes a public subset of compiler APIs, with Global you can use all functions and helpers defined in the compiler. It might be that you can do everything you need using public APIs, but it will most likely be more convenient to use Global.

I've ended up playing with global and building the trees and symbols by hand. However I'm running into the following problem:

scala.reflect.internal.Types$TypeError: value <none> is not a member of <type-who-calls-my-macro>

and I can't figure out how to locate where this is happening. I guess some of the symbols is not well assigned but I can't find where even debugging scalac. Anyone knows what is the source of this error and how I can spot what part of the code generated by my macro is producing it?

You could try printing out your macro expansion with showRaw(..., printIds = true). That will tell you which trees have their assigned and which ones don't. printTypes = true will help with looking into types.

I recommend that you compile with -Xprint:typer to show the entire tree after typechecking/macro expansion. You can also try -Ycheck:typer which might find corrupt symbol structures (symbols representing defintions must have the correct "owner" set; when you splice trees around this can become inconsistent).

As the author of scala-async, I can only share that this is super hard to get right. The best way you can get help is to try to boil down a small, self-contained example that exhibits the same problem that you can share with us.

The "really small" part that counts is actually the macro implementation. If you can prepare something that I can clone, run, and witness the crash, I can help. I don't spot the problem from that output alone.

As you'll see I've tried to create a simplified version of scala/async using some pieces of your code so I guess that'll make it easy for you to understand what's going on in the code. This being my first project with macros and due to my lack of understanding of the compiler the project is full of code for debugging, and rests of tests that I've done to try to figure out what was going on so I apologize in advance for the crap that you'll find inside : ).

As you'll see I've tried to create a simplified version of scala/async using some pieces of your code so I guess that'll make it easy for you to understand what's going on in the code. This being my first project with macros and due to my lack of understanding of the compiler the project is full of code for debugging, and rests of tests that I've done to try to figure out what was going on so I apologize in advance for the crap that you'll find inside : ).

These symbol ids are printed here because I used the option `-uniqid`, which is handy to disambiguate same-named symbols.

The anonymous function's symbol, `value $anonfun#52977` is owned by `value result#52663`. That's the symbol that logically encloses the macro application (aka the "current owner" in the typer's Context.)

But, the correct owner for that place in the tree is the closest enclosing definition (val/def/function), which is this case should be `method resume$async#52872`.

So, when it comes to expanding the `MyStateMachine.this` in the body of that closure, the backend follows the owner-chain expecting to hit the enclosing class. The chain of `outer$` selects show its ever-hopeful, but ultimately fruitless search.

I'm having problems again and I think it's again a problem with the owner of my symbol, event though I've tried all possible ways to assign it the right owner. Basically I'm creating a field for my class by creating the valdef and then adding it to the template of my class. Finally I change the owner of my valdef to the class symbol but even after typechecking the symbol table of my class doesn't contain any reference to the field what makes compilation fail in a phase that generates intermediate code.

I'm having problems again and I think it's again a problem with the owner of my symbol, event though I've tried all possible ways to assign it the right owner. Basically I'm creating a field for my class by creating the valdef and then adding it to the template of my class. Finally I change the owner of my valdef to the class symbol but even after typechecking the symbol table of my class doesn't contain any reference to the field what makes compilation fail in a phase that generates intermediate code.

I'm working for the first time intensively with Scala macros and I still lack the more basic concepts. For example I'd like to build the definition of a method that access a free variable and then insert that method into a classdef and bind the free variable to some of the fields of the class. What's is the best way of doing that?