Hello: I've been learning Scala recently (my related background is mostly in C++ templates), and I've run into something I currently don't understand about Scala, and it is driving me insane. :(

(Also, this is my first post to StackOverflow, where I've noticed most of the really awesome Scala people seem to hang out, so I'm really sorry if I do something horrendously stupid with the mechanism.)

My specific confusion relates to implicit argument binding: I have come up with a specific case where the implicit argument refuses to bind, but a function with seemingly identical semantics does.

Now, it of course could be a compiler bug, but given that I just started working with Scala, the probability of me having already run into some kind of serious bug are sufficiently small that I'm expecting someone to explain what I did wrong. ;P

I have gone through the code and whittled it quite a bit in order to come up with the single example that doesn't work. Unfortunately, that example is still reasonably complex, as the problem seems to only occur in the generalization. :(

My expectation in this case is that runAll would be bound to the implicit run argument to runAny.

Now, if I modify runAll so that, instead of taking its two arguments directly, it instead returns a function that in turn takes those two arguments (a trick I thought to try as I saw it in someone else's code), it works:

2) modified code that has the same runtime behavior and actually works

Any explanation anyone may have for this would be much appreciated. :(

(Currently, my best guess is that the order of the implicit argument with respect to the other arguments is the key factor that I'm missing, but one that I'm confused by: runAny has an implicit argument at the end as well, so the obvious "implicit def doesn't work well with trailing implicit" doesn't make sense to me.)

runNil and runAll are both needed (in order to have the repeating and terminating cases: runNil works fine, and as demonstrated by #3 hardcoding it to two levels then works); meanwhile, context is also needed (as demonstrated by #4, where I remove it and it then works); I already simplified the HList down as much as I could (which is why I have to use HNil() instead of the HNil that most similar code actually uses); finally, I need runAny, as that's the function that actually takes the implicit parameter; so no: as stated, I already tried and can't simplify it more than this :(
–
Jay Freeman -saurik-May 30 '11 at 8:51

2 Answers
2

then the compiler can automatically create an implicit Function object from this definition for you, and will insert it where an implicit of type Int => String is expected. This is what happens in your line HList.runAny(HNil())(null).

But when you define implicit defs which themselves accept implicit parameters (like your runAll method), it doesn't work any more, as the compiler cannot create a Function object whose apply method would require an implicit — much less guarantee that such an implicit would be available at the call site.

The solution to this is to define something like this instead of runAll:

This definition is a bit more explicit, as the compiler now won't need to try to create a Function object from your def, but will instead call your def directly to get the needed function object. And, while calling it, will automatically insert the needed implicit parameter, which it can resolve right away.

In my opinion, whenever you expect implicit functions of this type, you should provide an implicit def that does indeed return a Function object. (Other users may disagree… anyone?) The fact that the compiler is able to create Function wrappers around an implicit def is there mainly, I suppose, to support implicit conversions with a more natural syntax, e.g. using view bounds together with simple implicit defs like my first Int to String conversion.

First off, thank you so much for taking the time to help me! So, your answer fully/directly explains why my original attempt (#1) didn't work, and why hoisting the implicit argument to the front did (#2). However, the other two implementations, #3 and #4, while not being direct replacements for the original code, should have this same problem. Why do these work, given this explanation? In #4, I have removed the context argument, but otherwise have the same situation: an implicit def with an implicit parameter; the removed argument was of a boring type (Object)... why does this help?
–
Jay Freeman -saurik-May 30 '11 at 21:09

Yes: I could tell that specifying my implicit defs in that manner worked better, but I was hoping to actually learn how Scala's implicit parameter binding mechanism worked, so I'd be able to better predict its abilities in future situations that might not quite look like this one (the same as I've come to understand, and even internalize, Koenig lookup from C++). So far, it seems we are still left with "seems like a complex example that stresses the black magic we don't understand well", and have not actually ruled out "this could be a compiler bug demonstrated by these complex examples" :(.
–
Jay Freeman -saurik-May 30 '11 at 22:33

Ok, I've been sitting around comparing horrendous compiler output traces dumped with -Ytyper-debug, and this explanation is incorrect: the compiler, in fact, is able to (and does) bind implicit arguments, even for this specific non-working case, on an implicit def that occur after the other arguments. What isn't working, however, is that the type of Output is being detected as Nothing in this case, so when it attempts to bind the implicit argument for runAll itself, it does not believe that runNil is a compatible match. I still haven't determined exactly why this happens, though.
–
Jay Freeman -saurik-May 31 '11 at 1:06

So, based on the output from -Ytyper-debug I thought I might try adding a slightly more specific type bound on runAll (adding >:HList), and I managed to "fix" what was being detected by the type inferencer: implicit def runAll[Input <:HList, Output >:HList <:HList]. However, this now seems entirely backwards to me as Output in this context is supposed to be HNil, which is a subtype of HList, but here I've gone ahead and asserted that Output is a supertype (in addition to being a subtype, so really: equivalent) to HList, which should exclude HNil from being accepted here. :(
–
Jay Freeman -saurik-May 31 '11 at 2:03

(Note: this is a summary of the discussion that took place in possibly more detail in the comments section of another answer on this question.)

It turns out that the problem here is that the implicit parameter is not first in runAny, but not because the implicit binding mechanism is ignoring it: instead, the issue is that the type parameter Output is not bound to anything, and needs to be indirectly inferred from the type of the run implicit parameter, which is happening "too late".

In essence, the code for "undetermined type parameters" (which is what Output is in this circumstance) only gets used in situations where the method in question is considered to be "implicit", which is determined by its direct parameter list: in this case, runAny's parameter list is actually just(input :Input), and isn't "implicit".

So, the type parameter for Input manages to work (getting set to Int::HNil), but Output is simply set to Nothing, which "sticks" and causes the type of the run argument to be Int::HNil=>Object=>Nothing, which is not satisfiable by runNil, causing runAny's type inferencing to fail, disqualifying it for usage as an implicit argument to runAll.

By reorganizing the parameters as done in my modified code sample #2, we make runAny itself be "implicit", allowing it to first get its type parameters fully determined before applying its remaining arguments: this happens because its implicit argument will first get bound to runNil (or runAny again for more than two levels), whose return type will get taken/bound.

To tie up the loose ends: the reason that the code sample #3 worked in this situation is that the Output parameter wasn't even required: the fact that it was bound to Nothing didn't affect any subsequent attempts to bind it to anything or use it for anything, and runNil was easily chosen to bind to its version of the run implicit parameter.

Finally, code sample #4 was actually degenerate, and shouldn't even have been considered to "work" (I had only verified that it compiled, not that it generated the appropriate output): the data types of its implicit parameters were so simplistic (Input=>Output, where Input and Output were actually intended to be the same type) that it would simply get bound to conforms:<:<[Input,Output]: a function that in turn acted as the identity in this circumstance.