Author: Richard A. O'Keefe
Status: Accepted
Type: Standards Track
Erlang-Version: R14B04
Content-Type: text/plain
Created: 27-May-2011
Post-History:
****
EEP 37: Funs with names
---
Abstract
========
The syntax of funs is extended to allow a variable name to
be consistently present before each argument list. This
allows funs to be recursive. The knot is tied in the
opcodes that apply a fun to its arguments, so no change to
the garbage collector design is required.
Specification
=============
Currently, there are three forms for a fun:
fun Name/Arity
fun Module:Name/Arity
and
fun Fun_Clause {; Fun_Clause}... end
We add another form:
fun Variable Fun_Clause {; Variable Fun_Clause}... end
If any `Fun_Clause` has a `Variable`, all must, and they must all
be the same variable. Like the variables in the argument list(s),
this variable is local to the fun, not shared with its context.
Within the fun, the variable is bound to the value of the fun
expression.
There are several possible ways to implement this. One is
rather neat because it preserves the cycle-freedom of the
data structures the garbage collector has to deal with.
One way to implement existing funs is this:
- **a** Create an auxiliary function with a generated name
(...,X1,...,Xk) ...;
...
(...,X1,...,Xk) ....
having the same argument lists, guards, and clause bodies
as the fun, except that each variable shared with the context
appears as an extra argument.
- **b** Translate the fun expression as
'%mk-fun'({fun /n+k, X1, ..., Xk})
which gives the tuple a special tag to say that it represents
a fun value.
- **c** Translate `Foo(E1,...,Em)`
as `A1 := E1, ..., Am := Em; funcall_m(Foo)`
where the `funcall_m` instruction checks that its argument is
a closure expecting `m` arguments, moves the `X1,...,Xk` fields
of the tuple to argument registers `A..A`, and then
jumps to the address in the first field.
All it takes to implement recursive funs is
- **a'** Create an auxiliary function
(...,X1,...,Xk,Variable) ...;
...
(...,X1,...,Xk,Variable) ....
- **b'** Translate the fun expression as
'%mk-rec-fun'({fun /, X1, ..., Xk})
which simply applies a second special tag.
- **c'** The `funcall_m` opcode acts the same for both old and
recursive funs, except that just before jumping, it
adds tne fun value `Foo` itself as argument `A`.
This "ties the knot".
So a recursive fun takes no more space or time to create than
an existing one, and does not involve creating any cycles of
pointers. Its code can be inserted into the failure path for
the `funcall_m` instructions, whatever their form.
Motivation
==========
Fun names can serve three purposes.
First, they can simply be documentation. For example,
cfun_files(CFun) ->
fun(F1, F2) ->
[[?OBJ(T1,_) | _] | _] = F1,
[[?OBJ(T2,_) | _] | _] = F2,
CFun(T1, T2)
end.
can be written as
cfun_files(CFun) ->
fun Compare([[?OBJ(T1,_)|_]|_], [[?OBJ(T2,_)|_]|_]) ->
CFun(T1, T2)
end.
A named fun whose name is not used can be implemented as if
the name were not there.
Second, the fun's name can be built into its generated name.
At the time of writing, we might have
'-F/N-fun-K-'
where `F/N` is the name of the function that includes the fun
and `K` is the number of earlier funs in `F/N`. We could build
the name in instead, using
'-F/N-fun-Name-[K-]'
where `K` is present only if the outer function contains more
than one fun with the same name. The point of this is that
such names are more likely to be useful after hot loading.
For example, if we start with
f(...Xs, Ys, ...) ->
...
sort(Xs, fun X_Key({_,N,_}) -> N end),
sort(Ys, fun Y_Key({N,_,_}) -> N end),
...
and then we revise it, swapping the two calls to `sort/2`.
With named funs, the two funs retain their generated names,
and the module is safe. With anonymous functions, the
chances are that the two funs with swap names; oops!
Third, a frequently asked question in the Erlang mailing
list is "why can't I have recursive funs?" to which we
will now be able to rely, "you can; here is what they
look like."
This still does not permit mutually recursive funs, but
people do not seem to ask for that much.
Finally, the next time someone argues that Erlang syntax
is inconsistent because function clauses have repeated
names and fun clauses do not, we shall be able to reply
"but fun clauses CAN have repeated names and probably
should."
Rationale
=========
There really seemed to be only two main questions.
What should the scope of the fun name variable be?
Some variables in a fun are shared between the fun
and its context. Doing that would let us write
f(...) ->
fun G(...) -> ... end,
fun H(...) -> ... end,
... use G and H ...
rather like using nested "define" in Scheme, except that
while `H` could use `G`, `G` couldn't use `H`.
Since you do not get mutual recursion this way, you should
not be tricked into thinking you might. It's better that
you have to write
f(...) ->
GG = fun G(...) -> ... end,
HH = fun H(...) -> ... end,
... use GG and HH ...
so that you understand clearly what you are getting.
While variables in the body of a fun clause may be shared
with the context, variables in the arguments are not,
something I have found confusing. At least this way the
fun name follows the same scope rule as the variables in
the argument list right next to it.
The other main question was whether recursive fun values
should be exactly the same representation as existing
fun values, but with a cycle in it (tying the knot at
construction time), or whether to introduce a new tag
(tying the knot at call time). The lack of cycles in
Erlang heaps has been a major factor in the design of
several garbage collectors. I would expect changing
that to be an order of magnitude harder than the
changes required for this proposal. It was seeing that
the knot could be tied at call with (without slowing
down calls to existing funs) that made me dare to hope
that this proposal might some day be accepted.
The main issue now is that this does not let us define
a group of mutually recursive local functions.
Adopting this proposal now might get in the way of a
better proposal that handles mutual recursion as well.
I don't see such a proposal as being likely to arrive soon.
There is a special case of this where the fun name is used
only in tail call positions, which can be handled entirely
by the compiler generating a jump back to the beginning.
This need not have any consequences for the run time system
at all.
Backwards Compatibility
=======================
Code that does not use the new feature does not change its
meaning. There may be code that relies on the form of
generated function names; that would need changing.
All syntax tools would need to be revised to handle the new form.
Existing parse transforms might well fail on code containing the
new form, but would work unchanged on code that does not.
At least one new instruction is needed to create suitably
distinguished closures. Existing programs that analyse BEAM
files will not understand this until they are revised.
As described under 'motivation', naming functions is
useful even if you do not use the name in any clause body.
This means that we can have a staged delivery of the feature.
1. Make the parser recognise fun names and check their identity.
Have it report an error if the fun name is used in a body.
Have it erase the fun names from the AST before any
downstream tool sees it.
At this stage, fun names may serve as documentation.
2. Upgrade the downstream tools to recognise an extended `'fun'`
AST node with two extra fields: the fun name as an atom and
a flag saying whether it is not used, used only in tail
position, or used more generally.
Upgrade the parser to report fun names, but retain the
check that they are not used. Test the down stream tools.
3. Modify the compiler to use the new, safer, form of generated
name. Ensure that the generated names are accessed only
through an interface, so all is consistent.
At this stage, fun names help to reduce the danger from
code revisions that add, remove, or re-order funs; a
change that does not alter the number of funs with a
particular name in a function should not change its name.
4. (Optional.) Revise the code generator to accept the fun
name in tail call position and generate a jump. Modify
the parser to allow this.
At this point, it is possible to pass a loop as a parameter,
like a list traversal or a binary search. No changes to the
representation of Erlang terms or the BEAM engine have been
required yet.
5. Add a new tag. Revise the funcall instructions to check for
it if the existing check fails, and push the closure itself.
Add a new instruction to make a new closure. Revise the
Erlang term representation to encode recursive funs. Revise
the type test instructions to recognise the new values.
Teach HiPE what to do.
This is the last stage.
Reference Implementation
========================
None in this draft. Stage 1 can be done fairly easily.
Stage 2 would be hard for me because I'm not even sure what
all the relevant modules are.
References
==========
None.
Copyright
=========
This document has been placed in the public domain.
[EmacsVar]: <> "Local Variables:"
[EmacsVar]: <> "mode: indented-text"
[EmacsVar]: <> "indent-tabs-mode: nil"
[EmacsVar]: <> "sentence-end-double-space: t"
[EmacsVar]: <> "fill-column: 70"
[EmacsVar]: <> "coding: utf-8"
[EmacsVar]: <> "End:"