Consider the following code. What will it print?
auto arr=new ulong delegate()[5];
foreach(i;0..arr.length){
arr[i]=()=>i;
}
writeln(arr.map!`a()`());
It is natural to expect that it will print [0, 1, 2, 3, 4], but
it actually prints [5, 5, 5, 5, 5]. The reason is obvious - all
the delegates refer to the same `i` in their closures, and at the
end of the loop that `i` is set to `5`.
While this makes some sense when you consider how the closures
are implemented, it is still not what you expect from that code.
`i` is supposed to be local to each iteration of the loop, and
here it "leaks" between the iterations.
There is an hackaround:
auto arr=new ulong delegate()[5];
foreach(i;0..arr.length){
arr[i]=(j=>()=>j)(i);
}
writeln(arr.map!`a()`());
This prints [0, 1, 2, 3, 4] as expected, since we use another
function to create a separate stack frame to store the value of
`i`. A different way to implement that idea:
auto arr=new ulong delegate()[5];
foreach(i;0..arr.length){
arr[i]={
auto j=i;
return ()=>j;
}();
}
writeln(arr.map!`a()`());
Can this be fixed? *Should* this be fixed?
Discuss

For what it's worth, Javascript works the same way. So I've come
to the pattern of having a function return a function when it is
looping in both languages.
So while it might not be ideal, it isn't totally unexpected since
other languages do it too.

For what it's worth, Javascript works the same way. So I've
come to the pattern of having a function return a function when
it is looping in both languages.
So while it might not be ideal, it isn't totally unexpected
since other languages do it too.

Ruby is the most interesting:
arr=[]
for i in 0...5
arr<< lambda{i}
end
puts arr.map{|e|e.call}.to_s
prints [4, 4, 4, 4, 4], because it uses the built-in for loop, so
`i` is reused in all iterations. On the other hand:
arr=[]
(0...5).each do|i|
arr<< lambda{i}
end
puts arr.map{|e|e.call}.to_s
prints [0, 1, 2, 3, 4], because each call to the rubyblock has
it's own stack frame and it's own `i`. This can also be done in D:
auto arr=new ulong delegate()[5];
auto iotaDelegate(T...)(T args){
return (int delegate(ref ulong) dlg){
foreach(i;iota(args)){
dlg(i);
}
return 0;
};
}
foreach(i;iotaDelegate(arr.length)){
arr[i]=()=>i;
}
writeln(arr.map!`a()`());
This prints [0, 1, 2, 3, 4], because each iteration of the
foreach is actually an function call, so it has it's own stack
frame.

Huh, this is disappointing. I haven't used closures enough in D
to notice this issue, but I know of this kind of issue very well,
because it is a common annoyance in JavaScript. Newer,
non-cross-browsers versions of JavaScript get around this with
'let' instead of 'var,' which introduces block scope. So
something like this works as expected:
let arr = Array(5);
for (let i = 0; i < 5; ++i) {
arr[i] = function() { return i; }; // The scoped i is used
here.
}
let anotherArr = Array(5);
for (let i = 0; i < 5; ++i) {
anotherArr[i] = arr[i](); // Return a different i each time.
}
console.log(anotherArr); // 0, 1, 2, 3, 4, as expected
Is it possible for similar semantics to exist in D by default? I
imagine this is harder to implement.

Consider the following code. What will it print?
auto arr=new ulong delegate()[5];
foreach(i;0..arr.length){
arr[i]=()=>i;
}
writeln(arr.map!`a()`());
It is natural to expect that it will print [0, 1, 2, 3, 4], but it
actually prints [5, 5, 5, 5, 5]. The reason is obvious - all the
delegates refer to the same `i` in their closures, and at the end of the
loop that `i` is set to `5`.
...

It is not that obvious. They refer to different i's that happen to
reside at the same place in the stack frame. It's a bug.
It is more obvious that it is a bug given this code snippet:
import std.stdio, std.algorithm;
void main(){
auto arr=new ulong delegate()[5];
foreach(immutable i;0..arr.length){
arr[i]={auto j=i;return {assert(j==i); return i;};}();
}
writeln(arr.map!`a()`());
}
As you can see, 'i' mutates even though it is declared immutable.

Consider the following code. What will it print?
auto arr=new ulong delegate()[5];
foreach(i;0..arr.length){
arr[i]=()=>i;
}
writeln(arr.map!`a()`());
It is natural to expect that it will print [0, 1, 2, 3, 4], but it
actually prints [5, 5, 5, 5, 5]. The reason is obvious - all the
delegates refer to the same `i` in their closures, and at the end
of the loop that `i` is set to `5`.
...

It is not that obvious. They refer to different i's that happen to
reside at the same place in the stack frame. It's a bug.
It is more obvious that it is a bug given this code snippet:
import std.stdio, std.algorithm;
void main(){
auto arr=new ulong delegate()[5];
foreach(immutable i;0..arr.length){
arr[i]={auto j=i;return {assert(j==i); return i;};}();
}
writeln(arr.map!`a()`());
}
As you can see, 'i' mutates even though it is declared immutable.

Keep in mind that this exhibits the same "mutating immutable" behavior:
// Prints 0, 1, 2, 3, 4, even though i is immutable
foreach(immutable i; 0..5){
writeln(i);
It's questionable as to whether that's really a problem. And even if it
is a problem, it would *only* be because you make i immutable. So
this is irrelevant to the OP's examples because:
1. He didn't use immutable, and
2. A delegate is *expected* to read the values inside its closure at
the time of delegate *invocation*, not at the time of delegate creation.

That refers to a local defined within the loop, not the iteration
variable itself, which is different from the OP. Also, it's
questionable that there's any problem there *other* than the immutable
stuff.

Consider the following code. What will it print?
auto arr=new ulong delegate()[5];
foreach(i;0..arr.length){
arr[i]=()=>i;
}
writeln(arr.map!`a()`());
It is natural to expect that it will print [0, 1, 2, 3, 4], but it
actually prints [5, 5, 5, 5, 5]. The reason is obvious - all the
delegates refer to the same `i` in their closures, and at the end
of the loop that `i` is set to `5`.
...

It is not that obvious. They refer to different i's that happen to
reside at the same place in the stack frame. It's a bug.
It is more obvious that it is a bug given this code snippet:
import std.stdio, std.algorithm;
void main(){
auto arr=new ulong delegate()[5];
foreach(immutable i;0..arr.length){
arr[i]={auto j=i;return {assert(j==i); return i;};}();
}
writeln(arr.map!`a()`());
}
As you can see, 'i' mutates even though it is declared immutable.

Keep in mind that this exhibits the same "mutating immutable" behavior:
// Prints 0, 1, 2, 3, 4, even though i is immutable
foreach(immutable i; 0..5){
writeln(i);

No, it does not! My code contains immutable variables j and i (in fact,
5 pairs of these), where with DMD's buggy behaviour it appears that at
one point in time j==i and at another point in time j!=i.
Your code contains five distinct immutable variables i with different
values.

It's questionable as to whether that's really a problem. And even if it
is a problem, it would *only* be because you make i immutable. So
this is irrelevant to the OP's examples because:
1. He didn't use immutable, and
...

No, this point is irrelevant. It is a bug in any case. If immutable is
used, this bug can be exploited to break the const system. Hence, I was
using immutable to make my point more clear. Apparently it had the
converse effect.

2. A delegate is *expected* to read the values inside its closure at
the time of delegate *invocation*, not at the time of delegate creation.

Obviously. What you seem to miss is that there is no aliasing between
any of the delegate closures in the above code.
Note that I understand exactly what you think is happening.

That refers to a local defined within the loop, not the iteration
variable itself, which is different from the OP.

No, it is not, as I have explained before. Since 2.063, the loop
variable that is exposed from the foreach loop behaves like a for-loop
loop body-local variable.

Also, it's
questionable that there's any problem there *other* than the immutable
stuff.

Obviously there is. The only reason why the code appears to behave as
you assume it does is because the loop-local variables happen to be
allocated at the same place for each iteration. It's a memory safety
issue. Allocated memory is allocated again before it is freed.
If that helps, other C-like languages with closure support (eg. C#)
correctly allocate a closure context per loop iteration.

Yea, see my other recent responses. For some reason I was
(incorrectly) thinking of closures being captured and stored at the end
of the current function call instead of (correctly) at the end of the
current lexical scope.

Consider the following code. What will it print?
auto arr=new ulong delegate()[5];
foreach(i;0..arr.length){
arr[i]=()=>i;
}
writeln(arr.map!`a()`());
It is natural to expect that it will print [0, 1, 2, 3, 4], but
it actually prints [5, 5, 5, 5, 5]. The reason is obvious - all
the delegates refer to the same `i` in their closures, and at the
end of the loop that `i` is set to `5`.

I think the problem is simply a misunderstanding of closures.
Consider this:
int a = 2;
auto dg = () => a;
a = 3;
// This prints 3
writeln(dg());
// Now this prints 5
a = 5;
writeln(dg());
The thing to keep in mind is that closures are *not* evaluated at the
point of creation (otherwise they may as well not be written as a
delegate at all). They're evaluated at the point of invocation. And
that's the whole point: to say "Ok program, I want you to *hold on* to
this set of instructions for now...ignore them right now, but I'll tell
you when I want you to 'open the envelope' and look at it".
In other words, the scope captured by a delegate is *by reference*, not
by value. It's the *same* scope, not a snapshot copy of the scope.
So when you put the delegate creation in a loop:
foreach(a; iota(0..6))
dg = () => a;
It *is* expected that you're *not* sticking 0, 1, 2, 3, etc inside the
delegate. That's because you're not evaluating "a" *at all* here,
you're just crerates a delegate that *refers* to "a" itself. You're
just creating the exact same delegate five times. In other words:
You're just saying:
Store the following set of instructions into 'dg': "Read the value of
'a' and then return it."
You're *not* saying:
Read the value of 'a' and *then* create a delegate that returns that
value.

Consider the following code. What will it print?
auto arr=new ulong delegate()[5];
foreach(i;0..arr.length){
arr[i]=()=>i;
}
writeln(arr.map!`a()`());
It is natural to expect that it will print [0, 1, 2, 3, 4], but
it actually prints [5, 5, 5, 5, 5]. The reason is obvious - all
the delegates refer to the same `i` in their closures, and at the
end of the loop that `i` is set to `5`.

I think the problem is simply a misunderstanding of closures.
...

I believe what you explain is exactly what he suspected is happening.

...
So when you put the delegate creation in a loop:
foreach(a; iota(0..6))
dg = () => a;
It *is* expected that you're *not* sticking 0, 1, 2, 3, etc inside the
delegate. That's because you're not evaluating "a" *at all* here,
you're just crerates a delegate that *refers* to "a" itself. You're
just creating the exact same delegate five times. In other words:
You're just saying:
Store the following set of instructions into 'dg': "Read the value of
'a' and then return it."
You're *not* saying:
Read the value of 'a' and *then* create a delegate that returns that
value.

'a' refers to a different location for every loop iteration. This is a
language change in 2.063.

...
So when you put the delegate creation in a loop:
foreach(a; iota(0..6))
dg = () => a;
It *is* expected that you're *not* sticking 0, 1, 2, 3, etc inside
the delegate. That's because you're not evaluating "a" *at all*
here, you're just crerates a delegate that *refers* to "a" itself.
You're just creating the exact same delegate five times. In other
words:
You're just saying:
Store the following set of instructions into 'dg': "Read the value
of 'a' and then return it."
You're *not* saying:
Read the value of 'a' and *then* create a delegate that returns that
value.

'a' refers to a different location for every loop iteration. This is
a language change in 2.063.

foreach(a;0..5)
writeln(&a);
For me, that prints the same address five times in both 2.062 and
2.063. I would argue that's as it should be.

...
So when you put the delegate creation in a loop:
foreach(a; iota(0..6))
dg = () => a;
It *is* expected that you're *not* sticking 0, 1, 2, 3, etc inside
the delegate. That's because you're not evaluating "a" *at all*
here, you're just crerates a delegate that *refers* to "a" itself.
You're just creating the exact same delegate five times. In other
words:
You're just saying:
Store the following set of instructions into 'dg': "Read the value
of 'a' and then return it."
You're *not* saying:
Read the value of 'a' and *then* create a delegate that returns that
value.

'a' refers to a different location for every loop iteration. This is
a language change in 2.063.

foreach(a;0..5)
writeln(&a);
For me, that prints the same address five times in both 2.062 and
2.063.

It does not matter. The lifetimes of the different a's do not overlap.

I would argue that's as it should be.

That argument would be mistaken. The compiler is free to eg. unroll the
loop completely and use different addresses.
Behind the scenes, the foreach loop is rewritten to something like:
for(int __a=0;__a<5;__a++){
int a = __a;
writeln(&a);
}

'a' refers to a different location for every loop iteration. This
is a language change in 2.063.

foreach(a;0..5)
writeln(&a);
For me, that prints the same address five times in both 2.062 and
2.063.

It does not matter. The lifetimes of the different a's do not overlap.

Ok, I see what you mean now: Closures are expected to capture the
current scope, not just the current function's scope (which wouldn't
have been right anyway as it would have prevented access to the loop
variable).
Ie:
int delegate() dg;
{
int a = 2;
dg = () => a;
}
int a = 100;
writeln(dg());
That prints 2, as expected.
I hadn't thought of that. With that in mind, then yes, the OP's example
should print 0, 1, 2, 3, etc, not all 5's.

Yea, I think the years I spent using C has corrupted my brain into
seeing...
foreach(int a; 0..5)
...as shorthand for:
for(int a=0; a < 5; a++) { /* use a */ }
Which is something I wrote far too many times in the 90's ;)

Yea, I think the years I spent using C has corrupted my brain into
seeing...
foreach(int a; 0..5)
...as shorthand for:
for(int a=0; a < 5; a++) { /* use a */ }
Which is something I wrote far too many times in the 90's ;)

It was exactly that until recently. With the language version
implemented by DMD 2.062, the behaviour in the OP was actually correct.

Consider the following code. What will it print?
auto arr=new ulong delegate()[5];
foreach(i;0..arr.length){
arr[i]=()=>i;
}
writeln(arr.map!`a()`());
It is natural to expect that it will print [0, 1, 2, 3, 4], but
it actually prints [5, 5, 5, 5, 5]. The reason is obvious - all
the delegates refer to the same `i` in their closures, and at the
end of the loop that `i` is set to `5`.

I think the problem is simply a misunderstanding of closures.

FWIW this was discussed at a C# conference. It was a change of behavior
or something. It was a sort of a big deal, and a matter in which
reasonable people were disagreeing.
Andrei

Consider the following code. What will it print?
auto arr=new ulong delegate()[5];
foreach(i;0..arr.length){
arr[i]=()=>i;
}
writeln(arr.map!`a()`());
It is natural to expect that it will print [0, 1, 2, 3, 4], but
it actually prints [5, 5, 5, 5, 5]. The reason is obvious - all
the delegates refer to the same `i` in their closures, and at the
end of the loop that `i` is set to `5`.

I think the problem is simply a misunderstanding of closures.

FWIW this

By 'this' I assume you mean aliasing semantics of a foreach variable?

was discussed at a C# conference. It was a change of behavior
or something. It was a sort of a big deal, and a matter in which
reasonable people were disagreeing.
...

C# used to behave the same as D behaves now, but in the latest update
to C# was changed to properly capture by value.
Closures should capture (and in most functional languages they do) by
value (at least primitive types) to avoid confusion.

but in the latest update to C# was changed to properly capture by value.

Again, no way.
Apparently they indeed did a change for C# 5, but all that happened was
that the foreach iteration variable exposed to the user was moved into
the loop body. This is already the case in D, and yet the behaviour is
still the broken one.
http://d.puremagic.com/issues/show_bug.cgi?id=2043.

Closures should capture (and in most functional languages they do) by value

In which functional languages?

(at least primitive types) to avoid confusion.

Maybe there is a confusion of terminology. What do you mean when you say
capture by value?

Consider the following code. What will it print?
auto arr=new ulong delegate()[5];
foreach(i;0..arr.length){
arr[i]=()=>i;
}
writeln(arr.map!`a()`());
It is natural to expect that it will print [0, 1, 2, 3, 4], but
it actually prints [5, 5, 5, 5, 5]. The reason is obvious - all
the delegates refer to the same `i` in their closures, and at
the end of the loop that `i` is set to `5`.
While this makes some sense when you consider how the closures
are implemented, it is still not what you expect from that
code. `i` is supposed to be local to each iteration of the
loop, and here it "leaks" between the iterations.

The same problem exists in C#.
The issue is rather simple but I'll make it more clear:
The delegates are called AFTER the for loop(since you are storing
them in an array, presumably to use them later). i = 5(or even
undetermined) after the loop, hence the behavior is correct.
It is wrong to use a local variable inside a delegate when the
delegate is called outside the scope of that variable. This
should technically be an error as it is undefined behavior.
I imagine the compiler isn't smart enough to know that you are
storing delegates to be used outside the scope of i.
Basically when the lifetime of a variable has ended all
references to it(such as in a delegate) are invalid. Only
delegate invocation referencing live variables will be valid.

It is wrong to use a local variable inside a delegate when the
delegate is called outside the scope of that variable. This should
technically be an error as it is undefined behavior.

That's not true. According to TDPL, the compiler should detect such
references, and allocate these variables on the heap instead of the
stack, so that the delegate acts like a closure. The variables will be
GC'd once the delegate is no longer used.
This is part of what makes delegates so powerful -- you can use them to
capture the local context of a function and, in effect, access that
context long after the function call has finished.

I imagine the compiler isn't smart enough to know that you are
storing delegates to be used outside the scope of i.

[...]
It should know (and in fact does, at least in the simple cases I tried).
If it doesn't, that's a bug that should be filed in bugzilla.
T
--
PNP = Plug 'N' Pray

I just ran into the problem myself, and it really caught me off
guard.
import std.stdio, core.thread;
void main() {
foreach (i; 0..3) {
immutable ii = i; // Copy for good measure?
writeln("Main: ", ii);
auto t = new Thread({
writeln("Worker Start: ", ii);
Thread.sleep(dur!"msecs"(1));
writeln("Worker End: ", ii);
});
t.start();
}
}
Output:
Main: 0
Main: 1
Worker Start: 1
Main: 2
Worker Start: 2
Worker Start: 2
Worker End: 2
Worker End: 2
Worker End: 2
I caught on that the problem has to do with capturing stack
variables that are going out of scope, but this is resulting a
mutating immutable, without subverting the type system, and
without a complaint from the compiler. (First iteration, 'ii' is
captured while 0. The captured 'ii' is then 1 when the thread
begins, and 2 when it ends.) This seems like a major problem in
the type system, without even being a problem with the type
system.
The following was suggested as a means to capture by value:
On Tuesday, 4 June 2013 at 19:19:57 UTC, Idan Arye wrote:

This indeed appears to work, but... why? From the looks of it,
it is doing the exact same thing. It is capturing a local stack
variable that immediately goes out of scope. Why isn't the extra
stack frame being overwritten each iteration? Also, that
anonymous function really looks like something that the compiler
would inline, so is there even an extra stack frame? I think I
see why Idan called it a 'hackaround'.
I gather two things from this:
1. We really need a legit way to force capture-by-value on a
variable by variable basis. (Unless there's already another way.
Anyone?)
2. Why are immutable local variables not ALWAYS captured by
value? It's not like they are expected to change at all in the
first place. This would at least fix the promise that the type
system makes regarding immutables. (Well... until pointers to
immutables on the stack come into play.)