I've been reading all the topics with those "radical" ideas about
the GC and dtors and, honestly, i'd rather call them "insane".
After all the reading and thinking, i came to conclusion, that
what Andrey suggests is to call dtors only on stack-allocated
structs. That also implies, that one can't put those in
containers and gc-allocated objects.
Since all of them: containers, structs, classes -- are all
first-class objects they must be all nicely combined in code,
without any unintuitive stuff.
I mean, really, let us look at c++ strings. There are const
char*, std::sting, QString, Poco::String, icu::UnicodeString, and
every big project uses it's own strings implementation that can't
be used together in a sane way. That is what great in D: you just
threw the idea of library-implemented strings away and made it
not only "standard", but special and that what makes it intuitive
and simple (unless you interact with c++). Never seen any
non-standart strings for D.
The point is, that every library and every coder add to the
project incompatible and difficult to use together stuff, so
every programmer already has to think of all the problems with
other's people code, and you just can't add same shit to the
language. All elements must nicely interact together and
otherwise it's a total disaster.
Back to the dtors: i understand, that all the stuff you propose
could make GC faster, simpler, and cooler, but it sounds insane
to anyone, who uses the language, not develops it, that if you
create struct object, dror will be called, but if you place the
copy in a container, it wont. It's just unanderstandable from
user's point of view.
Now, for the solution.
First, we can just fix this shit with arrays of structs and
that's it. That still lives us with false pointers problem: not
everything gets collected. That's no good. So, i propose to think
of actually separating gc-memory management (via GC) and other
resources management: via some new (or maybe old) mechanism.
Let me start with listing of existing solutions:
1) C.
That is the simplest way: fully-manual resource management.
It's obvious, we can't do that in D, because it's supposed to be
simpler for coding, than C.
2) Go.
Actually, this one is not that different: it uses GC for memory
only, and manual management for all the rest (with help of defer
operator). We can't do it either, for the same reasons.
3) C++.
This one is semi-automatic (talking about user code, not some
allocator libraries): you choose the scheme (refcounting, unique
reference) and then it'll do the rest.
4) Rust.
I'm not a pro here, but as i understand, it uses C++ way, and
adds gc-collected pointers, but not sure, so help me here.
5) Python.
GC-only, except one clever case: with statement calls close()
method.
Please, if there are any pros in other platforms, add your
knowledge to this list, i would very much love to learn (same, if
a made any mistakes).
Now, for D: obviously D has GC-managed heap. First, we should,
like in Go, leave only managing gc-memory to the GC -- this is
just rephrasing Andreys proposal.
The simplest way o manage all other resources would be manual,
Go-way:
A a = A();
scope(exit)
a.~A();
But it's to annoying, to that all the time, so we really want
dtors to save us lost of typing and debugging, but they can't be
called all the time, because we can put stuff in GC-collected
objects.
What i propose, is to include new concept in D: scoped objects.
Any object (no matter is it a class or struct instance) can be
either scoped or not.
Dtors for scoped objects are called when out of scope, dtors for
non-scoped objects are not called at all.
It is actually as simple as rewrite code
A a = A();
as
A a = A();
scope(exit)
a.~A();
For all a's, which are scoped objects.
For me, it is both a simple concept and good rationalization for
difficult dror-gets-called-or-not rules.
That leaves only to determine, what objects are scoped. Well,
that is obviously stack-allocated structs, gc-allocated scope
classes and gc-allocated structs in scope classes.
But that is just my idea. This post has so many words, because
it's very important, that D devs make good decision on that deep
problem, and the key to such decision is information and
discussion.
UPD:
Also, about arrays and slices: if we could easily pass them
around as cost ref-s, just like in C++, then we could make them
value-types and they wouldn't require any ref counting. I would
suggest, make all "in" function arguments const refs.

I've been reading all the topics with those "radical" ideas
about the GC and dtors and, honestly, i'd rather call them
"insane". After all the reading and thinking, i came to
conclusion, that what Andrey suggests is to call dtors only on
stack-allocated structs. That also implies, that one can't put
those in containers and gc-allocated objects.
Since all of them: containers, structs, classes -- are all
first-class objects they must be all nicely combined in code,
without any unintuitive stuff.
I mean, really, let us look at c++ strings. There are const
char*, std::sting, QString, Poco::String, icu::UnicodeString,
and every big project uses it's own strings implementation that
can't be used together in a sane way. That is what great in D:
you just threw the idea of library-implemented strings away and
made it not only "standard", but special and that what makes it
intuitive and simple (unless you interact with c++). Never seen
any non-standart strings for D.
The point is, that every library and every coder add to the
project incompatible and difficult to use together stuff, so
every programmer already has to think of all the problems with
other's people code, and you just can't add same shit to the
language. All elements must nicely interact together and
otherwise it's a total disaster.
Back to the dtors: i understand, that all the stuff you propose
could make GC faster, simpler, and cooler, but it sounds insane
to anyone, who uses the language, not develops it, that if you
create struct object, dror will be called, but if you place the
copy in a container, it wont. It's just unanderstandable from
user's point of view.

+1 about dtors.

Now, for the solution.
First, we can just fix this shit with arrays of structs and
that's it. That still lives us with false pointers problem: not
everything gets collected. That's no good. So, i propose to
think of actually separating gc-memory management (via GC) and
other resources management: via some new (or maybe old)
mechanism.
Let me start with listing of existing solutions:
1) C.
That is the simplest way: fully-manual resource management.
It's obvious, we can't do that in D, because it's supposed to
be simpler for coding, than C.
2) Go.
Actually, this one is not that different: it uses GC for memory
only, and manual management for all the rest (with help of
defer operator). We can't do it either, for the same reasons.
3) C++.
This one is semi-automatic (talking about user code, not some
allocator libraries): you choose the scheme (refcounting,
unique reference) and then it'll do the rest.
4) Rust.
I'm not a pro here, but as i understand, it uses C++ way, and
adds gc-collected pointers, but not sure, so help me here.
5) Python.
GC-only, except one clever case: with statement calls close()
method.
Please, if there are any pros in other platforms, add your
knowledge to this list, i would very much love to learn (same,
if a made any mistakes).
Now, for D: obviously D has GC-managed heap. First, we should,
like in Go, leave only managing gc-memory to the GC -- this is
just rephrasing Andreys proposal.
The simplest way o manage all other resources would be manual,
Go-way:
A a = A();
scope(exit)
a.~A();
But it's to annoying, to that all the time, so we really want
dtors to save us lost of typing and debugging, but they can't
be called all the time, because we can put stuff in
GC-collected objects.
What i propose, is to include new concept in D: scoped objects.
Any object (no matter is it a class or struct instance) can be
either scoped or not.
Dtors for scoped objects are called when out of scope, dtors
for non-scoped objects are not called at all.
It is actually as simple as rewrite code
A a = A();
as
A a = A();
scope(exit)
a.~A();
For all a's, which are scoped objects.
For me, it is both a simple concept and good rationalization
for difficult dror-gets-called-or-not rules.
That leaves only to determine, what objects are scoped. Well,
that is obviously stack-allocated structs, gc-allocated scope
classes and gc-allocated structs in scope classes.
But that is just my idea. This post has so many words, because
it's very important, that D devs make good decision on that
deep problem, and the key to such decision is information and
discussion.
UPD:
Also, about arrays and slices: if we could easily pass them
around as cost ref-s, just like in C++, then we could make them
value-types and they wouldn't require any ref counting. I would
suggest, make all "in" function arguments const refs.

I like the idea of scoping dtor's. But I still want the ability
to say:
Hey I have this global variable, if I assign a value to it and
later null it, it'll call its destructor if its not referenced
anywhere else.
Which in turn would make me think ref counting would be a good
idea.
But either way, I think we are getting ahead of ourselves with
all these 'major' proposed changes. I think its time to step back
and say hey D2 shouldn't be changed much more lets freeze it.
Now lets plan for D3.
Breakage between D2 and D3 is acceptable but not in D2 to the
extent some of these proposals is bringing to the table.

Hey I have this global variable, if I assign a value to it and
later null it, it'll call its destructor if its not referenced
anywhere else.
Which in turn would make me think ref counting would be a good
idea.

It seems, that ARC is the only way. There were idea to make all
non-scoped (in my terminology) objects ARC-d, if they have dtors.
That makes sense to me.

Hey I have this global variable, if I assign a value to it and later
null it, it'll call its destructor if its not referenced anywhere else.
Which in turn would make me think ref counting would be a good idea.

It seems, that ARC is the only way. There were idea to make all
non-scoped (in my terminology) objects ARC-d, if they have dtors. That
makes sense to me.

Interesting, we haven't explored that. The most problematic implication
would be that classes with destructors will form a hierarchy separate
from Object.
Andrei

Hey I have this global variable, if I assign a value to it
and later
null it, it'll call its destructor if its not referenced
anywhere else.
Which in turn would make me think ref counting would be a
good idea.

It seems, that ARC is the only way. There were idea to make all
non-scoped (in my terminology) objects ARC-d, if they have
dtors. That
makes sense to me.

Interesting, we haven't explored that. The most problematic
implication would be that classes with destructors will form a
hierarchy separate from Object.
Andrei

Yeah, that's a good point: How do you define a "class
with/without destructor", when they all derive from Object
anyways.
Necessarily, Object needs to have a destructor (even if it does
nothing) for any other sub-class to have them. Or vice versa, if
Object has no destructor, no subclass can have them?
So is it even possible to separate classes into two different
groups?

The most problematic implication would be that classes with
destructors will form a hierarchy separate from Object.

What for? As i understand object's dtor does nothing, so for any
class we can determine, if dtor is empty. I don't see a problem
here.
Cycles and locks for RC are the biggest problems, if you ask me.
Also, what about arrays and maps?

Interesting, we haven't explored that. The most problematic
implication would be that classes with destructors will form a
hierarchy separate from Object.

As i understood, you want to remove dtors for non-scoped objects
completely, so all classes will be without it, except user
defined ones.
Can we mark all dtors virtual by default, so there wouldn't need
to be dtor in object?

Interesting, we haven't explored that. The most problematic implication
would be that classes with destructors will form a hierarchy separate
from Object.

Seems like people have been ignoring my two posts in the thread
"radical ideas about gc and reference counting". I've been proposing
exactly that, and there's a way if you don't want a separate class
hierarchy.
--
Michel Fortin
michel.fortin michelf.ca
http://michelf.ca

Scoped-objects + ARC on non-scoped objects with dtors + GC on
non-scoped objects w/o dtors would arguably solve the problem,
especially, is arrays of scoped objects would be considered also
scoped, or just add separate scoped arrays.

I've been reading all the topics with those "radical" ideas about the GC
and dtors and, honestly, i'd rather call them "insane". After all the
reading and thinking, i came to conclusion, that what Andrey suggests is
to call dtors only on stack-allocated structs. That also implies, that
one can't put those in containers and gc-allocated objects.
Since all of them: containers, structs, classes -- are all first-class
objects they must be all nicely combined in code, without any
unintuitive stuff.
I mean, really, let us look at c++ strings. There are const char*,
std::sting, QString, Poco::String, icu::UnicodeString, and every big
project uses it's own strings implementation that can't be used together
in a sane way. That is what great in D: you just threw the idea of
library-implemented strings away and made it not only "standard", but
special and that what makes it intuitive and simple (unless you interact
with c++). Never seen any non-standart strings for D.
The point is, that every library and every coder add to the project
incompatible and difficult to use together stuff, so every programmer
already has to think of all the problems with other's people code, and
you just can't add same shit to the language. All elements must nicely
interact together and otherwise it's a total disaster.
Back to the dtors: i understand, that all the stuff you propose could
make GC faster, simpler, and cooler, but it sounds insane to anyone, who
uses the language, not develops it, that if you create struct object,
dror will be called, but if you place the copy in a container, it wont.
It's just unanderstandable from user's point of view.
Now, for the solution.
First, we can just fix this shit with arrays of structs and that's it.
That still lives us with false pointers problem: not everything gets
collected. That's no good. So, i propose to think of actually separating
gc-memory management (via GC) and other resources management: via some
new (or maybe old) mechanism.
Let me start with listing of existing solutions:
1) C.
That is the simplest way: fully-manual resource management.
It's obvious, we can't do that in D, because it's supposed to be simpler
for coding, than C.
2) Go.
Actually, this one is not that different: it uses GC for memory only,
and manual management for all the rest (with help of defer operator). We
can't do it either, for the same reasons.
3) C++.
This one is semi-automatic (talking about user code, not some allocator
libraries): you choose the scheme (refcounting, unique reference) and
then it'll do the rest.
4) Rust.
I'm not a pro here, but as i understand, it uses C++ way, and adds
gc-collected pointers, but not sure, so help me here.
5) Python.
GC-only, except one clever case: with statement calls close() method.
Please, if there are any pros in other platforms, add your knowledge to
this list, i would very much love to learn (same, if a made any mistakes).
Now, for D: obviously D has GC-managed heap. First, we should, like in
Go, leave only managing gc-memory to the GC -- this is just rephrasing
Andreys proposal.
The simplest way o manage all other resources would be manual, Go-way:
A a = A();
scope(exit)
a.~A();
But it's to annoying, to that all the time, so we really want dtors to
save us lost of typing and debugging, but they can't be called all the
time, because we can put stuff in GC-collected objects.
What i propose, is to include new concept in D: scoped objects.
Any object (no matter is it a class or struct instance) can be either
scoped or not.
Dtors for scoped objects are called when out of scope, dtors for
non-scoped objects are not called at all.
It is actually as simple as rewrite code
A a = A();
as
A a = A();
scope(exit)
a.~A();
For all a's, which are scoped objects.
For me, it is both a simple concept and good rationalization for
difficult dror-gets-called-or-not rules.
That leaves only to determine, what objects are scoped. Well, that is
obviously stack-allocated structs, gc-allocated scope classes and
gc-allocated structs in scope classes.
But that is just my idea. This post has so many words, because it's very
important, that D devs make good decision on that deep problem, and the
key to such decision is information and discussion.
UPD:
Also, about arrays and slices: if we could easily pass them around as
cost ref-s, just like in C++, then we could make them value-types and
they wouldn't require any ref counting. I would suggest, make all "in"
function arguments const refs.

C# and Java also have scoped blocks (using, try-with-resources), similar
to Python for resource management.
Ada has controlled types for it, where Finalize() plays the role of C++
destructor.
Additionally all languages with lambda support do offer resource
management via implicit control structures. Mostly visible in Lisp and
ML dialects. Great in the languages that allow the lambda as last
parameter to appear outside of the call.
For example, assume a doTransaction (connection, lambda) function, then
doTransaction(connection) {
db.insert (my data)
}
The approach taken is similar to implementing C++ templates for generic
RAII use cases.
--
Paulo

C# and Java also have scoped blocks (using,
try-with-resources), similar to Python for resource management.

Yeah, but it doesn't work for graphs that maintain resources,
such as a scene graph which hold onto texture memory.
Unfortunately, you don't want GC collection to release it either.
I think in most cases resources can either be handled by owned
pointers or a regional GC (in this case a GC that only trace
SceneNodePointers).

C# and Java also have scoped blocks (using, try-with-resources),
similar to Python for resource management.

Yeah, but it doesn't work for graphs that maintain resources, such as a
scene graph which hold onto texture memory. Unfortunately, you don't
want GC collection to release it either.
I think in most cases resources can either be handled by owned pointers
or a regional GC (in this case a GC that only trace SceneNodePointers).

- Make all scene graph nodes IDisposable.
- Have a using(rootNode) {} on your render loop

That would work for a static scene.
But you want to mark resources ready for release when nodes are
removed from the graph dynamically. In most cases shared pointers
(ref counting) would work, but if you allow "fractal" recursion
then you will get cycles. (e.g.
meshnode->scaleRND->rotateRND->stopIfTooSmall->meshnode )

- Make all scene graph nodes IDisposable.
- Have a using(rootNode) {} on your render loop

That would work for a static scene.
But you want to mark resources ready for release when nodes are removed
from the graph dynamically. In most cases shared pointers (ref counting)
would work, but if you allow "fractal" recursion then you will get
cycles. (e.g. meshnode->scaleRND->rotateRND->stopIfTooSmall->meshnode )

Easy, you already know that you don't need the node, just call the
dispose method, instead of marking for release.
Cannot release right away? Place them in a "to be removed" queue and do
the cleaning in a specific controlled point.
--
Paulo

Easy, you already know that you don't need the node, just call
the dispose method, instead of marking for release.

But you don't know unless you use RC or GC. Let's say you make a
jungle. Lots of pointers to the tree root node. And you also
don't want to wait with collection if you hold onto hardware
resources (forcing perhaps lower resolution graphics if you are
low on GPU resources).

Easy, you already know that you don't need the node, just call the
dispose method, instead of marking for release.

But you don't know unless you use RC or GC.

I am always speaking from the point of view of automatic memory
management, be it RC, GC or compiler aided dataflow analysis.

Let's say you make a jungle.
Lots of pointers to the tree root node. And you also don't want to wait
with collection if you hold onto hardware resources (forcing perhaps
lower resolution graphics if you are low on GPU resources).

If you mean you need to be sure that all references are gone from the
graph, before doing a dispose() invocation, then yeah you need some form
of RC, even with a GC.
However that doesn't mean that the developer/API client needs to care
about it.
It is a matter of how the graph node construction is exposed to the
clients. With proper a proper ADT that can be hidden from the API client.
Automatic resource management requires another type of thinking anyway.
--
Paulo

Back to the dtors: i understand, that all the stuff you propose
could make GC faster, simpler, and cooler,

Note that this is _not_ the motivation behind the discussed
changes. It's primarily about correctness and consistency. What
we currently have if something that doesn't work at all in some
cases (dynamic arrays, new with structs), and where it does work
(class destructors) users have to conform to various rules as to
what is allowed in a GC-called destructor (e.g. don't access
references to other objects, don't throw (?), don't use
thread-local globals, any probably many more subtle rules), that
mostly cannot be enforced statically or even at runtime,
potentially causing silent corruption if you fail to comply with
them. This is IMO more than enough motivation to do something
about it.

but it sounds insane to anyone, who uses the language, not
develops it, that if you create struct object, dror will be
called, but if you place the copy in a container, it wont. It's
just unanderstandable from user's point of view.

Not in a container, but making it GC managed. And IMO, it's
unreasonable to expect deterministic behaviour from a tracing
(non-precise) GC, which is inherently non-deterministic. The real
problem is that you can currently do these things accidentally,
without noticing it.

1) C.
That is the simplest way: fully-manual resource management.
It's obvious, we can't do that in D, because it's supposed to
be simpler for coding, than C.

Again, the main motivation for GC is safety and enabling more
idioms that would otherwise be impossible, _not_ lazyness on the
part of the programmer.

3) C++.
This one is semi-automatic (talking about user code, not some
allocator libraries): you choose the scheme (refcounting,
unique reference) and then it'll do the rest.
4) Rust.
I'm not a pro here, but as i understand, it uses C++ way, and
adds gc-collected pointers, but not sure, so help me here.

You could describe it like that, yes. But it's a lot more
involved. In a way, Rust's owned types are the opposite of a GC:
A GC destroys objects lazily and keeps them alive as long as they
are referenced. Owned objects are destroyed eagerly at the end of
their scope, but the type system ensures that there are no
references to them at this point.

What i propose, is to include new concept in D: scoped objects.
Any object (no matter is it a class or struct instance) can be
either scoped or not.
Dtors for scoped objects are called when out of scope, dtors
for non-scoped objects are not called at all.
It is actually as simple as rewrite code
A a = A();
as
A a = A();
scope(exit)
a.~A();

There is std.typecons.scoped for that, and the deprecated scope
classes. Also see deadalnix's `isolated` proposal:
http://forum.dlang.org/thread/yiwcgyfzfbkzcavuqdwz forum.dlang.org
But I'm afraid your suggestion is unsafe: There also needs to be
a way to guarantee that no references to the scoped object exist
when it is destroyed.
I also don't think it's a good idea to have a hard-coded list of
types that are scoped. This either needs to be specified by the
user of the type (at instantion) and then become part of the type
(like const), or it needs to be specified by the implementer of
the type, as a requirement that it can only be instantiated as a
scoped type, or as garbage-collected or reference-counted or ...

Back to the dtors: i understand, that all the stuff you
propose could make GC faster, simpler, and cooler,

Note that this is _not_ the motivation behind the discussed
changes. It's primarily about correctness and consistency. What
we currently have if something that doesn't work at all in some
cases (dynamic arrays, new with structs), and where it does
work (class destructors) users have to conform to various rules
as to what is allowed in a GC-called destructor (e.g. don't
access references to other objects, don't throw (?), don't use
thread-local globals, any probably many more subtle rules),
that mostly cannot be enforced statically or even at runtime,
potentially causing silent corruption if you fail to comply
with them. This is IMO more than enough motivation to do
something about it.

But I'm afraid your suggestion is unsafe: There also needs to be a way
to guarantee that no references to the scoped object exist when it is
destroyed.

Actually, it should be fine to call the destructor, then blast T.init
over the object, while keeping the actual memory in the GC. This
possible approach has come up a number of times, and I think it has
promise. -- Andrei

But I'm afraid your suggestion is unsafe: There also needs to
be a way
to guarantee that no references to the scoped object exist
when it is
destroyed.

Actually, it should be fine to call the destructor, then blast
T.init over the object, while keeping the actual memory in the
GC. This possible approach has come up a number of times, and I
think it has promise. -- Andrei

Then accesses at runtime would still appear to work, but you're
actually accessing something else than you believe you do. IMO,
this is almost as bad as silent heap corruption. Such code should
just be rejected at compile-time, if at all possible.

But I'm afraid your suggestion is unsafe: There also needs to be a way
to guarantee that no references to the scoped object exist when it is
destroyed.

Actually, it should be fine to call the destructor, then blast T.init
over the object, while keeping the actual memory in the GC. This
possible approach has come up a number of times, and I think it has
promise. -- Andrei

Then accesses at runtime would still appear to work, but you're actually
accessing something else than you believe you do. IMO, this is almost as
bad as silent heap corruption.

Not as bad because memory safety is preserved and the errors are
reproducible.

Such code should just be rejected at
compile-time, if at all possible.

Very short feedback about original proposal:
1) managing local objects is not really a problem, we already
have `scoped` in Phobos for that (and unimplemented scope
qualifier as possible more reliable approach)
2) real problem is managing global objects without clear
destruction point while still keeping those compatible with
Object hierarchy (for inheritance / polymorphism). There is
nothing proposed to address it.