With nothrow and nogc annotations, we've been motivated to add these
annotations to C system API functions, because obviously such functions aren't
going to throw D exceptions or call the D garbage collector.
But this exposed a problem - functions like C's qsort() take a pointer to a
callback function. The callback function, being supplied by the D programmer,
may throw and may call the garbage collector. By requiring the callback
function
to be also nothrow nogc, this is an unreasonable requirement besides breaking
most existing D code that uses qsort().
This problem applies as well to the Windows APIs and the Posix APIs with
callbacks.
The solution is to use overloading so that if your callback is nothrow, it will
call the nothrow version of qsort, if it is throwable, it calls the throwable
version of qsort.
Never mind that those two versions of qsort are actually the same function (!),
even though D's type system regards them as different. Although this looks like
an usafe hack, it actually is quite safe, presuming that the rest of the qsort
code itself does not throw. This technique relies on the fact that extern(C)
functions do not get their types mangled into the names.
Some example code:
extern (C) { alias int function() fp_t; }
extern (C) nothrow { alias int function() fpnothrow_t; }
extern (C) int foo(int a, fp_t fp);
extern (C) nothrow int foo(int a, fpnothrow_t fp);
extern (C) int bar();
extern (C) nothrow int barnothrow();
void test() {
foo(1, &bar); // calls the 'throwing' foo()
foo(1, &barnothrow); // calls the 'nothrow' foo()
}

With nothrow and nogc annotations, we've been motivated to add these
annotations to C system API functions, because obviously such
functions aren't going to throw D exceptions or call the D garbage
collector.
But this exposed a problem - functions like C's qsort() take a pointer
to a callback function. The callback function, being supplied by the D
programmer, may throw and may call the garbage collector. By requiring
the callback function to be also nothrow nogc, this is an
unreasonable requirement besides breaking most existing D code that
uses qsort().
This problem applies as well to the Windows APIs and the Posix APIs
with callbacks.
The solution is to use overloading so that if your callback is
nothrow, it will call the nothrow version of qsort, if it is
throwable, it calls the throwable version of qsort.
Never mind that those two versions of qsort are actually the same
function (!), even though D's type system regards them as different.
Although this looks like an usafe hack, it actually is quite safe,
presuming that the rest of the qsort code itself does not throw. This
technique relies on the fact that extern(C) functions do not get their
types mangled into the names.
Some example code:
extern (C) { alias int function() fp_t; }
extern (C) nothrow { alias int function() fpnothrow_t; }
extern (C) int foo(int a, fp_t fp);
extern (C) nothrow int foo(int a, fpnothrow_t fp);
extern (C) int bar();
extern (C) nothrow int barnothrow();
void test() {
foo(1, &bar); // calls the 'throwing' foo()
foo(1, &barnothrow); // calls the 'nothrow' foo()
}

This is a clever hack to work around the type system, but it introduces
boilerplate, and doesn't ultimately fix the underlying problem. I agree
that it's "good enough" for now -- to get those system APIs working
without massive breakage of existing code -- but I think for the long
run, we should face the root issue: functions that call callbacks have
attributes that *depend* on an input argument.
So we really should have the equivalent of "inout" for other attributes
than const. For example, a function can be "dependently nothrow",
meaning that the body of the function is nothrow, except for that call
to user-supplied delegate:
// This is hypothetical syntax.
int dgCaller(scope int delegate(int) dg inout(nothrow))
inout(nothrow)
{
//throw new Exception(...); // illegal: body of function must not throw
int result = dg(1); // OK: we inherit nothrow-ness from dg().
return result;
}
void f1() nothrow {
// OK: the delegate is nothrow, so dgCaller is nothrow,
// so it's permitted to call it from a nothrow function.
auto x = dgCaller((x) => x+1);
}
void f2() {
// OK: the delegate is throwing, which makes dgCaller
// nothrowing w.r.t. this call. So this function becomes
// throwing.
auto x = dgCaller((int) { throw new Exception(...); });
}
void f3() nothrow {
// ILLEGAL: the delegate throws, so dgCaller may throw,
// so we can't call it from a nothrow function.
auto x = dgCaller((int) { throw new Exception(...); });
}
This capability will solve a lot of attribute-related issues especially
in generic code. For example, a container that implements opApply()
currently cannot be marked pure, because it would impose purity on the
delegate passed to it, which greatly limits its applicability. But
accepting a non-pure delegate makes it uncallable from pure code, even
if opApply() itself doesn't do anything impure. So we end up needing to
write two identical versions of opApply: one pure, and one non-pure. And
this has to be multiplied for each attribute we wish to support (pure,
nothrow, safe, nogc -- that's 2^4 = 16 copies of the same function),
which is clearly untenable.
With inout(nothrow), inout(pure), etc., we can collapse all of these
variants into a single function, and at the same time have the compiler
statically verify that the result does not violate any attribute.
T
--
"I'm not childish; I'm just in touch with the child within!" - RL

+1
I have always wondered why `inout` is limited to const when
problem is almost identical with all other restrictive
attributes.

I think the most common function this kind of thing could be
useful for would be opApply functions. I haven't yet figured out
a good way to make opApply implementations get all of the nice
qualifiers without writing a bunch of overloads. Of course, in my
own code I often just enforce the qualifiers to the exclusion of
code without them, but that's no good for a standard library.

+1
I have always wondered why `inout` is limited to const when problem
is almost identical with all other restrictive attributes.

I think the most common function this kind of thing could be useful
for would be opApply functions. I haven't yet figured out a good way
to make opApply implementations get all of the nice qualifiers without
writing a bunch of overloads. Of course, in my own code I often just
enforce the qualifiers to the exclusion of code without them, but
that's no good for a standard library.

Yes I mention opApply in my post. ;-)
This isn't the only issue with opApply, though. The other issue is that
you can't (easily) control the ref-ness of the loop index.
And another the (lack of) inlining of loop bodies when opApply is
involved. This makes them slightly less attractive in
performance-sensitive situations, which is unfortunate.
T
--
It is impossible to make anything foolproof because fools are so ingenious. --
Sammy

+1
I have always wondered why `inout` is limited to const when problem is
almost identical with all other restrictive attributes.

I have furthermore always wondered why there can always only be one
`inout' wildcard in scope. This is not the best existing way to solve
this kind of problem: Parametric (i.e. not query-able at either runtime
or compile time inside the function) compile-time arguments do it better.
I.e. instead of:
inout(int)[] foo(inout(int)[] arg){ return arg; }
do:
T foo![T <: const(int)[]](T arg){ return arg; }
this can be extended to other attributes, for example in the following
way (this is just an example):
void evaluate![transitive_attributes a](void delegate() a dg) a{
dg();
}
{void foo() safe{}
evaluate(&foo);} // evaluate is safe
{void foo()pure{}
evaluate(&foo);} // evaluate is pure
{void foo()pure safe{}
evaluate(&foo); // evaluate is pure safe
evaluate![pure](&foo);} // argument explicitly specified, system pure

+1
I have always wondered why `inout` is limited to const when problem
is almost identical with all other restrictive attributes.

I have furthermore always wondered why there can always only be one
`inout' wildcard in scope. This is not the best existing way to solve
this kind of problem: Parametric (i.e. not query-able at either
runtime or compile time inside the function) compile-time arguments do
it better.
I.e. instead of:
inout(int)[] foo(inout(int)[] arg){ return arg; }
do:
T foo![T <: const(int)[]](T arg){ return arg; }
this can be extended to other attributes, for example in the following way
(this is just an example):
void evaluate![transitive_attributes a](void delegate() a dg) a{
dg();
}

[...]
What if there are multiple delegate arguments?
void evalTwo(void delegate() a dg1, void delegate() b dg2)
// how to express union of a and b?
{
dg1();
dg2();
}
What if the delegate arguments themselves take delegate arguments?
void eval(void delegate(void delegate() a) b dg1,
void delegate() c dg2) d
// how to express that b depends on a, and
// d depends on b and c?
{
dg1(dg2);
}
Pretty soon, we need an attribute algebra to express these complicated
relationships.
It would be nice to have a solution that can handle all of these cases
without exploding complexity in the syntax.
T
--
"The number you have dialed is imaginary. Please rotate your phone 90
degrees and try again."

Pretty soon, we need an attribute algebra to express these
complicated
relationships.
It would be nice to have a solution that can handle all of
these cases
without exploding complexity in the syntax.

Here's one idea:
Attributes can be thought of as "declarative" compile-time
parameters--they don't directly define a computation, but they
feel like compile-time parameters in that
- They are variable in number.
- They may take on many possible values.
- They may be parameterized (like `extern`).
- They play a role in overload resolution.
- Their combinations may lead to combinatorial explosions
that can hopefully be mitigated with metaprogramming.
My point is, many of the problems we're encountering with
attributes have already been solved (via templates, tuples, CTFE,
`static if`, etc.)
Attribute propagation would be simple (not trivial, but
appropriately simple) for user-defined attributes, because they
can be arbitrary expressions. If built-in attributes were just
treated as pre-defined symbols, they could be manipulated with
templates and CTFE:
enum mayThrow(alias func) =
staticIndexOf!(nothrow, functionAttributes!func) == -1;
template throwsLike(alias func) {
static if (mayThrow!func)
enum throwsLike = tuple().expand;
else
enum throwsLike = nothrow;
}
void call(alias func)() throwsLike!func {
func();
}
(This thread may be relevant:
http://forum.dlang.org/thread/hfmulninvghjntqkpguk forum.dlang.org)

Pretty soon, we need an attribute algebra to express these
complicated relationships.

I think problems come from refusing to have formalized features
in D. Having an attribute algebra is the lesser problem. Having
formalized features causes problems, but no having them
formalized is worse, as visible with D safety, D pureness, D
uniqueness, that are a growing mess. D design needs more
mathematicians and less piling of patches on patches.
Bye,
bearophile

Pretty soon, we need an attribute algebra to express these complicated
relationships.

I think problems come from refusing to have formalized features in D.
Having an attribute algebra is the lesser problem. Having formalized
features causes problems, but no having them formalized is worse, as
visible with D safety, D pureness, D uniqueness, that are a growing
mess. D design needs more mathematicians and less piling of patches on
patches.

+1
I often find myself having second thoughts about all of the 'implicit'
rules esp. with regard to value-range propagation, and half-formalized
such as inout.
--
Dmitry Olshansky

I have always wondered why `inout` is limited to const when problem
is almost identical with all other restrictive attributes.

I have furthermore always wondered why there can always only be one
`inout' wildcard in scope. This is not the best existing way to solve

T foo![T <: const(int)[]](T arg){ return arg; }
this can be extended to other attributes, for example in the following way
(this is just an example):
void evaluate![transitive_attributes a](void delegate() a dg) a{
dg();
}

What if there are multiple delegate arguments?

What if the delegate arguments themselves take delegate arguments?

Pretty soon, we need an attribute algebra to express these complicated
relationships.

Simple propagation, from just one source, would be enough for almost all
cases - there's no need to over-complicate this. The remaining cases could
be handled via introspection and ctfe; this way allows for more options,
not just using some pre-defined algebra subset, which happens to be
supported by a particular compiler (-version).

It would be nice to have a solution that can handle all of these cases
without exploding complexity in the syntax.

Actually supporting parametrized attributes, is something that I think
everybody agrees on in principle (hence the lack of discussions when
this topic is mentioned, every few weeks or so). The required semantics
are pretty clear; what I still haven't seen is a good enough syntax
proposal. One syntax that might have worked for built-in attributes
could have been "const!A" etc, but I'm not sure if the parameter
inference would be intuitive enough, and it would appear, at least
superficially, to potentially clash with user defined attributes,
especially once those become more powerful.
artur

With nothrow and nogc annotations, we've been motivated to add these
annotations to C system API functions, because obviously such functions
aren't going to throw D exceptions or call the D garbage collector.
But this exposed a problem - functions like C's qsort() take a pointer
to a callback function. The callback function, being supplied by the D
programmer, may throw and may call the garbage collector. By requiring
the callback function to be also nothrow nogc, this is an unreasonable
requirement besides breaking most existing D code that uses qsort().
This problem applies as well to the Windows APIs and the Posix APIs with
callbacks.
The solution is to use overloading so that if your callback is nothrow,
it will call the nothrow version of qsort, if it is throwable, it calls
the throwable version of qsort.
Never mind that those two versions of qsort are actually the same
function (!), even though D's type system regards them as different.
Although this looks like an usafe hack, it actually is quite safe,
presuming that the rest of the qsort code itself does not throw. This
technique relies on the fact that extern(C) functions do not get their
types mangled into the names.
Some example code:
extern (C) { alias int function() fp_t; }
extern (C) nothrow { alias int function() fpnothrow_t; }
extern (C) int foo(int a, fp_t fp);
extern (C) nothrow int foo(int a, fpnothrow_t fp);
extern (C) int bar();
extern (C) nothrow int barnothrow();
void test() {
foo(1, &bar); // calls the 'throwing' foo()
foo(1, &barnothrow); // calls the 'nothrow' foo()
}

This only works for those functions that call the callback function
directly.
OS function do not always work this way. They register callbacks for
later use like a windows procedure or a signal handler.
This causes innocent looking functions to not behave as annotated
because they internally use the callback functions. E.g. a lot of the
Windows API functions might use message sending/dispatching internally,
which might execute both throwing or GC allocating callbacks. These are
currently not meeting the promise of their annotations.
We either have to be more conservative with annotating OS functions or
relax the guarantees of nothrow or nogc. Both alternatives are not very
compelling.

Callbacks passed into OS/clib functions are never supposed to
throw so
we must annotate them all with nothrow. C functions are never
designed
under the assumption callbacks may throw. -- Andrei

For nothrow that restriction makes sense, though there are
functions in the Windows "C API" that actually may throw, e.g.
HeapAlloc [1] and MmProbeAndLockPages [2]. No callbacks are
involved in these examples, though.
Under unix, signal handlers that throw exceptions seem to be
used, though this might be non-standard. We even have one in
druntime [3].

The SEGV handles throws Errors, not Exceptions, it's ok to be
nothrow.

The callback function, being supplied by the D programmer, may
throw and may call the garbage collector. By requiring the
callback function to be also nothrow nogc, this is an
unreasonable requirement besides breaking most existing D code
that uses qsort().

<d.learn>
I'm missing something, as I'm annotating all my C/API/etc
callback function with nothrow: when the callback throws, what
happens?
I was thinking that this will mess-up the stack once the unwind
will proceed...
What's the use-case for having such a callback 'throwable'?
Thanks!
</d.learn>
---
Paolo

The callback function, being supplied by the D programmer, may
throw and may call the garbage collector. By requiring the
callback function to be also nothrow nogc, this is an
unreasonable requirement besides breaking most existing D code
that uses qsort().

<d.learn>
I'm missing something, as I'm annotating all my C/API/etc
callback function with nothrow: when the callback throws, what
happens?
I was thinking that this will mess-up the stack once the unwind
will proceed...
What's the use-case for having such a callback 'throwable'?
Thanks!
</d.learn>
---
Paolo

This is actually a really good point. How can a callback in C
code expect to throw exceptions? Surely it should be nothrow
anyway, because it's just not going to work otherwise. Maybe we
should just strengthen the constraints for that, and make people
update their code which isn't likely to work anyway. You can make
any throwing function nothrow by catching Exceptions and throwing
Errors instead at least. Ideally you wouldn't even throw Errors
in C callbacks.

With nothrow and nogc annotations, we've been motivated to add these
annotations to C system API functions, because obviously such functions aren't
going to throw D exceptions or call the D garbage collector.
But this exposed a problem - functions like C's qsort() take a pointer to a
callback function. The callback function, being supplied by the D programmer,
may throw and may call the garbage collector. By requiring the callback
function
to be also nothrow nogc, this is an unreasonable requirement besides breaking
most existing D code that uses qsort().
This problem applies as well to the Windows APIs and the Posix APIs with
callbacks.

I just stumbled upon this thread now...
In general you cannot throw exceptions unless you know how the
call stack outside of your language's barrier is constructed.
What I mean is that while DMD on 64-bit Linux uses frame
pointers that druntime uses to unwind, GCC omits them. So
druntime bails out once it reaches the C part of the call
stack.
That makes for two options:
1) D callbacks passed to other languages must not throw, like
Andrei proposes if I understood that right.
2) Druntime must adapt to the system's C compiler's ABI.
(by the use of libunwind)
--
Marco