Thanks for the wiki notes, they're fixed.
As for the above, I expect it to be 64 bit issues, as I cannot reproduce it
on my similar 32 bit setup. It is not something we have had time to fully
support yet, but it is an immediate goal preparing for the next release and
the coming snapshot builds.
--
Lars Ivar Igesund
blog at http://larsivi.net
DSource, #d.tango & #D: larsivi
Dancing the Tango

As for the above, I expect it to be 64 bit issues, as I cannot reproduce it
on my similar 32 bit setup. It is not something we have had time to fully
support yet, but it is an immediate goal preparing for the next release and
the coming snapshot builds.

Indeed, it seems to compile fine if I install the 32-bit binary from
sourceforge and compile in a 32-bit chroot. Unfortunately, the main
reason I have GDC installed is the new 64-bit support :(.

This is a known issue I'm afraid. GDC designates va_start as an
intrinsic and it must be located in either std.stdarg or std.c.stdarg
(and the behavior is actually different for each of the two). The
easiest fix would be to add these two modules to the 'std' package in
Tango (just like intrinsic) and have the appropriate Tango modules
import them in a "GNU" block. Those modules are: tango.core.Vararg and
tango.stdc.stdarg. I'd have fixed this prior to release but I just
figured out the problem a day or two ago and haven't had time. Expect
this to be fixed in trunk in the next few days.
Sean

This is a known issue I'm afraid. GDC designates va_start as an
intrinsic and it must be located in either std.stdarg or std.c.stdarg
(and the behavior is actually different for each of the two). The
easiest fix would be to add these two modules to the 'std' package in
Tango (just like intrinsic) and have the appropriate Tango modules
import them in a "GNU" block. Those modules are: tango.core.Vararg and
tango.stdc.stdarg. I'd have fixed this prior to release but I just
figured out the problem a day or two ago and haven't had time. Expect
this to be fixed in trunk in the next few days.

Okay, I've checked a fix into trunk. If anyone is inclined to try it
out, please let me know if it works.
Sean

Okay, I've checked a fix into trunk. If anyone is inclined to try it
out, please let me know if it works.

Sorry, but no. At first I get an error because you forgot to define
va_list in std/c/stdarg.di. After copying
---
private import gcc.builtins;
alias __builtin_va_list va_list;
---
into the version(GNU) block, I get basically the same error message as
before:
---
gdc -o lifetime.o -g -frelease -O2 -fversion=GC_Use_Alloc_MMap
-fversion=GC_Use_Stack_GLibC -fversion=GC_Use_Data_Fixed -nostdinc -pipe
-I../../.. \
-c lifetime.d
../../../std/c/stdarg.di:16: Error: cannot have out or inout parameter
of type ubyte[24][1]
../../../std/c/stdarg.di:815: template instance
std.c.stdarg.va_start!(uint) error instantiating
lifetime.d:815: Error: cannot change reference to static array 'va'
make[2]: *** [lifetime.o] Error 1
make[2]: Leaving directory
`/home/urxae/opt/tango/svn-trunk/lib/compiler/gdc'
make[1]: *** [all] Error 2
make[1]: Leaving directory
`/home/urxae/opt/tango/svn-trunk/lib/compiler/gdc'
make: *** [lib] Error 2
---
However, copying GDC's std.c.stdarg module over it lets it compile a bit
further. Unfortunately, it then chokes on something else:
---
gcc -c -O -m32 core/ThreadASM.S -ocore/ThreadASM.o
gcc -c -O -m32 stdc/wrap.c -ostdc/wrap.o
In file included from /usr/include/features.h:346,
from /usr/include/errno.h:29,
from stdc/wrap.c:1:
/usr/include/gnu/stubs.h:7:27: error: gnu/stubs-32.h: No such file or
directory
make[1]: *** [stdc/wrap.o] Error 1
make[1]: Leaving directory
`/home/urxae/opt/tango/svn-trunk/lib/common/tango'
make: *** [lib] Error 2
---
I've seen this error before when trying to get my 64-bit GCC to compile
32-bit code[1], but I'm wondering *why* it tries to do that. I don't
think it'll run properly (or perhaps even fail to link against 64-bit
binaries) even *if* you get it to compile...
(Removing the -m32 and executing the second 'gcc' command manually works
fine, by the way)
I guess the first command worked mostly because it doesn't contain any
code outside "#if defined( __ppc__ )".
[a bit later]
After removing -m32 from CFLAGS in all relevant makefiles (and perhaps
some irrelevant ones; the only one I left was in
lib/compiler/dmd/posix.mak since I figured it wouldn't use that one :) )
build-gdc.sh completes successfully.
I attached a patch, though I'm not sure how much use it'll be if you
insist on your own version of std.c.stdarg ...
[1]: "-m32" means "emulate 32-bit compiler", right?

I attached a patch, though I'm not sure how much use it'll be if you
insist on your own version of std.c.stdarg ...

Thanks. I think the stdarg issues should now be resolved and I'll look
into making the other changes. About the only place I'm not entirely
sure they will work is common/tango/stdc/wrap.c, since this is built by
both DMD and GDC. I think I'm going to pass -m32 through from
dmd-posix.mak, etc, using a C equivalent of ADDFLAGS. So:
ADDFLAGS -> ADD_DFLAGS
(new) -> ADD_CFLAGS
This should support all configurations.

[Fair warning: this seems to have become a pretty big post]
Sean Kelly wrote:

Frits van Bommel wrote:

I attached a patch, though I'm not sure how much use it'll be if you
insist on your own version of std.c.stdarg ...

Thanks. I think the stdarg issues should now be resolved [...]

Sorry to disappoint again.
Unfortunately, build-gdc.sh only compiles files in lib/ (and some in
std/ imported by them). These compile successfully with my patch, AFAICT.
The stuff in tango/ doesn't work quite as well though.
For instance, it turns out tango.text.convert.Layout is pretty screwed
in the current implementation. Again, this has to do with varargs.
At the top it has:
-----
/*******************************************************************************
Platform issues ...
*******************************************************************************/
version (DigitalMars)
alias void* Arg;
else
alias char* Arg;
-----
and it then proceeds to use Arg to receive va_list arguments. Well, I
thought this would be an easy fix:
-----
private import tango.core.Vararg;
alias va_list Arg;
-----
Unfortunately, that doesn't work. Specifically because of how Arg is
used in "public final uint convert (Sink sink, TypeInfo[] arguments, Arg
args, T[] formatStr)":
-----
Arg[64] arglist = void;
foreach (i, arg; arguments)
{
arglist[i] = args;
args += (arg.tsize + int.sizeof - 1) & ~ (int.sizeof - 1);
}
return parse (formatStr, arguments, arglist, sink);
-----
Here it looks like it tries to manually extract the arguments, assuming
they all lie (on the stack) at int.sizeof boundaries. While this may
work on (32-bit?) platforms where varargs are all passed on the stack
like x86 with C or D calling convention[1], on amd64 this is not the
case: va_list isn't a pointer and the variable arguments aren't
necessarily all on the stack.
Since it seems GDC basically uses the C calling convention, I'll give a
short description of the problems I see with it:
The C(++?) calling convention on amd64[2] passes the first several
parameters in registers, and it seems not to deviate from this in case
of varargs. Worse, it uses different sets of registers for different
types of arguments (6 general-purpose registers for integer types &
pointers etc., 8 SSE regs for float/double, and the rest in memory).
Aggregates are potentially split up(!) if certain conditions are met (<
16 bytes, all members are naturally aligned, fits entirely into
registers, and a few more).
Basically, what I'm trying to say is: This would most likely be hell to
implement manually, and the code using it still unportable to yet other
calling conventions.
IMHO what we really need to do is either something like educating
TypeInfo to know how to "increment" a va_list to skip the appropriate
type[3], or changing GDC to just pass varargs on the stack when
compiling an extern(D) function, no matter the local C calling
convention :).
In the mean time, perhaps this function (and any vararg functions
calling it with their own va_list) could be changed to use variadic
template args (I do hope none of these are overridden anywhere, or this
won't work). This will likely result in some code bloat though, with
most calls using their own private instantiation, so it's not a really
nice long-term solution. It _should_ be pretty portable though :).
There were some other errors I less thoroughly investigated, but most of
these look like they can probably easily be fixed (in the first two
cases perhaps even by simply using the same code as for x86?):
* tango.sys.linux.socket static asserts(0) if version(X86) isn't
defined, with comment "// Different values on other platforms." (the X86
branch defines a constant named "SOL_SOCKET", whatever that may be)
* tango.math.IEEE doesn't define the FPU masks for amd64 (just for x86
and PPC it seems).
* tango.text.Regex also seems to assume that va_list is a pointer to a
1-byte type instead of using va_start/va_arg/va_end.
* tango.stdc.posix.setjmp straight static asserts(false, "Architecture
not supported.") on version(X86_64) (though the actual error is an
undefined identifier used after that, presumably because static asserts
are evaluated a bit late in the parsing process)
[1]: IIRC x86 GDC uses the same calling convention for both, and DMD
uses a very similar convention in case of varargs.
[2]: As detailed in http://www.x86-64.org/documentation/abi.pdf,
specifically the section "Parameter passing" on pages 15-22.
[3]: Or a va_arg variant that uses a run-time TypeInfo argument instead
of a compile-time template type argument...

[Fair warning: this seems to have become a pretty big post]
Unfortunately, that doesn't work. Specifically because of how Arg is
used in "public final uint convert (Sink sink, TypeInfo[] arguments, Arg
args, T[] formatStr)":
-----
Arg[64] arglist = void;
foreach (i, arg; arguments)
{
arglist[i] = args;
args += (arg.tsize + int.sizeof - 1) & ~ (int.sizeof - 1);
}
return parse (formatStr, arguments, arglist, sink);
-----
Here it looks like it tries to manually extract the arguments, assuming
they all lie (on the stack) at int.sizeof boundaries. While this may
work on (32-bit?) platforms where varargs are all passed on the stack
like x86 with C or D calling convention[1], on amd64 this is not the
case: va_list isn't a pointer and the variable arguments aren't
necessarily all on the stack.

Aye, it is intended to make each argument indexable. Only works on
32-bit platform, making the same assumptions as the code in std.stdarg

Since it seems GDC basically uses the C calling convention, I'll give a
short description of the problems I see with it:
The C(++?) calling convention on amd64[2] passes the first several
parameters in registers, and it seems not to deviate from this in case
of varargs. Worse, it uses different sets of registers for different
types of arguments (6 general-purpose registers for integer types &
pointers etc., 8 SSE regs for float/double, and the rest in memory).
Aggregates are potentially split up(!) if certain conditions are met (<
16 bytes, all members are naturally aligned, fits entirely into
registers, and a few more).

No problem. I assure you, it's mostly for selfish motives ;). I just
want it to work without having to resort to 32-bit compilers on a 64-bit
system.
(Though I get by with Phobos just fine)

[Fair warning: this seems to have become a pretty big post]
Unfortunately, that doesn't work. Specifically because of how Arg is
used in "public final uint convert (Sink sink, TypeInfo[] arguments,
Arg args, T[] formatStr)":
-----
Arg[64] arglist = void;
foreach (i, arg; arguments)
{
arglist[i] = args;
args += (arg.tsize + int.sizeof - 1) & ~ (int.sizeof - 1);
}
return parse (formatStr, arguments, arglist, sink);
-----
Here it looks like it tries to manually extract the arguments,
assuming they all lie (on the stack) at int.sizeof boundaries. While
this may work on (32-bit?) platforms where varargs are all passed on
the stack like x86 with C or D calling convention[1], on amd64 this is
not the case: va_list isn't a pointer and the variable arguments
aren't necessarily all on the stack.

Aye, it is intended to make each argument indexable.

That's what I thought.
I actually tried to change it to a more portable equivalent but gave up
once I realized that wasn't easily done without having access to the
actual types of the parameters at compile time.

Only works on
32-bit platform, making the same assumptions as the code in std.stdarg

AFAICT only DMD's version of std.stdarg makes those assumptions. (And it
technically has every right to, since DMD is currently 32-bit only)

Since it seems GDC basically uses the C calling convention, I'll give
a short description of the problems I see with it:
The C(++?) calling convention on amd64[2] passes the first several
parameters in registers, and it seems not to deviate from this in case
of varargs. Worse, it uses different sets of registers for different
types of arguments (6 general-purpose registers for integer types &
pointers etc., 8 SSE regs for float/double, and the rest in memory).
Aggregates are potentially split up(!) if certain conditions are met
(< 16 bytes, all members are naturally aligned, fits entirely into
registers, and a few more).

stdarg.d would fail also. That is not good :(

It shouldn't for the GDC version, I think.
Note these snippets from the code for std.stdarg distributed with GDC:
---
// va_arg is handled magically by the compiler
// [...]
template va_arg(T)
{
T va_arg(inout va_list _argptr)
{
/*
T arg = *cast(T*)_argptr;
_argptr = _argptr + ((T.sizeof + int.sizeof - 1) & ~(int.sizeof - 1));
return arg;
*/
T t; return t;
}
}
---
Note how little va_arg does (just enough to compile), and the comment
mentioning special compiler magic that presumably fixes it to do the
Right Thing(TM).
I haven't tried it out though, other than using writefln() in programs
and getting the expected output :).

[Fair warning: this seems to have become a pretty big post]
Sean Kelly wrote:

Frits van Bommel wrote:

I attached a patch, though I'm not sure how much use it'll be if you
insist on your own version of std.c.stdarg ...

Thanks. I think the stdarg issues should now be resolved [...]

Sorry to disappoint again.
Unfortunately, build-gdc.sh only compiles files in lib/ (and some in
std/ imported by them). These compile successfully with my patch, AFAICT.

Is this related to the stdarg issues? Just want to make sure I
understand what you meant by "sorry to disappoint."

The stuff in tango/ doesn't work quite as well though.
For instance, it turns out tango.text.convert.Layout is pretty screwed
in the current implementation. Again, this has to do with varargs.

Saw that. I passed the issue to Kris, but I suspect we may have to work
together on this one.

There were some other errors I less thoroughly investigated, but most of
these look like they can probably easily be fixed (in the first two
cases perhaps even by simply using the same code as for x86?):
* tango.sys.linux.socket static asserts(0) if version(X86) isn't
defined, with comment "// Different values on other platforms." (the X86
branch defines a constant named "SOL_SOCKET", whatever that may be)
* tango.math.IEEE doesn't define the FPU masks for amd64 (just for x86
and PPC it seems).
* tango.text.Regex also seems to assume that va_list is a pointer to a
1-byte type instead of using va_start/va_arg/va_end.

Thanks. We'll look into these. And please feel free to submit tickets
for these or any other problem you find.

* tango.stdc.posix.setjmp straight static asserts(false, "Architecture
not supported.") on version(X86_64) (though the actual error is an
undefined identifier used after that, presumably because static asserts
are evaluated a bit late in the parsing process)

This was deliberate, as a clear and obvious reminder to expand support
as needed. But perhaps this should be allowed to compile, accompanied
by a pragma(msg) that Fibers won't work? That said, if someone wants to
pass on the relevant setjmp headers for a 64-bit glibc then I'll see
about expanding support. With this in mind, I don't suppose GDC yet
supports inline ASM for the new 64-bit registers, etc?
Sean

[Fair warning: this seems to have become a pretty big post]
Sean Kelly wrote:

Frits van Bommel wrote:

I attached a patch, though I'm not sure how much use it'll be if you
insist on your own version of std.c.stdarg ...

Thanks. I think the stdarg issues should now be resolved [...]

Sorry to disappoint again.
Unfortunately, build-gdc.sh only compiles files in lib/ (and some in
std/ imported by them). These compile successfully with my patch, AFAICT.

Is this related to the stdarg issues? Just want to make sure I
understand what you meant by "sorry to disappoint."

Well, the biggest issue was definitely vararg-related, yeah. And
specifically unportable assumptions made in regards to them.

The stuff in tango/ doesn't work quite as well though.
For instance, it turns out tango.text.convert.Layout is pretty screwed
in the current implementation. Again, this has to do with varargs.

Saw that. I passed the issue to Kris, but I suspect we may have to work
together on this one.

There were some other errors I less thoroughly investigated, but most
of these look like they can probably easily be fixed (in the first two
cases perhaps even by simply using the same code as for x86?):
* tango.sys.linux.socket static asserts(0) if version(X86) isn't
defined, with comment "// Different values on other platforms." (the
X86 branch defines a constant named "SOL_SOCKET", whatever that may be)
* tango.math.IEEE doesn't define the FPU masks for amd64 (just for x86
and PPC it seems).
* tango.text.Regex also seems to assume that va_list is a pointer to a
1-byte type instead of using va_start/va_arg/va_end.

Thanks. We'll look into these. And please feel free to submit tickets
for these or any other problem you find.

I don't think I'll find any others until these are fixed, since all I
did was try to compile every .d file in the tango/ hierarchy with "find
-name *.d -exec gdc [...]" :).
Though that last one should be trivial to fix.

* tango.stdc.posix.setjmp straight static asserts(false, "Architecture
not supported.") on version(X86_64) (though the actual error is an
undefined identifier used after that, presumably because static
asserts are evaluated a bit late in the parsing process)

This was deliberate, as a clear and obvious reminder to expand support
as needed.

Consider yourself reminded :).

But perhaps this should be allowed to compile, accompanied
by a pragma(msg) that Fibers won't work?

Oh, I didn't even know what it was used for, or that it was used by
Tango itself at all. Like mentioned above, I just indiscriminately tried
to compile everything...

That said, if someone wants to
pass on the relevant setjmp headers for a 64-bit glibc then I'll see
about expanding support.

You mean the ones attached? Or do you need any others?

With this in mind, I don't suppose GDC yet
supports inline ASM for the new 64-bit registers, etc?

Not using the regular DMD-style with Intel syntax, IIRC. But the
"extended asm" (using at&t syntax in strings) supports them, I think.
Actually, if it's anything like regular GCC then it's almost literally
dumped into the input sent to gas (the gnu assembler).

pass on the relevant setjmp headers for a 64-bit glibc then I'll see
about expanding support.

You mean the ones attached? Or do you need any others?

That should do it. Though if there are any related comments on what
registers are stored in __jmp_buf, that would come in handy. The ones I
have for Linux contain this in bits/setjmp.h:
#if defined __USE_MISC || defined _ASM
# define JB_BX 0
# define JB_SI 1
# define JB_DI 2
# define JB_BP 3
# define JB_SP 4
# define JB_PC 5
# define JB_SIZE 24
#endif
Your headers don't contain that however, so I'm not entirely sure how to
slice & dice jmp_buf to inject the proper information. I'll google a
bit and try to find out as well.

> With this in mind, I don't suppose GDC yet

supports inline ASM for the new 64-bit registers, etc?

Not using the regular DMD-style with Intel syntax, IIRC. But the
"extended asm" (using at&t syntax in strings) supports them, I think.
Actually, if it's anything like regular GCC then it's almost literally
dumped into the input sent to gas (the gnu assembler).

pass on the relevant setjmp headers for a 64-bit glibc then I'll see
about expanding support.

You mean the ones attached? Or do you need any others?

That should do it. Though if there are any related comments on what
registers are stored in __jmp_buf, that would come in handy. The ones I
have for Linux contain this in bits/setjmp.h:
#if defined __USE_MISC || defined _ASM
# define JB_BX 0
# define JB_SI 1
# define JB_DI 2
# define JB_BP 3
# define JB_SP 4
# define JB_PC 5
# define JB_SIZE 24
#endif
Your headers don't contain that however, so I'm not entirely sure how to
slice & dice jmp_buf to inject the proper information. I'll google a
bit and try to find out as well.

Not using the regular DMD-style with Intel syntax, IIRC. But the
"extended asm" (using at&t syntax in strings) supports them, I think.
Actually, if it's anything like regular GCC then it's almost literally
dumped into the input sent to gas (the gnu assembler).

pass on the relevant setjmp headers for a 64-bit glibc then I'll see
about expanding support.

You mean the ones attached? Or do you need any others?

That should do it. Though if there are any related comments on what
registers are stored in __jmp_buf, that would come in handy. The ones
I have for Linux contain this in bits/setjmp.h:
#if defined __USE_MISC || defined _ASM
# define JB_BX 0
# define JB_SI 1
# define JB_DI 2
# define JB_BP 3
# define JB_SP 4
# define JB_PC 5
# define JB_SIZE 24
#endif
Your headers don't contain that however, so I'm not entirely sure how
to slice & dice jmp_buf to inject the proper information. I'll google
a bit and try to find out as well.

The above error is fixed.
However, after reverting my changes, I still get the old va_arg error:

...

I dug into it a bit more and it turns out GDC chokes if va_start and/or
va_arg are inside version(GNU)(!).

Ack! Fixed now. Thanks :-)

I don't think this should cause problems for DMD since the only import
of this module is inside version(GNU) itself...

Yup. I mostly did it to be safe.

User-level code importing std.c.stdarg may be a problem. Unfortunately,
this seems unavoidable since the code is needed by tango.stdc.stdarg.
Unless perhaps it could be moved to lib/compiler/gdc and only copied to
an import path when building/installing for GDC?

I'd considered moving it somewhere else entirely (gcc, perhaps) and just
naming the module std.c.stdarg internally, but I have no idea if that
would work with the way intrinsics are handled in GDC. If you're
inclined to try this and it works, however, please let me know. If this
becomes a long-term fix I may also look into having GDC install them as
you suggest. I'll admit I'm still secretly hoping to not have to
maintain 'std' to support intrinsics forever.
Sean

I'd considered moving it somewhere else entirely (gcc, perhaps) and just
naming the module std.c.stdarg internally, but I have no idea if that
would work with the way intrinsics are handled in GDC. If you're
inclined to try this and it works, however, please let me know. If this
becomes a long-term fix I may also look into having GDC install them as
you suggest. I'll admit I'm still secretly hoping to not have to
maintain 'std' to support intrinsics forever.

Doesn't seem to work. Apparently it needs to be imported as std.c.stdarg
for it to work...
Maybe you guys should talk to the compiler developers and suggest a more
movable way to mark special modules.
Perhaps a pragma:
---
version(CompilerSource) pragma(magic_module, c_stdarg);
---
(with CompilerSource in [GNU, DigitalMars])
That sort of thing is what pragmas were intended to do, wasn't it?
(From http://www.digitalmars.com/d/pragma.html: "Pragmas are a way to
pass special information to the compiler and to add vendor specific
extensions to D.")

I'd considered moving it somewhere else entirely (gcc, perhaps) and
just naming the module std.c.stdarg internally, but I have no idea if
that would work with the way intrinsics are handled in GDC. If you're
inclined to try this and it works, however, please let me know. If
this becomes a long-term fix I may also look into having GDC install
them as you suggest. I'll admit I'm still secretly hoping to not have
to maintain 'std' to support intrinsics forever.

Doesn't seem to work. Apparently it needs to be imported as std.c.stdarg
for it to work...
Maybe you guys should talk to the compiler developers and suggest a more
movable way to mark special modules.
Perhaps a pragma:
---
version(CompilerSource) pragma(magic_module, c_stdarg);
---
(with CompilerSource in [GNU, DigitalMars])
That sort of thing is what pragmas were intended to do, wasn't it?
(From http://www.digitalmars.com/d/pragma.html: "Pragmas are a way to
pass special information to the compiler and to add vendor specific
extensions to D.")

A pragma is a good idea. That seems like it might allow for a clean
separation between library design and compiler implementation, whether
the library is Phobos, Tango, or something else entirely. In fact,
there are some intrinsics Tango isn't taking advantage of simply because
the hardcoded module names are too much of a bother in some cases. For
example, DMD provides intrinsics for some of the functions in std.math.
Sean

Within a
few days, daily source-snapshots plus binaries will be enabled also.

It took longer than planned, but the first daily snapshots are now up, see
the download page, close to the bottom. Please report any problems that you
find or things that could be improved. I hope to improve the relevant
install docs sometime during easter.