tl;dr: The main things I need help with: Installing 32-bit libcurl on
64-bit Debian, testing on 64-bit FreeBSD, and figuring out WTF is so
screwed up on Windows.
It's almost finished, but I've hit some problems I'm having a hell of a
time figuring out. I'm getting a kinda brain-fried and need some help
with it:
Here's what I have right now (it's named "create_dmd_release"):
https://github.com/Abscissa/installer/tree/create-zip
The "how to" documentation is at the top of the source file:
https://github.com/Abscissa/installer/blob/create-zip/create_dmd_release/create_dmd_release.d
Important note for trying it out:
--------------------------------------
This tool depends on itself actually being in the official repo. Since
this hasn't actually been pulled in yet (I don't think it's ready for a
pull request just yet), you will need one small change to the
instructions for step #2 to make it work right:
Run the command line in step #2, but after it finishes cloning, go
ahead and Ctrl-C kill it (or wait for it to end - either way). Then,
copy the "create_dmd_release" directory tree into "[your temp
dir]/.create_dmd_release/installer". You can check the temp dir it uses
with "create_dmd_release --help". Finally, re-run the same
original command, but add the flag "--skip-clone".
DMD versions:
-----------------------
This is intended for DMDs *after* v2.063.2, because it requires some
changes in the "master" of the "tools" repo.
However, you *can* coax it to work on v2.063.2 by halting it after the
"clone" stage, then updating
"[tmp-dir]/.create_dmd_release/tools/*.mak" with the latest versions
from master, and then resuming the process by re-running with
"--skip-clone". But additional hacking may be needed on Windows.
The current status:
-----------------------
- OSX 10.7: Works fine. (At least it did last time I tried it, a couple
weeks ago. It *could* have regressed but I won't have access to the
machine again until maybe Sunday.)
- Debian 6 (32-bit OS): This tool isn't intended to support 32-bit OSes
(since it must compile both 32-bit and 64-bit phobos), but it works
fine up until it gets to the 64-bit stuff. If I manually comment out
the 64-bit stuff, then everything else works fine.
- Debian 6 (64-bit OS): It fails when linking the 32-bit phobos
complaining that it can't link with libcurl. I'm fairly certain this
is because I have no freaking idea how to install the 32-bit
libcurl*-*-dev on a 64-bit Debian. (I *did* install
"libcurl4-openssl-dev", but I'm pretty sure it only installed for
64-bit. And I can't figure out how to get the 32-bit.)
- FreeBSD 9.1 (32-bit OS): Same as 32-bit Debian 6: It works, except
for the 64-bit stuff.
- FreeBSD 9.1 (64-bit OS): No idea, I don't have access to a 64-bit
FreeBSD machine, and my stupid Intel CPU lacks the ability to run a
64-bit OS in a VM.
- Windows 7 (64-bit OS): This is where I'm having the biggest trouble.
I can coaxing it to handle v2.063.2 just fine (ie, if I update the
"tools" makefiles as described above, and comment out all the
"libcurl" and "chm" stuff.) However, on master, after it compiles
DMD/druntime/phobos, the resulting DMD/phobos can't compile anything
that uses phobos because OPTLINK will spew out a bunch of errors. I
am *completely* at a loss on this one. It seems like an sc.ini issue,
but I've spent days checking everything and I still can't make heads
or tails of it.

tl;dr: The main things I need help with: Installing 32-bit
libcurl on
64-bit Debian, testing on 64-bit FreeBSD, and figuring out WTF
is so
screwed up on Windows.
It's almost finished, but I've hit some problems I'm having a
hell of a
time figuring out. I'm getting a kinda brain-fried and need
some help
with it:
Here's what I have right now (it's named "create_dmd_release"):
https://github.com/Abscissa/installer/tree/create-zip
The "how to" documentation is at the top of the source file:
https://github.com/Abscissa/installer/blob/create-zip/create_dmd_release/create_dmd_release.d
Important note for trying it out:
--------------------------------------
This tool depends on itself actually being in the official
repo. Since
this hasn't actually been pulled in yet (I don't think it's
ready for a
pull request just yet), you will need one small change to the
instructions for step #2 to make it work right:
Run the command line in step #2, but after it finishes cloning,
go
ahead and Ctrl-C kill it (or wait for it to end - either way).
Then,
copy the "create_dmd_release" directory tree into "[your temp
dir]/.create_dmd_release/installer". You can check the temp dir
it uses
with "create_dmd_release --help". Finally, re-run the same
original command, but add the flag "--skip-clone".
DMD versions:
-----------------------
This is intended for DMDs *after* v2.063.2, because it requires
some
changes in the "master" of the "tools" repo.
However, you *can* coax it to work on v2.063.2 by halting it
after the
"clone" stage, then updating
"[tmp-dir]/.create_dmd_release/tools/*.mak" with the latest
versions
from master, and then resuming the process by re-running with
"--skip-clone". But additional hacking may be needed on Windows.
The current status:
-----------------------
- OSX 10.7: Works fine. (At least it did last time I tried it,
a couple
weeks ago. It *could* have regressed but I won't have access
to the
machine again until maybe Sunday.)
- Debian 6 (32-bit OS): This tool isn't intended to support
32-bit OSes
(since it must compile both 32-bit and 64-bit phobos), but it
works
fine up until it gets to the 64-bit stuff. If I manually
comment out
the 64-bit stuff, then everything else works fine.
- Debian 6 (64-bit OS): It fails when linking the 32-bit phobos
complaining that it can't link with libcurl. I'm fairly
certain this
is because I have no freaking idea how to install the 32-bit
libcurl*-*-dev on a 64-bit Debian. (I *did* install
"libcurl4-openssl-dev", but I'm pretty sure it only installed
for
64-bit. And I can't figure out how to get the 32-bit.)
- FreeBSD 9.1 (32-bit OS): Same as 32-bit Debian 6: It works,
except
for the 64-bit stuff.
- FreeBSD 9.1 (64-bit OS): No idea, I don't have access to a
64-bit
FreeBSD machine, and my stupid Intel CPU lacks the ability to
run a
64-bit OS in a VM.
- Windows 7 (64-bit OS): This is where I'm having the biggest
trouble.
I can coaxing it to handle v2.063.2 just fine (ie, if I
update the
"tools" makefiles as described above, and comment out all the
"libcurl" and "chm" stuff.) However, on master, after it
compiles
DMD/druntime/phobos, the resulting DMD/phobos can't compile
anything
that uses phobos because OPTLINK will spew out a bunch of
errors. I
am *completely* at a loss on this one. It seems like an
sc.ini issue,
but I've spent days checking everything and I still can't
make heads
or tails of it.

You have to manually tell dpkg that it can use specific
architectures like 32bit in a 64bit system [1].
However this takes a bit of configuration to get working so this
might have to be left to the user and not placed into the script.
[1] https://wiki.debian.org/Multiarch/HOWTO

You have to manually tell dpkg that it can use specific
architectures like 32bit in a 64bit system [1].
However this takes a bit of configuration to get working so this
might have to be left to the user and not placed into the script.
[1] https://wiki.debian.org/Multiarch/HOWTO

Eh, damn, it looks like I may need to put together a Debian 7 persistent
live disc/usb because I can't seem to get the necessary versions of
dpkg/apt in Deb 6, even in backports.
But...don't linux bins tend to not work on systems (ie, glibc)
older than the one used to compile? If that's the case then maybe I
should make the 32- and 64-bit passes separable so a multilib system
isn't required (which appears to mandate very new versions of linux,
at least for debian anyway).
What kind of linux system is currently being used to generate the linux
releases?

You have to manually tell dpkg that it can use specific
architectures like 32bit in a 64bit system [1].
However this takes a bit of configuration to get working so
this might have to be left to the user and not placed into the
script.
[1] https://wiki.debian.org/Multiarch/HOWTO

Eh, damn, it looks like I may need to put together a Debian 7
persistent
live disc/usb because I can't seem to get the necessary
versions of
dpkg/apt in Deb 6, even in backports.
But...don't linux bins tend to not work on systems (ie, glibc)
older than the one used to compile? If that's the case then
maybe I
should make the 32- and 64-bit passes separable so a multilib
system
isn't required (which appears to mandate very new versions of
linux,
at least for debian anyway).
What kind of linux system is currently being used to generate
the linux
releases?

I think in the case of Debian based systems that having one for
32bit and 64bit might work. However don't quote me on that.
I took a look at Ubuntu's support and its very similar to
Debian's.
"In Debian dpkg this is present since 1.16.2. In Ubuntu this is
present since natty (v1.15.8.10ubuntu1)."
That may be of some use to you. Thats for multiarch support.
I can't really help much more than this I'm afraid.

The current status:
-----------------------
- OSX 10.7: Works fine. (At least it did last time I tried it, a couple
weeks ago. It *could* have regressed but I won't have access to the
machine again until maybe Sunday.)

I can test it on Mac OS X 10.8. Are you building a universal binary of
the libraries?

- Debian 6 (32-bit OS): This tool isn't intended to support 32-bit OSes
(since it must compile both 32-bit and 64-bit phobos), but it works
fine up until it gets to the 64-bit stuff. If I manually comment out
the 64-bit stuff, then everything else works fine.

Why don't you just version that out?

- Debian 6 (64-bit OS): It fails when linking the 32-bit phobos
complaining that it can't link with libcurl. I'm fairly certain this
is because I have no freaking idea how to install the 32-bit
libcurl*-*-dev on a 64-bit Debian. (I *did* install
"libcurl4-openssl-dev", but I'm pretty sure it only installed for
64-bit. And I can't figure out how to get the 32-bit.)

Using "apt-get install package:architecture" as suggested here:
https://wiki.debian.org/Multiarch/HOWTO
Requires Debian 7.0 or later. For Debian 6, install the the packages
"gcc-multilib" and "ia32-libs".

- FreeBSD 9.1 (32-bit OS): Same as 32-bit Debian 6: It works, except
for the 64-bit stuff.
- FreeBSD 9.1 (64-bit OS): No idea, I don't have access to a 64-bit
FreeBSD machine, and my stupid Intel CPU lacks the ability to run a
64-bit OS in a VM.

I can give you a help with this, if no one else beats me.

- Windows 7 (64-bit OS): This is where I'm having the biggest trouble.
I can coaxing it to handle v2.063.2 just fine (ie, if I update the
"tools" makefiles as described above, and comment out all the
"libcurl" and "chm" stuff.) However, on master, after it compiles
DMD/druntime/phobos, the resulting DMD/phobos can't compile anything
that uses phobos because OPTLINK will spew out a bunch of errors. I
am *completely* at a loss on this one. It seems like an sc.ini issue,
but I've spent days checking everything and I still can't make heads
or tails of it.

The current status:
-----------------------
- OSX 10.7: Works fine. (At least it did last time I tried it, a
couple weeks ago. It *could* have regressed but I won't have access
to the machine again until maybe Sunday.)

I can test it on Mac OS X 10.8.

Great. :)

Are you building a universal binary
of the libraries?

Yes.

- Debian 6 (32-bit OS): This tool isn't intended to support 32-bit
OSes (since it must compile both 32-bit and 64-bit phobos), but it
works fine up until it gets to the 64-bit stuff. If I manually
comment out the 64-bit stuff, then everything else works fine.

Why don't you just version that out?

I'm thinking I may just add --32-bit(-only) and --64-bit(-only)
switches.

- Debian 6 (64-bit OS): It fails when linking the 32-bit phobos
complaining that it can't link with libcurl. I'm fairly certain
this is because I have no freaking idea how to install the 32-bit
libcurl*-*-dev on a 64-bit Debian. (I *did* install
"libcurl4-openssl-dev", but I'm pretty sure it only installed for
64-bit. And I can't figure out how to get the 32-bit.)

Using "apt-get install package:architecture" as suggested here:
https://wiki.debian.org/Multiarch/HOWTO
Requires Debian 7.0 or later. For Debian 6, install the the packages
"gcc-multilib" and "ia32-libs".

I definitely have multilib on there since other stuff works 32-bit
(like rdmd). It's just libcurl that I can't seem to get 32-bit or
multilib.

- FreeBSD 9.1 (64-bit OS): No idea, I don't have access to a 64-bit
FreeBSD machine, and my stupid Intel CPU lacks the ability to
run a 64-bit OS in a VM.

Requires Debian 7.0 or later. For Debian 6, install the the
packages "gcc-multilib" and "ia32-libs".

I definitely have multilib on there since other stuff works 32-bit
(like rdmd). It's just libcurl that I can't seem to get 32-bit or
multilib.

I would be very surprised if libcurl isn't included in ia32-libs.

Yes, "ia32-libs" depends on "ia32-libs-i386:i386" which depends on
"libcurl3:i386"

It turns out I was missing "ia32-libs", however I just installed it an
I'm still getting "cannot find -lcurl" (or something like that) from
the linker. "gcc-multilib" is already installed, and again, 32-bit stuff
that doesn't use libcurl works fine.

Requires Debian 7.0 or later. For Debian 6, install the the
packages "gcc-multilib" and "ia32-libs".

I definitely have multilib on there since other stuff works 32-bit
(like rdmd). It's just libcurl that I can't seem to get 32-bit or
multilib.

I would be very surprised if libcurl isn't included in ia32-libs.

Yes, "ia32-libs" depends on "ia32-libs-i386:i386" which depends on
"libcurl3:i386"

It turns out I was missing "ia32-libs", however I just installed it an
I'm still getting "cannot find -lcurl" (or something like that) from
the linker. "gcc-multilib" is already installed, and again, 32-bit stuff
that doesn't use libcurl works fine.

Did you install libcurl*-dev? Debian lib packages generally only include
runtime .so's; to get compile-time stuff you need the corresponding -dev
packages.
And I'm not sure why, but it seems that in debian/unstable there are
multiple incompatible versions of libcurl-dev. The one I have is
libcurl4-openssl-dev, just FYI.
T
--
Turning your clock 15 minutes ahead won't cure lateness---you're just making
time go faster!

It turns out I was missing "ia32-libs", however I just installed it
an I'm still getting "cannot find -lcurl" (or something like that)
from the linker. "gcc-multilib" is already installed, and again,
32-bit stuff that doesn't use libcurl works fine.

Did you install libcurl*-dev? Debian lib packages generally only
include runtime .so's; to get compile-time stuff you need the
corresponding -dev packages.

Yea, I had already installed libcurl4-openssl-dev. I even removed it
and reinstalled it after installing "ia32-libs". But still no dice.

It turns out I was missing "ia32-libs", however I just installed it
an I'm still getting "cannot find -lcurl" (or something like that)
from the linker. "gcc-multilib" is already installed, and again,
32-bit stuff that doesn't use libcurl works fine.

Did you install libcurl*-dev? Debian lib packages generally only
include runtime .so's; to get compile-time stuff you need the
corresponding -dev packages.

Yea, I had already installed libcurl4-openssl-dev. I even removed it
and reinstalled it after installing "ia32-libs". But still no dice.

You need to install the 32-bit version of "libcurl4-openssl-dev", but due to
dependencies this will force to replace "pkg-config:amd64" by
"pkg-config:i386", because cannot have both 32-bit and 64-bit installed at same
time.
The easiest way to do it is just create a symlink to the already installed
shared curl library.
With "libcurl3:i386" installed, you can try this:
$ sudo ln -s libcurl.so.4 /usr/lib/i386-linux-gnu/libcurl.so
--
Jordi Sayol

It turns out I was missing "ia32-libs", however I just installed
it an I'm still getting "cannot find -lcurl" (or something like
that) from the linker. "gcc-multilib" is already installed, and
again, 32-bit stuff that doesn't use libcurl works fine.

Did you install libcurl*-dev? Debian lib packages generally only
include runtime .so's; to get compile-time stuff you need the
corresponding -dev packages.

Yea, I had already installed libcurl4-openssl-dev. I even removed it
and reinstalled it after installing "ia32-libs". But still no dice.

Oh wait, you're complaining about 32-bit support, aren't you? You'll
need to:
dpkg --add-architecture i386
apt-get update
apt-get install libcurl4-openssl-dev:i386
Under Debian multiarch, packages for different architectures are
installed separately, so you have to explicitly install the same package
for both architectures if you need it for both.
T
--
People say I'm indecisive, but I'm not sure about that. -- YHL, CONLANG

Oh wait, you're complaining about 32-bit support, aren't you? You'll
need to:
dpkg --add-architecture i386
apt-get update
apt-get install libcurl4-openssl-dev:i386
Under Debian multiarch, packages for different architectures are
installed separately, so you have to explicitly install the same package
for both architectures if you need it for both.

That's only available on Debian 7 and later. He uses Debian 6.
--
/Jacob Carlborg

A couple of issues/questions:
* Mac OS X uses "dylib" as the extension for dynamic libraries
* On Mac OS X Xcode is required. It's possible to just download the
command line tools but I don't think that has been verified to work
* " --combine-zip (Posix-only) Combine all platform-specific archives in
current directory into cross-platform zip archive.
Cannot be used on Windows because the symlinks would be
destroyed. Implies --skip-package."
Can't zip for Windows handle symlinks? Windows Vista (I think) and later
supports symlinks.
* The extra files should be put under version control
* Does this builds a release for FreeBSD 64bit? If not, it should
--
/Jacob Carlborg

A couple of issues/questions:
* Mac OS X uses "dylib" as the extension for dynamic libraries

Thanks. Will fix.

* On Mac OS X Xcode is required. It's possible to just download the
command line tools but I don't think that has been verified to work

I'll make a note in the docs section.

* " --combine-zip (Posix-only) Combine all platform-specific archives
in current directory into cross-platform zip archive.
Cannot be used on Windows because the symlinks would be
destroyed. Implies --skip-package."
Can't zip for Windows handle symlinks? Windows Vista (I think) and
later supports symlinks.

In an attempt to keep the platform differences as minimal as possible,
I have it set up to automatically download and use Info-ZIP and 7zip on
Windows, the same programs typically used on Posix. Unfortunately,
these did not appear to handle symlinks correctly. Anyone know of one I
could use that does?

Actually, that's a good point, I have to double-check that. I don't
remember adding any code to skip 64-bit on FreeBSD, *but* I was careful
to make everything match the v2.063.2 release zip as closely as
possible, so I might have omitted it under the assumption that "Well,
this release zip doesn't have bin64 or lib64 for bsd, so I guess it's
not considered ready for prime-time yet."

Actually, that's a good point, I have to double-check that. I don't
remember adding any code to skip 64-bit on FreeBSD, *but* I was careful
to make everything match the v2.063.2 release zip as closely as
possible, so I might have omitted it under the assumption that "Well,
this release zip doesn't have bin64 or lib64 for bsd, so I guess it's
not considered ready for prime-time yet."

The latest release of DMD for FreeBSD only includes 32bit binaries.
--
/Jacob Carlborg

Actually, that's a good point, I have to double-check that. I don't
remember adding any code to skip 64-bit on FreeBSD, *but* I was
careful to make everything match the v2.063.2 release zip as
closely as possible, so I might have omitted it under the
assumption that "Well, this release zip doesn't have bin64 or lib64
for bsd, so I guess it's not considered ready for prime-time yet."

The latest release of DMD for FreeBSD only includes 32bit binaries.

Yea, that's why it's possible I might have excluded 64-bit for FreeBSD
(and then forgot I did so). I'll check.

Actually, that's a good point, I have to double-check that. I
don't remember adding any code to skip 64-bit on FreeBSD, *but* I
was careful to make everything match the v2.063.2 release zip as
closely as possible, so I might have omitted it under the
assumption that "Well, this release zip doesn't have bin64 or
lib64 for bsd, so I guess it's not considered ready for
prime-time yet."

The latest release of DMD for FreeBSD only includes 32bit binaries.

Yea, that's why it's possible I might have excluded 64-bit for FreeBSD
(and then forgot I did so). I'll check.

Just checked: The tool *is* set up to build and package 64-bit on
FreeBSD (assuming the 32-bit stuff doesn't fail before it gets to the
64-bit pass).

In an attempt to keep the platform differences as minimal as possible,
I have it set up to automatically download and use Info-ZIP and 7zip on
Windows, the same programs typically used on Posix. Unfortunately,
these did not appear to handle symlinks correctly. Anyone know of one I
could use that does?

Why don't you use std.zip? Although I don't know if that handles symlinks.
--
/Jacob Carlborg

In an attempt to keep the platform differences as minimal as
possible, I have it set up to automatically download and use
Info-ZIP and 7zip on Windows, the same programs typically used on
Posix. Unfortunately, these did not appear to handle symlinks
correctly. Anyone know of one I could use that does?

Why don't you use std.zip? Although I don't know if that handles
symlinks.

Judging by the doc page for that, it seems like 1. it's not quite as
mature as Info-ZIP, and 2. It doesn't actually read/write files
directly, just does the in-memory part. Naturally, I could still use it
and use std.file for the i/o, but I'm not sure it would be worth the
bother for this.
It may very well be worth attempting later though.

Judging by the doc page for that, it seems like 1. it's not quite as
mature as Info-ZIP, and 2. It doesn't actually read/write files
directly, just does the in-memory part. Naturally, I could still use it
and use std.file for the i/o, but I'm not sure it would be worth the
bother for this.

Right, I thought I saw an API for reading/writing files as well.
--
/Jacob Carlborg

I think I just copy-pasted it from somewhere. It is really
better to do
the "git github.com:blah/blah.git"?

As far as I know, yes.

git:// is faster
https:// is less likely to be blocked by various firewalls and
filters
Choice is yours :)

Ok, in that case, it sounds like https is better for right now, but it
would be good to have an optional flag to use git:// instead. Actually,
maybe it should try git:// and if it fails, then try https://. But that
can probably wait. The important thing is to get this working.

- OSX 10.7: Works fine. (At least it did last time I tried it, a
couple weeks ago. It *could* have regressed but I won't have access
to the machine again until maybe Sunday.)

As far as I can see, dmd.conf is missing. I used the following
command:
./create_dmd_release master
--extras=./dmd-localextras/localextras-osx --archive-zip

They're in the built-in "extras":
<https://github.com/Abscissa/installer/tree/create-zip/create_dmd_release/extras/osx/dmd2/osx/bin>
The short answer is: "That problem only happens because
create_dmd_release isn't in the official github repo yet".
Long answer and workaround:
When create_dmd_release clones all the repos from github, everything
inside the *cloned* "installer/create_dmd_release/extras/all" and
"installer/create_dmd_release/extras/[current os]" will be
automatically added to the release (During the "package" phase, it grabs
these "extras" from the cloned "create_dmd_release" *instead* of from
the copy of "create_dmd_release" being run so the "extras" files can be
different for different releases of DMD).
But since create_dmd_release, and the attached "extras", hasn't yet
been merged into the official "D-Programming-Language/installer" repo,
you're missing that "extras" directory, including the dmd.conf files.
When create_dmd_release is all official and merged in, that won't be
an issue. But until then, you have to stop the process after it clones
everything, manually copy the "create_dmd_release/extras" directory to
"[temp]/.create_dmd_release/installer/create_dmd_release/extras" and
then re-run create_dmd_release with the --skip-clone flag.
Or better yet, instead of halting create_dmd_release, copying
"extras", and re-runnnig with --skip-clone (which is kind of a pain),
you could just copy the contents of both
"installer/create_dmd_release/extras/all" and
"installer/create_dmd_release/extras/[current os]" into your
"localextras-[current os]". That will work too. But again, this work
around is *only* needed because create_dmd_release isn't in the
official github repo yet.
I *could* add an option to specify an alternate repo to clone, but I'm
not sure it's worth it since it would become useless as soon as this
gets merged.

I've noticed the you support MSVC builds but require some
environment variables to be set. As part of my installer#22 pull
request[1] I worked out how to retrieve both the Visual C and SDK
paths from the registry. You could retrieve them from there
instead of requiring environment variables if you'd like.
Obviously just getting everything working in the release
generator (great work, by the way) comes first but this would be
one less thing people would need to do before they can use it).
Visual Studio 12.0 (2013):
HKEY_LOCAL_MACHINE\Software\Microsoft\VisualStudio\12.0\Setup\VC\ProductDir
(libraries are in <productdir>\lib (32-bit) or
<productdir>\lib\amd64)
Visual Studio 11.0 (2012):
HKEY_LOCAL_MACHINE\Software\Microsoft\VisualStudio\11.0\Setup\VC\ProductDir
(libraries are in <productdir>\lib (32-bit) or
<productdir>\lib\amd64)
Visual Studio 10.0 (2013):
HKEY_LOCAL_MACHINE\Software\Microsoft\VisualStudio\10.0\Setup\VC\ProductDir
(libraries are in <productdir>\lib (32-bit) or
<productdir>\lib\amd64)
Windows Kit 8.1:
HKEY_LOCAL_MACHINE\Software\Microsoft\Windows Kits\Installed
Roots\KitsRoot81
(libraries are in <kitroot>\Lib\winv6.3\um\{x86,x64,arm})
Windows Kit 8.0:
HKEY_LOCAL_MACHINE\Software\Microsoft\Windows Kits\Installed
Roots\KitsRoot
(libraries are in <kitroot>\Lib\winv6.3\um\{x86,x64,arm})
Microsoft SDK 7.1A:
HKEY_LOCAL_MACHINE\Software\Microsoft\Microsoft
SDKs\Windows\v7.1A\InstallationFolder
(libraries are in <sdkroot>\Lib (32-bit) or <sdkroot>\Lib\x64)
Microsoft SDK 7.0A:
HKEY_LOCAL_MACHINE\Software\Microsoft\Microsoft
SDKs\Windows\v7.0A\InstallationFolder
(libraries are in <sdkroot>\Lib (32-bit) or <sdkroot>\Lib\x64)
1. https://github.com/D-Programming-Language/installer/pull/22

I've noticed the you support MSVC builds but require some
environment variables to be set. As part of my installer#22 pull
request[1] I worked out how to retrieve both the Visual C and SDK
paths from the registry. You could retrieve them from there
instead of requiring environment variables if you'd like.
Obviously just getting everything working in the release
generator (great work, by the way) comes first but this would be
one less thing people would need to do before they can use it).

- FreeBSD 9.1 (64-bit OS): No idea, I don't have access to a
64-bit
FreeBSD machine, and my stupid Intel CPU lacks the ability to
run a
64-bit OS in a VM.

It tried the tool on FreeBSD (PC-BSD) 64bit. I got this error:
Building Tools 64-bit
std.md5 is scheduled for deprecation. Please use std.digest.md
instead
gmake: *** No rule to make target
`../dlang.org/web/expression.html', needed by `expression.tag'.
Stop.
create_dmd_release: Error: Command failed (ran from dir
'/tmp/.create_dmd_release/tools'): gmake MODEL=64 dman
DMD=../dmd/src/dmd DOC=../dlang.org/web
PHOBOSDOC=../dlang.org/web/phobos -f posix.mak > /dev/null
I don't know if it's me that did something wrong or if it's your
tool. I had to disable the 32bit builds.

One of the tools (dman, IIRC) requires the docs to be built, and since
the docs are independent of 32/64-bit, they're built during the 32-bit
pass (which you skipped).
I've decided I'm going to add proper support for 32-bit-only and
64-bit-only via optional cmdline flags. That should make a lot of
things easier and help mitigate difficulties with multilib.

I've decided I'm going to add proper support for 32-bit-only and
64-bit-only via optional cmdline flags. That should make a lot of
things easier and help mitigate difficulties with multilib.

Yeah, that's will make things a lot easier.

Ok, I've just added optional switches --only-32 and --only-64. But I'm
heading out right now so I didn't test it very well, and the
--combine-* flags currently ignore the new non-multilib archives
generated by those switches. I'll probably be able to take care of
those later tonight, but hopefully this should make things easier.

I've decided I'm going to add proper support for 32-bit-only and
64-bit-only via optional cmdline flags. That should make a lot of
things easier and help mitigate difficulties with multilib.

Yeah, that's will make things a lot easier.

Ok, I've just added optional switches --only-32 and --only-64. But I'm
heading out right now so I didn't test it very well, and the
--combine-* flags currently ignore the new non-multilib archives
generated by those switches. I'll probably be able to take care of
those later tonight, but hopefully this should make things easier.

The --combine-* flags now handle the 32-bit-only and 64-bit-only
archives correctly.
I've also adjusted it to clone using the git protocol, and
automatically fallback to https if that fails.

- Windows 7 (64-bit OS): This is where I'm having the biggest trouble.
I can coaxing it to handle v2.063.2 just fine (ie, if I update the
"tools" makefiles as described above, and comment out all the
"libcurl" and "chm" stuff.) However, on master, after it compiles
DMD/druntime/phobos, the resulting DMD/phobos can't compile anything
that uses phobos because OPTLINK will spew out a bunch of errors. I
am *completely* at a loss on this one. It seems like an sc.ini
issue, but I've spent days checking everything and I still can't make
heads or tails of it.

That issue is still blocking the release generator (seems to be the
same issue at least one other person was already having):
http://d.puremagic.com/issues/show_bug.cgi?id=10729
Can anyone who is able to build a *working* 32-bit DMD/phobos from
master on Windows pop into that issue page and see what this does for
them?:
mkdir junk
create_dmd_release master --extras=junk --only-32 -v

- Windows 7 (64-bit OS): This is where I'm having the biggest trouble.
I can coaxing it to handle v2.063.2 just fine (ie, if I update the
"tools" makefiles as described above, and comment out all the
"libcurl" and "chm" stuff.) However, on master, after it compiles
DMD/druntime/phobos, the resulting DMD/phobos can't compile anything
that uses phobos because OPTLINK will spew out a bunch of errors. I
am *completely* at a loss on this one. It seems like an sc.ini
issue, but I've spent days checking everything and I still can't make
heads or tails of it.

That issue is still blocking the release generator (seems to be the
same issue at least one other person was already having):
http://d.puremagic.com/issues/show_bug.cgi?id=10729
Can anyone who is able to build a *working* 32-bit DMD/phobos from
master on Windows pop into that issue page and see what this does for
them?:
mkdir junk
create_dmd_release master --extras=junk --only-32 -v

The generated sc.ini in dmd/src is no good, the LIB environment variable
should be set like this:
LIB="% P%\..\..\phobos"
Even if you do so, it does not work because optlink is also looking for
an sc.ini (contrary or in addition to the documentation that mentions
optlink.cfg) and extracts the LIB environment variable. If the link.exe
in the PATH is from a different version of dmd, the wrong phobos.lib is
taken.
It is easiest to always put link.exe into the same directory as dmd.exe.
While you are at it, I'd recommend an sc.ini that looks about like this
in the release:
; environment for both 32/64 bit
[Environment]
LIB="% P%\..\lib"
DFLAGS="-I% P%\..\..\src\phobos" "-I% P%\..\..\src\druntime\import"
LINKCMD=% P%\link.exe
[Environment64]
; VS2008 and VS2012
LINKCMD64=%VCINSTALLDIR%\bin\amd64\link.exe
; VS2012
; LINKCMD64=%VCINSTALLDIR%\bin\x86_amd64\link.exe
; needed with /DEBUG to find mspdb*.dll
PATH=%PATH%;%VCINSTALLDIR%\bin\x86_amd64;%VCINSTALLDIR%\..\Common7\IDE
; needed to avoid COMDAT folding (bugzilla 10664)
DFLAGS=%DFLAGS% -L/OPT:NOICF
LIB="% P%\..\lib64"
;;;; search path for C Runtime libraries
; the following lib path works with VS2008, VS2010 and VS2012
LIB=%LIB%;"%VCINSTALLDIR%\lib\amd64"
;;;; search path for Platform libraries
; the following lib path works with Windows SDK 6.x and 7.x
LIB=%LIB%;%WindowsSdkDir%\Lib\x64
; the following lib path works with Windows SDK 8.0
LIB=%LIB%;%WindowsSdkDir%\Lib\win8\um\x64
This includes the install of win64 libraries into a lib64 folder instead
of mixing it with OMF libraries in the lib folder.
3 more nitpicks:
- the working directory defaultWorkDir is never created
- I get an error that '.' cannot be removed, but I can continue
answering 'n' to the following question for retry
- my copies of the Windows SDKs 6 and 7 don't have folders License,
Redist, Samples and Setup. I would just skip these tests.

- Windows 7 (64-bit OS): This is where I'm having the biggest
trouble. I can coaxing it to handle v2.063.2 just fine (ie, if I
update the "tools" makefiles as described above, and comment out
all the "libcurl" and "chm" stuff.) However, on master, after it
compiles DMD/druntime/phobos, the resulting DMD/phobos can't
compile anything that uses phobos because OPTLINK will spew out a
bunch of errors. I am *completely* at a loss on this one. It seems
like an sc.ini issue, but I've spent days checking everything and
I still can't make heads or tails of it.

That issue is still blocking the release generator (seems to be the
same issue at least one other person was already having):
http://d.puremagic.com/issues/show_bug.cgi?id=10729
Can anyone who is able to build a *working* 32-bit DMD/phobos from
master on Windows pop into that issue page and see what this does
for them?:
mkdir junk
create_dmd_release master --extras=junk --only-32 -v

The generated sc.ini in dmd/src is no good, the LIB environment
variable should be set like this:
LIB="% P%\..\..\phobos"
Even if you do so, it does not work because optlink is also looking
for an sc.ini (contrary or in addition to the documentation that
mentions optlink.cfg) and extracts the LIB environment variable. If
the link.exe in the PATH is from a different version of dmd, the
wrong phobos.lib is taken.
It is easiest to always put link.exe into the same directory as
dmd.exe.

Ahh, ok I had been wondering about all of that, and even toyed around
with those changes, but I still couldn't get it to work. But by then, I
had already been reduced to kinda tossing around theories
semi-randomly, so maybe I still messed it up. I'll try again - at least
this time I know ;)

While you are at it, I'd recommend an sc.ini that looks about like
this in the release:
[...]

Thanks, I'll update it.

3 more nitpicks:
- the working directory defaultWorkDir is never created

Really? Hmm:
- Is it possible that might be a permissions issue? Since this isn't
really an end-user tool, I didn't feel a need to make sure it
worked with a limited user account. What temp directory
is reported when you do "create_dmd_release --help"?
- Are you using --use-clone=path? If so, then it won't create
defaultWorkDir unless it needs to - although I might have missed
something there, so I'll check into that.

- I get an error that '.' cannot be removed, but I can continue
answering 'n' to the following question for retry

Ok now that seems bizarre. Can you post a log or screenshot of that?
I have no idea what part of the process might be doing that -
definitely something involving an external tool being invoked, because I
didn't add any interactive prompts to it.
You're not running it *from* the [your temp dir]/.create_dmd_release
directory, are you?

- my copies of the Windows SDKs 6 and 7 don't have folders License,
Redist, Samples and Setup. I would just skip these tests.

3 more nitpicks:
- the working directory defaultWorkDir is never created

Really? Hmm:
- Is it possible that might be a permissions issue? Since this isn't
really an end-user tool, I didn't feel a need to make sure it
worked with a limited user account. What temp directory
is reported when you do "create_dmd_release --help"?
- Are you using --use-clone=path? If so, then it won't create
defaultWorkDir unless it needs to - although I might have missed
something there, so I'll check into that.

Actually I could not reproduce this problem now. Could it be that this
happened with a version from a few days ago and was already fixed?

- I get an error that '.' cannot be removed, but I can continue
answering 'n' to the following question for retry

Ok now that seems bizarre. Can you post a log or screenshot of that?
I have no idea what part of the process might be doing that -
definitely something involving an external tool being invoked, because I
didn't add any interactive prompts to it.
You're not running it*from* the [your temp dir]/.create_dmd_release
directory, are you?

The generated sc.ini in dmd/src is no good, the LIB environment
variable should be set like this:
LIB="% P%\..\..\phobos"
Even if you do so, it does not work because optlink is also looking
for an sc.ini (contrary or in addition to the documentation that
mentions optlink.cfg) and extracts the LIB environment variable. If
the link.exe in the PATH is from a different version of dmd, the
wrong phobos.lib is taken.
It is easiest to always put link.exe into the same directory as
dmd.exe.

Aha! That was it! I don't know why my experiments doing that didn't
work before (I must have messed up something else at the time), but
fixing sc.ini and copying OPTLINK to the same dir fixed that problem!
I still have another small issue with the chm generation, and some
thorough testing to do, but things are looking REALLY good now!

- the working directory defaultWorkDir is never created

Really? Hmm:
- Is it possible that might be a permissions issue? Since this isn't
really an end-user tool, I didn't feel a need to make sure it
worked with a limited user account. What temp directory
is reported when you do "create_dmd_release --help"?
- Are you using --use-clone=path? If so, then it won't create
defaultWorkDir unless it needs to - although I might have missed
something there, so I'll check into that.

Actually I could not reproduce this problem now. Could it be that
this happened with a version from a few days ago and was already
fixed?

That turned out to be a regression I had introduced with a recent
change. I didn't catch it because, for me, the directory was already
there. It went away for you because the directory must have gotten
created on another run.
It's fixed on GitHub now.

- I get an error that '.' cannot be removed, but I can continue
answering 'n' to the following question for retry

Ok now that seems bizarre. Can you post a log or screenshot of that?
I have no idea what part of the process might be doing that -
definitely something involving an external tool being invoked,
because I didn't add any interactive prompts to it.
You're not running it*from* the [your temp dir]/.create_dmd_release
directory, are you?

Ok, I see what's happening. It's git that's doing that:
This create_dmd_release tool *tries* to clone using git/ssh protocol
because it's supposed to be faster, albeit less compatible. But if that
fails (as it does for both you and me...some rights/certificate issue
apparently), then the script tries again using the slower, but more
reliable HTTPS protocol.
But here's the thing: When git tries to clone, and the clone fails,
then git's nice about it and tries to clean up the mess it left behind
by deleting the directory with the failed clone. Unfortunately, Windows
likes to randomly hold file locks open (or maybe it's a bug in your
version of git that's holding an old file handle open). So git's unable
to delete the failed-clone directory and asks if you want to try again.
I can't find a cmdline switch to tell "git clone" to always retry
(or to never retry), so I don't think there's much create_dmd_release
can do about it, short of reading git's output, scanning for that
question, and sending a 'y'. But that seems like overkill. It may be
better to just let people answer on their own if they get that.

Ok, I see what's happening. It's git that's doing that:
This create_dmd_release tool*tries* to clone using git/ssh protocol
because it's supposed to be faster, albeit less compatible. But if that
fails (as it does for both you and me...some rights/certificate issue
apparently), then the script tries again using the slower, but more
reliable HTTPS protocol.
But here's the thing: When git tries to clone, and the clone fails,
then git's nice about it and tries to clean up the mess it left behind
by deleting the directory with the failed clone. Unfortunately, Windows
likes to randomly hold file locks open (or maybe it's a bug in your
version of git that's holding an old file handle open). So git's unable
to delete the failed-clone directory and asks if you want to try again.
I can't find a cmdline switch to tell "git clone" to always retry
(or to never retry), so I don't think there's much create_dmd_release
can do about it, short of reading git's output, scanning for that
question, and sending a 'y'. But that seems like overkill. It may be
better to just let people answer on their own if they get that.

I noticed that it is the git process that has a handle on that
directory. It seems to able to delete the directory if you run git from
the parent directory and pass "dmd" instead of ".".

On 20.09.2013 09:29, Nick Sabalausky wrote:
I noticed that it is the git process that has a handle on that
directory. It seems to able to delete the directory if you run git
from the parent directory and pass "dmd" instead of ".".

Hmm, well it's been working fine on my Win machine as-is, but I'll look
into doing that. I think I had some reason for doing it this way, but
it probably wasn't anything major.

I noticed that it is the git process that has a handle on that
directory. It seems to able to delete the directory if you run git
from the parent directory and pass "dmd" instead of ".".

Hmm, well it's been working fine on my Win machine as-is, but I'll
look into doing that. I think I had some reason for doing it this
way, but it probably wasn't anything major.

Just before I was about to make that fix, I started experiencing the
same problem, too.
I've made the change you suggested and it seems to work for me. Can you
verify?

I retried and had the issue with access to %TEMP%/".create_dmd_release
problem problem during startup again. This time it was TortoiseGitCache
holding a handle to that directory. Killing that process helped.
Then, the 32-bit build ran flawlessly if I
- add the dmc binary path to the PATH environment variable
- add the path to some existing dmd installation to the path. rdmd is
needed to run get_dlibcurl32.bat. [OT: I noticed it uses a VB script to
do the download. Shudder ;-) Here is a http-get function without needing
curl:
https://github.com/D-Programming-Language/visuald/blob/master/stdext/httpget.d
, though I don't know how usable it is. ]
I usually don't have these paths added because the folders contain too
many executables that try to replace more useful versions of them
(lib,link,shell,make). I'd prefer if you could specify the path to dmc
as an option, but it's not a big deal as the task of building the
release is pretty special anyway.
It would be nice if rdmd is used from the current build, though.
After the successful build I tried the 64-bit build. It stopped here:
Building Druntime 64-bit
Entering dir:
C:\Users\Rainer\AppData\Local\Temp\.create_dmd_release\druntime
Running: make MODEL=64 DMD=../dmd/src/dmd -f win64.mak
mkdir import\core\stdc
A subdirectory or file import\core\stdc already exists.
mkdir import\core\sys\freebsd\sys
A subdirectory or file import\core\sys\freebsd\sys already exists.
mkdir import\core\sys\linux\sys
A subdirectory or file import\core\sys\linux\sys already exists.
mkdir import\core\sys\osx\mach
A subdirectory or file import\core\sys\osx\mach already exists.
mkdir import\core\sys\posix\arpa
A subdirectory or file import\core\sys\posix\arpa already exists.
mkdir import\core\sys\posix\net
A subdirectory or file import\core\sys\posix\net already exists.
mkdir import\core\sys\posix\netinet
A subdirectory or file import\core\sys\posix\netinet already exists.
mkdir import\core\sys\posix\sys
A subdirectory or file import\core\sys\posix\sys already exists.
mkdir import\core\sys\windows
A subdirectory or file import\core\sys\windows already exists.
mkdir import\etc\linux
A subdirectory or file import\etc\linux already exists.
"\Program Files (x86)\Microsoft Visual Studio 10.0\VC"\bin\amd64\cl -c
/Z7 /I"\Program Files (x86)\Microsoft Visual Studio 10.0\VC"\INCLUDE
/I"\Program Files (x86)\Microsoft SDKs\Windows\v7.0A"\Include
src\core\stdc\errno.c -Foerrno_c.obj
Error: '\Program Files (x86)\Microsoft Visual Studio
10.0\VC\bin\amd64\cl' not found
Entering dir: C:\tmp\d\installer2\create_dmd_release\x
create_dmd_release: Error: Command failed (ran from dir
'C:\Users\Rainer\AppData\Local\Temp\.create_dmd_release\druntime'): make
MODEL=64 DMD=../dmd/src/dmd -f win64.mak
This happens because the VCDIR variable is not passed to the makefile
(to work with VS2012+ you'll have to pass CC and AR explicitely as the
path to the binaries inside the VS installation has changed).

I retried and had the issue with access to
%TEMP%/".create_dmd_release problem problem during startup again.
This time it was TortoiseGitCache holding a handle to that directory.
Killing that process helped.

I admit, the VBScript thing is a bit of a hack. On the upside, the
actual connection/dload is handled by a lib that's been in Windows for
some time, so hopefully it should be pretty battle-hardened. I'll
definitely keep yours in mind in the future, I think that's awesome to
have. But in the interest of getting this (and some other things) out
the door I think I'll keep it as-is for now.

I usually don't have these paths added because the folders contain
too many executables that try to replace more useful versions of them
(lib,link,shell,make). I'd prefer if you could specify the path to
dmc as an option, but it's not a big deal as the task of building the
release is pretty special anyway.

Ugh, I wish MS would just pick a design as stick with it :/... I'll try
to take care of that. On the bright side though, this *should* be the
final critical issue blocking create_dmd_release from meeting its basic
charter and being "pull request"-ready. Everything else seems to be
working fine now on Windows and various Posixen in all my tests so far,
so fingers crossed...

I retried and had the issue with access to
%TEMP%/".create_dmd_release problem problem during startup again.
This time it was TortoiseGitCache holding a handle to that directory.
Killing that process helped.

Yea, Windows is a pain with file/directory locking :/

Well, actually Windows is way better than POSIX in this regard, because
Windows like a few other systems does implement locking at the kernel level.
Whereas POSIX locking relies in a form of userspace cooperative locking
that does not prevent someone to come in between and mess up with the files.
Nothing like deleting a file or write to it, while it is in use by other
processes, without any means to prevent such actions in a portable way.
--
Paulo

I retried and had the issue with access to
%TEMP%/".create_dmd_release problem problem during startup again.
This time it was TortoiseGitCache holding a handle to that
directory. Killing that process helped.

Yea, Windows is a pain with file/directory locking :/

Well, actually Windows is way better than POSIX in this regard,
because Windows like a few other systems does implement locking at
the kernel level.
Whereas POSIX locking relies in a form of userspace cooperative
locking that does not prevent someone to come in between and mess up
with the files.
Nothing like deleting a file or write to it, while it is in use by
other processes, without any means to prevent such actions in a
portable way.

Oh right, that's certainly true, which I why I have a bit of a
love/hate thing for how Windows does it. It's safer and more "correct",
but also frequently a big pain. Stoopid tradeoffs...

Ugh, I wish MS would just pick a design as stick with it :/... I'll
try to take care of that. On the bright side though, this *should* be
the final critical issue blocking create_dmd_release from meeting its
basic charter and being "pull request"-ready. Everything else seems
to be working fine now on Windows and various Posixen in all my tests
so far, so fingers crossed...

Yes, but I wasn't sure how to handle selecting the version of optlink:
For packaging the latest new version of DMD, then yea, the latest
optlink should probably be chosen. But for earlier versions of DMD,
shouldn't it grab the same version of optlink that had originally come
with it? If so, how? By requiring Walter to add dmd release tags to
optlink just like for dmd/phobos/druntime/etc? By manually maintaining
a table somewhere to lookup the proper oplink commit for a given DMD
version?

tl;dr: The main things I need help with: Installing 32-bit libcurl on
64-bit Debian, testing on 64-bit FreeBSD, and figuring out WTF is so
screwed up on Windows.
It's almost finished, but I've hit some problems I'm having a hell of
a time figuring out. I'm getting a kinda brain-fried and need some
help with it: