NetBSD Problem Report #42704

From kre@munnari.OZ.AU Sat Jan 30 11:19:01 2010
Return-Path: <kre@munnari.OZ.AU>
Received: from mail.netbsd.org (mail.netbsd.org [204.152.190.11])
by www.NetBSD.org (Postfix) with ESMTP id 2371263B886
for <gnats-bugs@gnats.NetBSD.org>; Sat, 30 Jan 2010 11:19:01 +0000 (UTC)
Message-Id: <201001301118.o0UBIpKD002548@jade.coe.psu.ac.th>
Date: Sat, 30 Jan 2010 18:18:51 +0700 (ICT)
From: kre@munnari.OZ.AU
To: gnats-bugs@gnats.NetBSD.org
Subject: Adding support for bmake as a pkgsrc tool (ala gmake)
X-Send-Pr-Version: 3.95
>Number: 42704
>Category: pkg
>Synopsis: Adding support for bmake as a pkgsrc tool (ala gmake)
>Confidential: no
>Severity: serious
>Priority: high
>Responsible: pkg-manager
>State: closed
>Class: change-request
>Submitter-Id: net
>Arrival-Date: Sat Jan 30 11:20:00 +0000 2010
>Closed-Date: Mon Dec 29 09:02:49 +0000 2014
>Last-Modified: Sun Mar 08 19:15:00 +0000 2015
>Originator: Robert Elz
>Release: NetBSD 4.0 / i386 (pkgsrc current 2010-01-30)
>Organization:
Prince of Songkla University
>Environment:
System: NetBSD jade.coe.psu.ac.th 4.0_STABLE NetBSD 4.0_STABLE (JADE-1.696-20080517) #9: Fri May 23 18:55:13 ICT 2008 kre@jade.coe.psu.ac.th:/usr/obj/4/kernels/JADE i386
Architecture: i386
Machine: i386
>Description:
There are more and more packages (gnome derived ones mostly)
that are triggering the bug in NetBSD 4's make program related
to long input lines (usually found in a .deps/*.po type file)
Since NetBSD 4 is still supposed to be supported by pkgsrc,
and since it seems unlikely that there will ever be a new NetBSD 4
release to incorporate a fixed version of make, all the packages
that encounter this problem need to use a pkgsrc supplied
version of make rather than the system version (when building
in a NetBSD 4 environment).
Until now, the only alternative has been gmake, and while gmake
and bmake are largely compatible, "largely" should probably be
in quotes - it seems to me that if a package builds using the
standard NetBSD make on NetBSD 5 (or later) it should use as close
to that as possible on NetBSD 4.
Since we have devel/bmake already in pkgsrc, I thought I'd see how
easy it would be to make that available as a build tool for
pkgsrc (as gmake is, for packages that have gmake extensions in
their Makefiles). This turns out to be easy...
What's more, it seems to work...
>How-To-Repeat:
I'm not sur ethis section makes much sense in a PR that's a
change request, but to see an example of a package that has
the problem today, try time/gtodo-applet - attempt to build
that on NetBSD 4, and expect to see ...
===> Installing for gtodo-applet-0.1nb18
=> Generating pre-install file lists
Making install in src
[1] Segmentation fault (core dumped) (cd ${subdir} &&...
*** Error code 1
Stop.
make: stopped in /pkg_comp/obj/pkgsrc/time/gtodo-applet/4x/gtodo-applet-0.1
the core file dumped is from make, and sure enough, there's a
BIG .deps/xxx.po file (I forget its name, it isn't important here)
After the following patch is applied, adding
USE_TOOLS+= bmake
to gtodo-applet's Makefile allows that package to build successfully.
Quite possibly using gmake would as well - since these are mostly
gnome applications that encounter the bug, it would be surprising if
gmake wouldn't work ... neevrtheless, the NetBSD 4 make bug could be
experienced anywhere, so I submit that being able to use bmake as
a tool is a useful function to add.
>Fix:
The following patch (applied in pkgsrc/mk) will add support
for adding bmake as a tool.
Index: tools/defaults.mk
===================================================================
RCS file: /cvsroot/NetBSD/pkgsrc/mk/tools/defaults.mk,v
retrieving revision 1.57
diff -u -r1.57 defaults.mk
--- tools/defaults.mk 8 Jul 2009 12:51:16 -0000 1.57
+++ tools/defaults.mk 30 Jan 2010 10:48:11 -0000
@@ -55,6 +55,7 @@
_TOOLS_VARNAME.basename= BASENAME
_TOOLS_VARNAME.bash= BASH
_TOOLS_VARNAME.bison-yacc= YACC
+_TOOLS_VARNAME.bmake= BMAKE
_TOOLS_VARNAME.byacc= YACC
_TOOLS_VARNAME.bzcat= BZCAT
_TOOLS_VARNAME.bzip2= BZIP2
Index: tools/make.mk
===================================================================
RCS file: /cvsroot/NetBSD/pkgsrc/mk/tools/make.mk,v
retrieving revision 1.13
diff -u -r1.13 make.mk
--- tools/make.mk 17 Jul 2005 21:36:24 -0000 1.13
+++ tools/make.mk 30 Jan 2010 10:54:12 -0000
@@ -49,6 +49,8 @@
# Define MAKE_PROGRAM to be the "make" used to build the package.
.if !empty(_USE_TOOLS:Mgmake)
MAKE_PROGRAM= ${GMAKE}
+.elif !empty(_USE_TOOLS:Mbmake)
+MAKE_PROGRAM= ${BMAKE}
.elif !empty(_USE_TOOLS:Mimake) && defined(IMAKE_MAKE)
MAKE_PROGRAM= ${IMAKE_MAKE}
.else
Index: tools/replace.mk
===================================================================
RCS file: /cvsroot/NetBSD/pkgsrc/mk/tools/replace.mk,v
retrieving revision 1.223
diff -u -r1.223 replace.mk
--- tools/replace.mk 20 Jan 2010 14:47:30 -0000 1.223
+++ tools/replace.mk 30 Jan 2010 10:46:06 -0000
@@ -258,6 +258,17 @@
TOOLS_VALUE_GNU.bison-yacc= ${TOOLS_CMDLINE.bison-yacc}
.endif
+.if !defined(TOOLS_IGNORE.bmake) && !empty(_USE_TOOLS:Mbmake)
+. if !empty(PKGPATH:Mdevel/bmake)
+MAKEFLAGS+= TOOLS_IGNORE.bmake=
+. elif !empty(_TOOLS_USE_PKGSRC.bmake:M[yY][eE][sS])
+TOOLS_DEPENDS.bmake?= bmake>=20090909:../../devel/bmake
+TOOLS_CREATE+= bmake
+TOOLS_FIND_PREFIX+= TOOLS_PREFIX.bmake=bmake
+TOOLS_PATH.bmake= ${TOOLS_PREFIX.bmake}/bin/bmake
+. endif
+.endif
+
.if !defined(TOOLS_IGNORE.bsdtar) && !empty(_USE_TOOLS:Mbsdtar)
. if !empty(PKGPATH:Marchivers/bsdtar)
MAKEFLAGS+= TOOLS_IGNORE.bsdtar=
>Release-Note:>Audit-Trail:
From: "OBATA Akio" <obache@netbsd.org>
To: gnats-bugs@netbsd.org
Cc:
Subject: Re: pkg/42704: Adding support for bmake as a pkgsrc tool (ala gmake)
Date: Sat, 30 Jan 2010 20:31:47 +0900
How about install devel/bmake and use "bmake" to build packages instead
of builtin "make" on NetBSD-4?
From: Robert Elz <kre@munnari.OZ.AU>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: pkg/42704: Adding support for bmake as a pkgsrc tool (ala gmake)
Date: Sat, 30 Jan 2010 19:08:42 +0700
Date: Sat, 30 Jan 2010 11:35:01 +0000 (UTC)
From: "OBATA Akio" <obache@NetBSD.org>
Message-ID: <20100130113501.D56D863BBFC@www.NetBSD.org>
| How about install devel/bmake and use "bmake" to build packages instead
| of builtin "make" on NetBSD-4?
If you mean making bmake be the default make to use for NetBSD 4 systems
(so it just always gets installed there very early, much like the latest
version of pkgtools/pkg_install does) then that sounds like an excellent
idea - it would certainly avoid someone needing to trip over each and every
package that triggers the bug, then add patches to them all (ie: that would
make PR 42705 just vanish into the dust).
The only downside I see is that this would add some overhead to NetBSD 4
package builders - so far I think I've come across about 8 or 10 packages
that cause make core dumps (and not all of those are verified to be this
problem, some are still at the state of "something dumped core - needs
investigating") of the 8000 or so packages that exist (ie: unless you happen
to want to build one of the few, the standard system make is adequate.)
I guess I lean towards "screw overhead, make it correct" personally, but
I'm not sure everyone does (then again, how many NetBSD 4 package builders
remain these days?)
I'd still suggest that having bmake available as a build tool is a good
idea, aside from dealing with the bug in NetBSD 4's make, this would also
allow pkgsrc developers to make use of the latest additions to bmake, which
are currently more or less off limits - as anything recently added to make
is certainly not going to exist in old, but still supported, systems.
For this usage though, more support, beyond my pkgsrc abilities, is probably
needed, so that bmake only gets installed on systems that need the specific'
feature (like testing what version is installed on the base system, and
installing devel/bmake only if the base is too old for the feature used by
a particular package.)
kre
From: Joerg Sonnenberger <joerg@britannica.bec.de>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: pkg/42704: Adding support for bmake as a pkgsrc tool (ala
gmake)
Date: Wed, 3 Feb 2010 00:54:56 +0100
On Sat, Jan 30, 2010 at 11:20:00AM +0000, kre@munnari.OZ.AU wrote:
> Since NetBSD 4 is still supposed to be supported by pkgsrc,
> and since it seems unlikely that there will ever be a new NetBSD 4
> release to incorporate a fixed version of make, all the packages
> that encounter this problem need to use a pkgsrc supplied
> version of make rather than the system version (when building
> in a NetBSD 4 environment).
Frankly, I would just mark them as broken with the system make.
If someone bothers enough, he/she can always bootstrap to get the modern
make. I explicitly do not want to have another special case for this.
Joerg
From: Robert Elz <kre@munnari.OZ.AU>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: pkg/42704: Adding support for bmake as a pkgsrc tool (ala gmake)
Date: Fri, 05 Feb 2010 06:32:08 +0700
Date: Tue, 2 Feb 2010 23:55:05 +0000 (UTC)
From: Joerg Sonnenberger <joerg@britannica.bec.de>
Message-ID: <20100202235505.230D663C462@www.NetBSD.org>
| Frankly, I would just mark them as broken with the system make.
That's certainly a possibility, I wouldn't mind marking half
of gnome as broken (with the percentage probably growing over time).
| If someone bothers enough, he/she can always bootstrap to get the modern
| make.
For me, at least, that's practical iff it is automated as part of pkgsrc.
I tear down and rebuild the entire sandbox (starting from the release set
files) for every package I build - extract the release, pkg_add the
dependencies, then build, make a package, and remove it all again.
"bootstrap" works only if it is automated (the way a new required set
of pkgtools is automated, for example.) If there's some automated way to
do it (whether or not that means running an additional command), then I
could deal with it (but I don't want to build version dependent dependencies
into my build scripts, so I don't want to just always "pkg_add bmake-*" as
[art of the setup process, as on anything other than NetBSD 4, that's
either already done, or not required.
| I explicitly do not want to have another special case for this.
For just dealing with the NetBSD 4 (and earlier, but that doesn't matter)
make bug, and the bizarre gnome packages with depend lists beyond
comprehension, I'd sympathise.
But I think allowing bmake as an alternative make is more useful than that.
At the minute, when a few feature is added to make, /usr/src gets to use
it 5 minutes later. pkgsrc probably has to wait 5 years (two full
release cycles, at least) - before then you cannot assume the system make
in a supported system has the new feature. That's very hard to do, especially
for new developers who have no idea that this particular feature is new
(or within 2 releases new). That's the primary advantage of allowing
devel/bmake as a supported tool I think, and given just how small the
change is, I'd suggest it is one well worth making.
kre
From: David Holland <dholland-pbugs@netbsd.org>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: pkg/42704: Adding support for bmake as a pkgsrc tool (ala
gmake)
Date: Mon, 15 Feb 2010 21:35:14 +0000
On Thu, Feb 04, 2010 at 11:35:01PM +0000, Robert Elz wrote:
> At the minute, when a few feature is added to make, /usr/src gets
> to use it 5 minutes later. pkgsrc probably has to wait 5 years
> (two full release cycles, at least) - before then you cannot assume
> the system make in a supported system has the new feature. That's
> very hard to do, especially for new developers who have no idea
> that this particular feature is new (or within 2 releases new).
> That's the primary advantage of allowing devel/bmake as a supported
> tool I think, and given just how small the change is, I'd suggest
> it is one well worth making.
I'd agree, FWIW.
Although maybe USE_TOOLS isn't the right framework, since what we're
really after is a conditional dependency based on whether the system
make is new enough for the package.
Maybe "new enough" can be construed globally though...
--
David A. Holland
dholland@netbsd.org
State-Changed-From-To: open->closed
State-Changed-By: dholland@NetBSD.org
State-Changed-When: Mon, 29 Dec 2014 09:02:49 +0000
State-Changed-Why:
netbsd-4 is EOL for some time and so this issue is no longer relevant.
From: Robert Elz <kre@munnari.OZ.AU>
To: gnats-bugs@NetBSD.org
Cc:
Subject: Re: pkg/42704 (Adding support for bmake as a pkgsrc tool (ala gmake))
Date: Mon, 09 Mar 2015 02:10:34 +0700
Date: Mon, 29 Dec 2014 09:02:49 +0000 (UTC)
From: dholland@NetBSD.org
Message-ID: <20141229090250.9156BA65D6@mollari.NetBSD.org>
| netbsd-4 is EOL for some time and so this issue is no longer relevant.
I disagree with this one - the particular issue that led to the proposed
change may no longer be relevant, but the underlying problem will always
be present.
Exactly the same issue could apply today for NetBSD 5, and even perhaps
for NetBSD 6 - (b)make keeps being upgraded &/or fixed, a developer with
a new (and/or fixed) version is naturally going to want to use it fully
(that's why new features are added, and how bugs are found so they can be
fixed).
Outlawing that for pkgsrc packages would be absurd - but since enhancements
cannot be retroactively added to a released version (at best they can be,
sometimes, added into the next point release) as long as the older release
is to be supported by pkgsrc, that would be the effect - unless, as we do
for everything else (X libraries, etc, gcc, libtool, iconv, even the pkg tools)
we allow pkgsrc versions to replace the version that comes native with the
system. The one glaring exception to this is (b)make.
Since it is so trivial to allow it, I see no reason not to do so.
I suspect that the only reason that we don't run across problems with
this more often, is that very fex pkgsrc packages are actually developed
on NetBSD. That's sad - but making it harder cannot help.
kre
>Unformatted: