Tag: powerpc

I wanted to resume writing up notes about what I’d been working on as with the “A week of pkgsrc” series, this actually spans the last few weeks. 🙂

Things basically came together around the event of a late 2009 13″ MacBook entering my life. First port of call was to see state of NetBSD support since I last visited it.
I’m still running FreeBSD on my 11″ Air and so was keen to see if there had been any progress in NetBSD with regards to supporting the Intel based Macs. Unfortunately the same issue is still present, the system panics very early in the boot process kern/52229 when using the UEFI image and fails when enumerating CPUs present. I was able to get the system to boot using the conventional (non-UEFI) image by opting to boot without SMP & ACPI enabled. I used the MacBook install wiki article as a rough guide on gpt partitioning and got a daily image installed. The wiki article needed some attention as the syntax for commands did not apply but through this process I discovered a crash in gpt where a missing check in the source means that a null pointer is passed to one of the gpt commands.
e.g gpt show -a

Running the system without ACPI was not a good idea, turns out the thermal management does not get initialised and the system eventually switches off as a failsafe measure. I found this out through leaving the machine bulk building some of my packages, only to return to a machine that’s powered off.

GCC 6 has landed in NetBSD-HEAD and work is in progress to bring things into shape as the next revision of GCC shipped in NetBSD. Unfortunately cross compilation from macOS does not work at the moment (toolchain/53013) which limited my ability to experiment with different NetBSD kernel configuration while I was trying to look into the MacBook issue.

Up until very recently there was only support for PCIe based G5 PowerMacs using the POWERMAC_G5_11_2 kernel configuration in NetBSD, I am lacking such a system but do have a first gen G5 iMac which is PCI-X based. The initial work to bring up NetBSD on the G5 was actually done on a PCI-X based system long ago so I was curious what had diverged since then. Previous GSoC project participants used a repo in the NetBSD-gsoc sourecforge project to share their work, the two G5 related GSoC projects are there in a single repo. Some further work also took place on the port-macppc mailing list in 2013. It was a fun weekend albeit no further progress to seeing even a copyright notice on my part. However I learnt lots about the kernel build process, poked at lowcore.S and the boot process. I also learnt of an emulator called Mambo. Mambo was a full PowerPC system simulator, produced by IBM research. There are kernel configuration files to support Mambo (theoretically) in NetBSD & FreeBSD but unfortunately, I couldn’t find any binaries to try Mambo, along with that I also found the links for the PowerPC 970FX (G5 CPU) documentation all dead and the documentation removed from the IBM site. 🙁
To rule out the kernel working on a G5 but console being an issue, I tried experimenting with different frame buffers and found macofcons(4) broke the build (port-macppc/53004). After discussing with macallan@, macofcons(4) and the OFB_ENABLE_CACHE option have now been removed [1][2], on the basis that though they may have worked at one point they cause more problems that solve.

In NetBSD, there is support for different buffer queue strategies for disk I/O, on the tier 1 ports such as i386 & amd64, the per-priority cyclical scan strategy is enabled by default, to bring macppc on par, it is now also enabled there by default too. Now to document the option so that it’s somewhat like the description of another strategy for read priority. See BUFQ_READPRIO and BUFQ_PRIOCSCAN in options(4).

This weekend I’ve been testing the HEAD-llvm builds on i386 & macppc as well as ATF testing, but I’ll write about that in another time.

Thanks to jmcneil, martin, mrg, pgoyette, uwe for the help and suggestions.

Before even getting into the internals of operating systems to learn about differences among a group of operating systems, It’s fairly evident that something as simple as naming is different between operating systems.

For example, the generations of trusty 32bit x86 PC is commonly named i386 in most operating systems, FreeBSD may also refer to it as just pc, Solaris & derivatives refer to it as i86pc, Mac OS X refers to it as i486 (NeXTSTEP never ran on a 386, it needed a minimum of a 486 and up until Sierra, machine(1) would report i486 despite being on a Core i7 system), this is one of the many architectures which needed to hadled within pkgsrc. To simplify things and reduce lengthy statements, all variants for an arch are translated to a common name whiche is then used for reference in pkgsrc. This means that all the examples above are grouped together under the MACHINE_ARCHi386. In the case of 64bit x86 or commonly referred to as amd64, we group under x86_64 or at least tried to. The exception to this grouping was OpenBSD/amd64, this resulted in the breakage of many packages because any special attention required was generally handled under the context of MACHINE_ARCH=x86_64. In some packages, developers had added a new exception for MACHINE_ARCH=amd64 when OPSYS=OPENBSD but it was not a sustainable strategy because to be affective, the entire tree would need to be handled. I covered the issue at the time in A week of pkgsrc #11 but to summarise, $machine_arch may be set at the start in the bootstrap script but as the process works through the list of tasks, the value of this variable is overriden despite being passed down the chain at the begining of a step. After some experimentation and the help of Jonathan Perkin, the hurdles were removed and thus OpenBSD/x86_64 was born in pkgsrc 😉

The value of this exercise for me was that I learnt the number of places within the internals of pkgsrc I could set something (by the nature of coupling components which share the same conventions (pkgtools, bsd make)) and really the only place I should be seeking to set something is at the start of the process and have that carry through, rather than trying to short circuit the process and repeat myself.

Thanks to John Klos, I was given control of a IBM Power 8+ S822LC running Ubuntu, which started setting up for pkgsrc bulk builds.
First issue I hit was pkgsrc not being able to find libc.so, this turned out to be the lack of handling for the multilib paths found on Debian & derivates for PowerPC based systems.
This system is a little endian 64bit PowerPC machine which is a new speciality in itself and so I set out to make my first mistake. Adding a new check for the wrong MACHINE_ARCH, long forgotten about the previous battle with OpenBSD/x86_64 I added a new statment to resolve the relevant paths for ppc64le systems. Bootstrap was happy with that & things moved forward. At this point I was pointed to lang/python27 most likely being borken by Maya Rashish, John had previously reported the issue and we started to poke at things. As we started rummaging through the internals of pkgsrc (pkgsrc/mk) I started to realise we’re heading down the wrong path of marking things up in multiple places again, rather than setting things once & propogating through.

It turned out that I only need to make 3 changes to add support for Linux running on little endian 64bit PowerPC to pkgsrc (2 additions & 1 correction 😉 )
First, add a case in the pkgsrc/bootstrap/bootstrap script to set $machine_arch to what we want to group under when the relevant machine type is detected. In this case it was when Linux running on a ppc64le host, set $machine_arch to powerpc64le. As this is a new machine arch, also ensure it’s listed in the correct endianness category of pkgsrc/mk/bsd.prefs.mk, in this case add powerpc64le to _LITTLEENDIANCPUS.
Then correct the first change to replace the reference to ppc64le for handling the multilib paths in pkgsrc/mk/platform/Linux.mk.

The bulkbuild is still in progress as I write this post but 5708/18148 packages in an the only fall out so far appears to be the ruby interepreters.

Preparation for a trip started off a little earlier this christmas. I planned to take my PowerBook on the road with me to Hamburg for 33c3. Previous attempts to use this machine as my primary system on the road in the past had been thwarted by leaving too little time to build & prepare before departure.
The system has been dual booting NetBSD & Mac OS X Tiger for some time now, recently I’ve been doing almost daily upgrades to NetBSD-HEAD on the system using the generated iso images from NYFTP.
My plan was to get the machine installed with a current build FireFox on NetBSD & bring the existing installed packages up to date. I managed to update the existing packages without any problems but it didn’t look like FireFox was going to build successfully. The package as-is currently in pkgsrc does not build on NetBSD/macppc. I was pointed to a patch in pkg/48595 which was pending commit and required testing. It cleared up the initial issue I ran into but the build still failed (see previous link on updates about the failure), though it took a little longer to fail in the day. After several days of failed build attempts I made sure I had an up to date copy of TenFourFox installed on Tiger and settled for Dillo on NetBSD instead.

My usage of Dillo stayed somewhat basic during the trip, despite having the Mozilla certificate bundle installed, I could see any obvious way to point Dillo to it & have it use it. Hence, any site using SSL I visited generated a certificate warning. Perhaps the config should’ve been done in wget?

Moving on, the AirPort Extreme card in the laptop is based on a Broadcom chipset which has a flaw, it’s incapable of addressing memory above 1GB (30 bits) which means the driver needs to care for that or else the card doesn’t work. This is not unique to this Broadcom chipset, the BCM4401 10/100 ethernet interfaces which use the bce(4) driver also suffer from the same problem (unable to address memory allocated above 30 bits), the BCM580x ethernet interfaces which use the bge(4) driver suffer from not being able to address more than 40 bits. Going back to the wireless chipset, the bwi(4) driver which is used in the BSDs, originated from DragonFly BSD. This driver was put together by Sepherosa Ziehau using the documentation from a reversing effort in the Linux community. The bwi driver was then imported in to Free/Open/NetBSD and was eventually removed from DragonFly BSD. A new wireless subsystem was introduced in DragonFly which required change to drivers to work again and the bwi driver was never adapted. It now lives on in the other BSDs.

The version of bwi(4) driver came to NetBSD from OpenBSD, ported by Taylor R. Campbell back in 2009. At the time neither version of drivers could handle the 30 bit bug so you either ran with less than 1GB of RAM or used another card. In 2014 Stefan Sperling committed a workaround for this in OpenBSD. I wanted this fix in NetBSD so my wifi could also work & asked the NetBSD developers if such a change was appropriate in NetBSD. I was introduced to bus_dma(9) and the bus_dmatag_subregion() function, the bce(4) driver was my reference on how to use the function. Looked fairly straight forward, a single call this function and off you go, wasn’t too sure how it would fit into the bwi driver but I thought I’d have a go.

This was one of the things I was hoping to work on during my trip but It turned out to be the only thing I attempt. I happened to meet Stefan at 33c3 and we discussed the driver, the work around and the mighty days of the past when Damien Bergamini was hacking on the OpenBSD WiFi stack. In the OpenBSD driver Stefan had opted to deal with the issue of allocating memory in a specific region directly in the driver rather than adding a new interface to the kernel for such a task so with a bit of thought about the past and a review of the driver, I was given a diff of the changes and suggestions about where I could start making changes.

I still don’t know yet if it’s possible to lift the changes from OpenBSD and apply them to the NetBSD version of the driver, because the DMA framework is different between the systems.
Partially implementing the change Stefan made without all the bounce buffers he’d added in the OpenBSD driver didn’t work and using the bus_dmatag_subregion() function didn’t work either. I pursued the bus_dmatag_subregion() path during 33c3 and didn’t get anywhere. At this point I started looking deeper in the system by looking at the implementation. It was at this point that I discovered this function was defined to EOPNOTSUPP on PowerPC based systems. No matter what I had tried with this function it was a waste of time^W^W^Wvaluable learning experience about keeping documentation up to date & consistent.

sevan: share/man/man9: bus_dma.9: Give a heads up about bus_dmatag_subregion()

Time again to write up what’s been happening on the pkgsrc front since last time, this time the focus hasn’t been so much around pkgsrc on Darwin/PowerPC but more about pkgsrc in general & not necessarily code related.
At the end of the last post I mentioned a gentleman who’d been working on pkgsrc/Haiku and posting videos of his progress, I managed to make contact with him (James) & discussed his work that he’d been doing on pkgsrc. He sent me copy of the repo he’d be working off so I could assist with the aim of getting everything upstreamed as in the current state everything would need to be reintegrated per quarterly release rather than only having to pay attention if a new issue has arisen.
After getting the correct version of Haiku installed in virtualbox, I discovered a nasty bug in the Haiku network kit, it was unable to detect when the end of the file had been reached & would continue (restart?), this was revealed when I tried to download the pkgsrc tar ball from via WebPositive, ftp from the terminal was not affected however. pkgsrc bootstrapped unprivileged without issue. Hint: use the nightly snapshots until there is a newer release than Alpha 1 available.
The integration of pkgsrc into the user-land on Haiku is not currently possible due to the way the user-land is constructed, from what I understood, each Haiku package contains a piece of the filesystem, all the packages are union mounted to construct the user-land dynamically when the system comes up. That aside, with my system bootstrapped, I attempted to build Perl and ran into another bug, it seems that the library path for libperl is not populated on haiku hence perl is able to “build” but unable to run, the workaround for this in the tree I was given was to symlink libperl into ~/pkg/lib & move on. I tried various things but was unsuccessful, I believe the problem is pkgsrc specific as the version of Perl available in haikuports do not need any special treatment and the rpath is passed in correctly.
The problem was trying to isolate the required change to fix the problem, whereas in pkgsrc a policy file is passed to the build to set how Perl should be built, haikuports clobbers the source & patches in a replacement, I stopped at that point.

At around about this time I received the good news about the NetBSD Foundation membership and commit bit so my focus moved to reading the various developer documentation & getting familiar with processes.

sevan.mit.edu finished a bulkbulid attempt of the entire tree which took the longest time so far to complete, through all the build attempts I uncovered a new bug, the range to use for numerical IDs of UID/GID, is not sufficient to cover all the packages in the tree that need to create an account. On further discussion with asau@, it was suggested the IDs are allocated randomly and should be fixed for consistency across builds. I started doing bulkbuilds of the entire tree on FreeBSD 10.1-RELEASE and stumbled across a very nasty bug. There is a version of tcsh package in the pkgsrc tree called shells/standalone-tcsh, this is tcsh built as a static binary & set to install to /bin (the only package in the pkgsrc tree which violates the rules and places files outside of $PREFIX by default?), this ended up overwriting the system bundled version of tcsh in FreeBSD & then deleting /bin/tcsh when the package is removed, this was fixed promptly by dholland@. It was also discovered that Python’s configure script had not been changed when FreeBSD switched to elf binaries and so still trimmed the name of libraries to account for the old linker which could not handle a minor version in a libraries filename (libpython.so.1, not libpython.so.1.0). All versions of Python had been patched in the pkgsrc tree to remove this change so that it used a consistent naming convention across all platforms. After discussing this with bapt@ (FreeBSD) at FOSDEM, it turned out to be bug and should be fixed in future Python versions once the fixes are upstreamed.

The opportunity to join the pkg-security team came up & for the past few weeks I’ve been getting familiar with the processes of dealing with security advisories & listing them so that users who fetch the pkg-vulnerabilities database are notified if they have any vulnerable packages installed. The general advisory process is a little infuriating, based on my recent experience I’d say at the top of my list are the Oracle security advisories as they do not divulge any details other than “unknown” in version(s) X, PHP for the frequency, OpenSSL for the impact. On the one hand I was quite impressed that CVE IDs were becoming so familiar that I could spot, on the fly, an advisory that had been accounted for, but on the other hand quite upset that I was using brain capacity on this. The availability of information is quite frustrating too, issues which are assigned an ID but cannot by checked on Mitre’s site take extra effort to find the necessary information to include (Mitre are responsible for allocating the CVEs!), I should note that this is from public advisories, say from a distribution. Example, CVE-2015-0240 was announced today, the Redhat security team published a blog post covering the issue, the Mitre site at present says:
“** RESERVED ** This candidate has been reserved by an organisation or individual that will use it when announcing a new security problem. When the candidate has been publicised, the details for this candidate will be provided.”
The wording & the lack of information can also be frustrating because it’s not clear what is affected. Looking at it positively, the requirement for clarification on these discrepancies means I get lots of opportunities to approach new people in different communities to ask questions.

One thing that’s clearly evident is my workflow needs attention, at the moment things are very clumsy, involving lots of switching around but hopefully that will be addressed in the coming month. The first thing I’ve done is setup templates for emails with the correct preferences specified so that I just need to fill the necessary information & hit send, the necessary settings are automatically applied. Still thinking about how to deal with the scenario where the system that work is being carried out on is different to the system where the patch is going to be committed from, this also happens to be a different system which a developer is using. How to deal with that in as few steps from reading, say a bug report, to generating a patch, testing it & committing a fix.

For testing patches on Mac OS X, I revisited running OS X as guest on a Mac running OS X with virtualbox. Attempts in the past had not been successful & it seemed from search results that the only approach taken was to use modified OS images for hackintosh which I did not want to take. I have a genuine machine & genuine license, I shouldn’t have to resort to 3rd party images to run this. After whinging on twitter & referencing some older links I was able to successfully virtualise Mac OS X 10.7 to 10.10 in virtualbox. Will follow up with the details on that in a separate post.

Since the last post I’ve made some further progress with pkgsrc on Darwin/PowerPC again, the biggest achievement was fixing lang/ruby19-base, lang/ruby200-base and lang/ruby21-base which accounted for the breakage of some 1500 packages (variation of 500 or so modules for each version of Ruby). This was caused by the failure to build the DBM module, which on OS X required the inclusion of dbm.h as well as ndbm.h otherwise all tests fail and the module is not built. The frustrating thing is that there appears to be no documentation for the build process of Ruby, luckily, by Ruby 2.0 there was a comment added to ext/db/extconf.rb to shed some light on the issue:# Berkeley DB's ndbm.h (since 1.85 at least) defines DBM_SUFFIX.
# Note that _DB_H_ is not defined on Mac OS X because
# it uses Berkeley DB 1 but ndbm.h doesn't include db.h.pkg/49508 was committed prior to the 2014Q4 pkgsrc release butpkg/49511 and pkg/49512 did not.

devel/commit-patch used the -a flag for cp(1) which isn’t available on older operating systems, pkg/49475 switched to the use of -pPR instead (which -a is an alias of).

graphics/ivtools failed to build successfully due to a packing issue due to the explicit specification of operating system in the name of one of the generated files. pkg/49497 switched the use of the LOWER_OPSYS & added missing item which addressed the issue.

security/CSP failed at the installation stage due to the target directory not existing, pkg/49499 fixed that.

mail/nullmailer referenced uid_t & guid_t but did not include sys/types.h, pkg/49523 fixed that.

net/dnsmasq referenced SIOCGIFAFLAG_IN6, IN6_IFF_TENTATIVE, IN6_IFF_DEPRECATED & SIOCGIFALIFETIME_IN6 but did not include netinet6/in6_var.h when building on OS X which broke the build. pkg/49524 fixed that.

lang/lua52 failed to build on Tiger due to sys/types.h, pkg/49526 fixed that.

lang/php55 bundles its own version of sqlite and requires the necessary flags to disable features not available pkg/49527 fixed that but the correct fix is to not build an entire new version solely for PHP’s use. I began to look but had flashbacks of dealing with the same issue in TCL.

For graphics/MesaLib I looked to build it using a newer version of binutils but it appears that support for Darwin/OS X/iOS and Mach-o is rudimentary and hence missing support in most of the tools. Support began being added upstream to binutils back in 2011 but is still not complete.

For devel/cmake supports the ability to specify the location of library & header files, this can be done by creating a file which includes the necessary declaration that is passed to the configuration process using the --init flag. Indeed when the configuration process displayed the correct versions of OpenSSL, CURL, Zlib, BZip among others from pkgsrc rather than the older system bundled versions, unfortunately the build still failed when it came to the linking stage as the paths to the libraries was prefixed with /Developer/SDKs/MacOSX10.4u.sdk, as a kludge just to progress with the builds, I symlinked /usr/pkg to /Developer/SDKs/MacOSX10.4u.sdk/usr, the build then succeeded without issue. Next task is to work out how to drop the /Developer/SDKs/MacOSX10.4u.sdk prefix correctly.

There are currently 11132 packages available on sevan.mit.edu for Mac OS X/PowerPC with a new bulkbuild of the entire tree in progress. There are also Intel builds of the entire tree being attempted by Save OS X (64-bit packages) and Jonathan Perkin (32-bit packages) which should further improve support for OS X in pkgsrc.

Whilst browsing I discovered a series of videos on youtube by DisneyDumbazz, he has also been covering his work on improving support for Haiku in pkgsrc at length.
He was also struggling with issue in Ruby, QT4 & Mesa it seems.

Definitely more than a week, I’ve not had a chance to devote much time to this over the past few months due but have made sufficient progress to qualify another post.
The most import thing is apart from one PR, all previously submitted patches have now been committed to pkgsrc-current, pkg/49082 still remains.

With the introduction of GCC 4.9, the same changes needed to be applied to lang/gcc49 as with previous versions, pkg/49178 took care of that, however this highlighted another problem. 32bit & 64bit hosts running Darwin both identify themselves as powerpc in the uname(1) output which means that GCC is always built with multilib support disabled, even when building on a 64bit host.

Some of the cross compilation tools for micro controllers were hardcoded to use ksh to build with when in fact it was only required for NetBSD >= 5, this caused the build to break on Tiger (assuming because of the old version of bundled ksh), pkg/49311 fixed that for cross/binutils but the changes were also applied to cross/freemint-binutils and devel/binutils by the maintainer.

cross/avr-libc was previously broken because it was using the system compiler & headers instead of avr-gcc and the headers installed in pkgsrc during builds. pkg/49316 fixed this issue and upgraded the version to v1.81 of avr-libc.

It’s no longer a requirement to declare the MACOSX_DEPLOYMENT_TARGET environment variable to build lang/perl5. By default Perl declares this to be 10.3 which is no longer applicable on modern systems and when building with clang mmacosx-version-min is specified, making it redundant. This had been removed in pkgsrc via a patch and it broke the build for GCC users as without this variable the target defaults to 10.1 and Perl needs specific attention for versions prior to Panther. pkg/49349 added this variable back in for Darwin 9 and prior which were GCC only releases. Bug #117433 in the Perl RT was the source of the patch proposed to resolve the issue.

lang/ocaml now builds on Tiger, the workaround for the lack of support for -no_compact_unwind in the shipped linker was applicable to prior releases and not just specifically Leopard, pkg/49417 fixed that.

devel/py-py2app previously failed to build on PowerPC OS X due to an error in the PLIST, the use of the MACHINE_ARCH variable would expand to powerpc which raised a packing error. pkg/49418 fixed that.

graphics/MesaLib and devel/cmake still remain broken in the pkgsrc tree for Darwin PowerPC, I was able to generate a MesaLib package successfully by forcing static binaries which allowed the previously unattempted packages to be tried in a bulk build of the entire tree. Unfortunately I hadn’t caught a merge conflict from when pkg/49077 was committed and so devel/icu was not built, this caused a another large subset of packages to not be built.

With these changes, there were over 10000 packages available on sevan.mit.edu but unfortunately that included lots of duplicate packages from previous bulk builds. pkgtools/pkglint has the ability to scan packages against a pkgsrc tree & remove duplicate/stale packages. Running lintpkgsrc -K /mnt/packages -pr took the number of packages down to 9200. There is an AWK based solution but I’ve not had a chance to try it.

I was able to get devel/cmake to build successfully by removing the references to /Developer/SDKs in Modules/Platform/Darwin.cmake and subsequently build packages such as databases/mysql56-client but I’ve not added the changes to the tree yet. Will look to add this in a future bulk build, I want to get MesaLib linking correctly first before adding more kludges into the mix. The next thing I want to try is using a newer version of linker from devel/binutils instead of the one bundled with Xcode.

Shortly after the last blog post I had access to a couple of AIX LPAR. This would be my first time on a IBM PowerPC system and AIX, I’d applied for two AIX 7.1 instances, one defined as “AIX 7.1 Porting Image” and the other as plain “AIX 7.1”. The difference at a glance seemed to be the porting image had more gnu / common open source tools e.g GNU/Tar though both images had a version of GCC installed.

The stock version came with GCC 4.2 built on AIX 6.1 whereas the porting image came with GCC 4.6.
Alongside the open source tools each instance also had proprietary tools installed including IBM’s compiler XLC, cc without any options invokes a man page which describes the different commands that represent a language at a level.

c99 – Invokes the compiler for C source files, with a default language level of STDC99 and specifies compiler option -qansialias (to allow type-based aliasing). Use this invocation for strict conformance to the ISO/IEC 9899:1999 standard..

The pkgsrc bootstrap process didn’t work too well by trying to allow it to workout things out for itself via cc so opted to use GCC specifically.

export CC=gcc

pkgsrc happily bootstrapped without privilege and I proceeded to install misc/tmux and shells/pdksh on AIX.

security/openssl comes with 4 different configuration settings for AIX, a pair of settings for the XLC & GCC compilers with a 32bit or 64bit target. It turned out that in pkgsrc it just defaulted to aix-cc (XLC with a 32bit target), pkg/49131 is now committed so the correct configuration is used, XLC successfully builds OpenSSL with a 32bit or 64bit ABI but GCC is only able to manage a 32bit target.

To switch compiler to xlc, declare it as the value to PKGSRC_COMPILER in your mk.conf.

Over the week I attempted to compile components of GCC 4.8 without much success, starting off with lang/gcc48-cc++ & falling back to lang/gcc48-libs.
The build process was very unstable, again as with the Tiger/PowerPC, the build would spin off & hang, pegging the CPU until killed. Attempting to restrict the processor time via ulimit didn’t have much effect.

Alongside trying to get GCC built on AIX, I kicked off building meta-pkgs/bulk-medium on sevan.mit.edu, the previously reported unfixed components prevented some of the packages from building again (ruby, MesaLib, cmake).

I began looking into fixing devel/cmake so that it would link against the correct version of curl libraries & use the matching header files, Modules/FindCURL.cmake in the cmake source references 4 variables which provide some control but I was unsuccessful in being able to pass these to the pkgsrc make process. While trying to resolve this issue I also discovered that on more recent version of Mac OS, the dependencies from pkgsrc ignored, opting for the use of the Apple supplied versions even though the pkgsrc version would be installed.

-- Found ZLIB: /usr/lib/libz.dylib (found version "1.2.5")
-- Found CURL: /usr/lib/libcurl.dylib (found version "7.30.0")
-- Found BZip2: /usr/lib/libbz2.dylib (found version "1.0.6")
-- Looking for BZ2_bzCompressInit in /usr/lib/libbz2.dylib
-- Looking for BZ2_bzCompressInit in /usr/lib/libbz2.dylib - found
-- Found LibArchive: /usr/lib/libarchive.dylib (found version "2.8.4")
-- Found EXPAT: /usr/lib/libexpat.dylib (found version "2.0.1")
-- Looking for wsyncup in /usr/lib/libcurses.dylib
-- Looking for wsyncup in /usr/lib/libcurses.dylib - found
-- Looking for cbreak in /usr/lib/libncurses.dylib
-- Looking for cbreak in /usr/lib/libncurses.dylib - found

mail/mailman had a missing README in PLIST which was handled differently between Tiger & newer releases. pkg/49143 was committed to fix that.

Didn’t uncover anything new in pkgsrc last week as my attention was more on coreboot, I had previously been building different parts of the tree on a couple of Mac’s which where disconnected from each other & copying packages to sevan.mit.edu manually for serving, as a first off this was a good idea but bad as an ongoing thing. What ends up happening is stale packages become left behind as they are unaccounted for, luckily there aren’t too many duplicates currently but it’s something which needs to be addressed in the set of packages currently available.

To test the status of AIX support in pkgsrc I joined the IBM Power Developer Platform which provides access to Power7/7+/8 systems running AIX 6.1 & 7.1 to build software on. This’ll be my first time on a Power system & AIX, looking forward to seeing what the OS is like.

With the addition of a G5 iMac to the effort kindly donated again by Thomas Brand, I started testing builds of lang/gcc48 on sevang5.mit.edu. Next step will be to get the two systems at MIT working together to build packages once I’ve been able to get GCC 4.8 to build successfully.

Following on from last week, I worked on components which caused large numbers of packages not to build.textproc/icu failed to build due to localtime_r() not being used if either _ANSI_SOURCE or _POSIX_C_SOURCE is defined & using an opcode that the shipped version of assembler didn’t understand. Ticket #9367 provided fixes for both issues spanning over 2 years, pkg/49077 covers this but has not been committed.databases/sqlite3 failed to link with ld: Undefined symbols: _OSAtomicCompareAndSwapPtrBarrier error, this is due to the lack of zone memory allocator, PR #49081 fixed this issue by defining -DSQLITE_WITHOUT_ZONEMALLOC for OS X releases prior to Leopard. This is PR was committed. A subsequent PR (pkg/49082) was raised to do the same for lang/tcl which also bundles its own copy of sqlite3 for its sqlite module, but has not been committed.

devel/pango was broken on OS X releases prior to Leopard as the package enabled the CoreText option by default but failed due to packing errors (CoreText is not available hence the .la file not existing when build has completed). pkg/49090 resolved the issue & was committed.

sh(22232) malloc: *** error for object 0x34e340: incorrect checksum for freed object - object was probably modified after being freed, break at szone_error to debug
sh(22232) malloc: *** set a breakpoint in szone_error to debug
sh(22232) malloc: *** Deallocation of a pointer not malloced: 0x34d7ab; This could be a double free(), or free() called with the middle of an allocated block; Try setting environment variable MallocHelp to see tools to help debug
sh(22232) malloc: *** Deallocation of a pointer not malloced: 0x34e340; This could be a double free(), or free() called with the middle of an allocated block; Try setting environment variable MallocHelp to see tools to help debug
checking sys/time.h usability... Makefile:16170: recipe for target 'configure-stage2-target-libgomp' failed
gmake[2]: *** [configure-stage2-target-libgomp] Error 137

The stability of sevan.mit.edu was improved by re-applying the 10.4.11 combo update.

Currently in the process of fixing devel/cmake, cmake now get through most of the build (it was previously failed at 3%) but fails at the linking stage due to path issues. It picks up the pkgsrc version of CURL as /usr/pkg/bin/curl but tries to link against libraries in /Developer/SDKs/MacOSX10.4u.sdk/usr/pkg/lib which doesn’t exist.

This is summary of the things I worked on along with the help of others over the last week on pkgsrc.
With the donation of sevan.mit.edu along with a G4 Mac Mini at pksrcCon 2014 I setup bulk package builds as per chapter 7 of the pkgsrc guide to generate packages for OS X.
The bootstrap process is now able to differentiate between gcc & clang, as clang tries to be GCC compatible it tries to pass itself as GCC in tests, this would cause an issue where the bootstrap would use /usr/bin/clang for some parts of the build & /usr/bin/gcc for others, on top of that, the bootstrap process was hardcoded to use gcc on Darwin. The bootstrap process now defaults to using cc & correctly detects if that is clang or gcc.

security/sudo now builds on Darwin (confirmed on Tiger PowerPC & Mavericks), the no_exec module doesn’t build on Darwin & is switched off in the Apple supplied build of sudo, this wasn’t switched off in pkgsrc & caused the build to fail. There are more options set in the Apple build to improve posture which are not set in pkgsrc version, that needs looking into further & is on the TODO list.

The new release of help2man committed last week broke on Tiger due to NLS being switched off & the new version introducing additional translations of info pages. The patch in pkg/49059 fixes things so shared libraries are taken care of as with Leopard & the package is built with NLS support.

Currently working on trying to get graphics/MesaLib building with XQuartz, the version shipped with Tiger is based on XFree86 & MesaLib fails to link libraries, macports seem to have some fixes related to building on Tiger which I’m hoping may fix some of the issues.

Will also be looking at devel/cmake as it’s currently broken on Tiger which means things such as mysql server cannot be built at the moment.

Through the existence of a directory called devel in /tmp which was owned by a user other than the the one pbulk runs under, some critical components such as autoconf & tradcpp did not build on the Mac Mini, this caused many builds to fail, that aside, the Mini has managed to build 1064 out of a queue of 2083 packages over the last week.
sevan.mit.edu is currently down (due to possible hardware issues) & awaiting a reboot.

In pkgsrc there’s a facility which allows you to perform bulk builds of packages called pbulk.
Using this facility on a couple of donated systems I have started to generate packages for PowerPC OS X. Currently builds are performed on 32bit PowerPC Macs running OS X with pkgsrc-current. The binaries should in theory work on 64bit PowerPC systems and on Leopard but have not tested to confirm.
The packages are made available at sevan.mit.edu.
To utilise the packages on your system, fetch & uncompress the bootstrap archive which contains the pkgsrc tools.curl -s http://sevan.mit.edu/packages/bootstrap.tar.gz | sudo tar -zxpf - -C /

Update your PATH & MANPATH variablesPATH=/usr/pkg/sbin:/usr/pkg/bin:$PATH
On TigerMANPATH=/usr/pkg/man can be declared in /usr/share/misc/man.conf
On Leopard use path_helper(8) and create a file in /etc/manpaths.d which just contains /usr/pkg/man.
This can also be extended to PATH by creating a file in /etc/paths.d/ containing one path element per line. This requires testing however as the impact is system wide.

Due to various factors, I’ve not had much of a chance to play with the PowerBook much this month, earlier this moth a follow up to PR/48740 happened, requesting feedback on new changes which had been committed that I’ve not had a chance to test yet.
One thing I did do tonight was to re-flash the SuperDrive with a RPC-1 firmware image which turns the DVD drive region-free.
The firmware images are hosted on MacBook.fr and cover Macs all the way back to G3’s.
Flashing was straightforward though I could only re-flash with the version currently on the drive. It was not possible to flash a newer stock or region-free image on the drive.
Aside from the firmware on the DVD drive, Mac OS also tries to enforce region locking, the Region X utility can reset the Mac OS related setting regarding content region.

Since I last posted about dealing with pkgsrc on my PowerBook earlier this week, a patch has been committed which solves the linker issue on lang/gcc45 along with another fix related to how stripping of binaries is handled on Darwin. Unfortunately PR/48740 mentioned gcc44 to 46 suffer from the same issue but the patch has not been applied to gc44 or 46, I’m waiting to hear back from the committer.
One thing I forgot to mention in the previous post is that there is another issue with build process for GCC. There appears to be a deadlock issue where sh sits there chewing up CPU & context switching, at this point, aborting the build with & restarting again allows the build to continue.
Attaching to sh with gdb does not give any further insight as to what’s going on 🙁

As I wrote the previous post, the PowerBook was attempting to compile lang/gcc48 from pkgsrc & since then I’ve still been attempting to build lang/gcc48 with varying levels of success which led me to branch out from trying to build GCC 4.8 to the other 4.x releases on pkgsrc.

In-between roasting the PowerBook I managed to obtain a re-writable CD, that allowed me to install OpenBSD on the PowerBook. Everything went smoothly apart from some bug with installer/documentation. The OpenBSD install was short-lived as I swapped the 5400RPM HDD for an 64GB SSD, the empty partition is there ready for reinstall but the GCC builds have consumed most of the time with the computer so I’ve not got around to reinstalling on the new disk.

I reverted to attempting to building with only C & C++ language support to speed up the time to success/fail. This reduced the build time down to 8-10 hours, it then became apparent (a little quicker) that all GCC 4.x releases in pkgsrc fail to build on Mac OS X Tiger.
GCC 4.4 was the easiest to fix as the only thing that prevented the build of the C/C++ language support was a space between the -I flag & the path to headers to include which caused the linker to complain, this issues is taken care of in lang/gcc47 & gcc48 but the fix wasn’t retroactively applied to previous releases.
Attempting to build with GCC 4.5 & newer revealed further issues.
Thanks to the maintainer of TigerBrew, Misty De Meo who was able to provide hints for issues that need to be addressed in-order to build a more recent version of GCC on Tiger, having gone through the same problems when adding support in TigerBrew.

I’m not sure how many iterations other builds use but on pkgsrc gcc is built with 3 iterations of tests where binaries generated are compared at each iteration. The tests would fail on the third iteration, forcing dwarf2 format for debug data allowed the tests to succeed. As the G4 is a 32-bit PowerPC CPU, the build then failed when it came to 64-bit binaries, disabling multilib support resolved this issue, the build then failed as the linker & assembler bundled with XCode 2.5 were too old & missing functionality. Bug 52482 covers this issue.

Apple provides updates as part of their open source initiative, cctools contain the as & ld sources but sadly with the move to intel & the startup of the Hackintosh scene Apple has made it a little difficult to build things, their first move was to stop generating the iso of Darwin builds & the documentation seems non-existent now. Funnily enough the legacy information on the iPhone jailbreak & Hackintosh scene seems to be the commonly available documentation, that is ofcourse after you’ve excluded bug reports from fink & mac/darwinports in your search results. The main problem with building the open-source components is the bespoke build mechanism which seems to have various issues with Mac OS X, mainly missing header files, with TigerBrew, they seem to work around this issue by using the headers from a newer version of OS X. Through my search for a solution I discovered the OpenDarwin project, the project repackaged the source for the open source Apple components around the GNU toolchain. A version is available in pkgsrc in emulators/darwin_lib though it’s intended for NetBSD/PowerPC to provide binary compatibility, sadly the version it attempts to build is older than the version available with XCode 2.5. The OpenDarwin site is broken & the project doesn’t appear to be in development but I was able to find a repository on Apples MacOS Forge for the OpenDarwin cctools which contained the cctools-758. This was sufficient to allow GCC 4.x to build successfully, AS & AS_TARGET variables need to be set to where the new version of as(1) installed from odcctools however. GCC 4.7 built successfully with C, C++, Fortran, Objective C, Objective C++ & Fortran support. The code is in a SVN repository, you can obtain precompiled binaries of old versions of svn from collab.net.Using built-in specs. COLLECT_GCC=/usr/pkg/gcc47/bin/gcc COLLECT_LTO_WRAPPER=/usr/pkg/gcc47/libexec/gcc/powerpc-apple-darwin8/4.7.3/lto-wrapper Target: powerpc-apple-darwin8 Configured with: ../gcc-4.7.3/configure --enable-languages='c obj-c++ objc java fortran c++' --enable-shared --enable-long-long --with-local-prefix=/usr/pkg/gcc47 --enable-libssp --enable-threads=posix --with-boot-ldflags='-static-libstdc++ -static-libgcc -L/usr/pkg/lib ' --with-dwarf2 --disable-multilib --disable-nls --with-gmp=/usr/pkg --with-mpc=/usr/pkg --with-mpfr=/usr/pkg --with-ecj-jar=/usr/pkgsrc/distfiles/ecj-4.5.jar --enable-java-home --with-os-directory=darwin --with-arch-directory=powerpc --with-jvm-root-dir=/usr/pkg/java/gcc47 --with-java-home=/usr/pkg/java/gcc47 --with-system-zlib --enable-__cxa_atexit --with-gxx-include-dir=/usr/pkg/gcc47/include/c++/ --with-libintl-prefix=/usr/pkg --prefix=/usr/pkg/gcc47 --build=powerpc-apple-darwin8 --host=powerpc-apple-darwin8 --infodir=/usr/pkg/gcc47/info --mandir=/usr/pkg/gcc47/man Thread model: posix gcc version 4.7.3 (GCC)
While attempting to get things to build, lang/gcc48 was updated in pkgsrc and languages were separated out to individual packages with gcc48 becoming a wrapper around this, these changes are very much in their infancy and have issues which need to be addressed. As it currently stands, it’s not possible to build individual languages as there is ties which need to be removed e.g attempting to build lang/gcc48-cc++ will build in lang/gcc48-fortran. For me, the next step is to get these changes prepared correctly & submitted for inclusion in pkgsrc, for the duration of getting things to run I hard coded the paths for as(1) & defined the configure arguments globally, where as these changes are specific to Tiger & prior / 32-bit PowerPC specific. Then there’s dealing with odcctools, I’m not sure if it will require a new package or emulators/darwin_lib should be extended. emulators/darwin_lib introduces more than just odcctools hence my leaning towards a separate package. I also want to try out building from a RAM disk using Make RAM disk to see if build can be sped up any further. Since I wrote the previous article I found out how to set the location in F.lux, thanks to co-developer of F.lux, Michael Herf

It seems that some people opt to cross-compile for OS X on another platform, including macports developers (sorry, I didn’t make a note of the bug report where the developer explained building packages on Linux). iTerm 0.10 from the original project is now my terminal emulator with the Inconsolata font, the font smoothing isn’t that great on this machine when using white text on black background so I’ve opted for the opposite here & it’s working out well.
There is a MAC POWERPC ? blog which provides regular articles. I found an article on how to rebuild the PowerBook battery with new cells if it should fail.

Following the instructions from floodgap.com to update the curl-ca-bundle bundled with OS X I discovered that the version included in Tiger was from 2000!!

With the talk on Twitter & App.net about old computers I started to get nostalgic. I had cleared out most of my collection back in 2012 & been resisting the urge to resume hoarding again largely, having successfully put off the purchase of a Ubiquiti EdgeRouter Lite to run FreeBSD on, I remembered that I was offered a G4 PowerBook a few months back which I turned down. It was still available if I wanted to take it, which made very happy. a 12″PowerBook6,4 that I’d assumed it was going to be a 15″ model. I’ve been playing about with it for the past couple of days, wiping the pre-installed copy of Leoapard & going through the Panther to Tiger path.

The system is now running 10.4.11, patching was a lot of fun, java update after java update, pretty sure it didn’t seem that bad at the time.
It was interesting to see that there was no iTunes update made available, having to manually fetch v9.2.1 from kb DL1056. Safari was updated to v4.1.3.

With no more updates on offer from the software update facility I disabled the java & macromedia plugins by moving them out of /Library/Internet Plug-ins & /Library/Application Support.

Going back to Tiger was a mixture of pleasure & pain, visually, I much prefer the brighter white look of aqua, as opposed to the grey theme which introduced in Leopard. Terminal.app in Tiger is not that great, font smooth is particularly poor, I may have to resort to sourcing a copy of the original iTerm. Plan9 from userspace built without issues using gcc from XCode 2.5, but I guess finder doesn’t like something about the bundled transparent icon of Glenda on the dock as it shows up with a white background & though acme launches correctly, the icon continues to bounce on the dock.F.lux 1.1 is the last supported PowerPC build which runs on Tiger, no support for UK in location settings of this build.TenFourFox takes the place of Firefox as an up to date, maintained version for the PowerPC Mac’s. Python was updated to 2.7.6 using a package straight from python.org.

There is a PowerPC Software site, which contains links to the last builds of popular software which supported the PowerPC Mac’s.

Mercurial & Ruby built successfully from source, pkgsrc also bootstrapped without any issue, the system is currently building GCC 4.8 from pkgsrc.Needed to declareMACOSX_DEPLOYMENT_TARGET=10.4 otherwise the build process would fail with ld: flag: -undefined
dynamic_lookup can't be used with MACOSX_DEPLOYMENT_TARGET environment
variable set to: 10.1
The system currently has 512MB of RAM & a 74GB HDD, 40GB allocated to OS X & the remaining intended for use with OpenBSD, will have to netinstall OpenBSD as I don’t have any blank CD’s with me, no USB hub, the USB ports don’t provide sufficient power to run a Zalman Virtual CD and I suspect the system is unable to boot from USB anyway. Been looking on Amazon for IDE SSD drives but probably will increase the RAM first.

Disable the onboard Broadcom chipset network card & stick a Intel i82558B chipset network card in the system, I used a Compaq NC3121 which can be picked up off ebay for next to nothing. The card doesn’t specifically have to be i82558B based, I think OS X supports the entire range of the Intel chipsets supported under the *BSD fxp driver.

From the Application Preferences in Virtual PC install the script menu. Then pick the VM from the Virtual PC list you’d like to speed up, goto the Virtual PC Scripts section in the script menu & choose Toggle System Disc Cache.

Solaris 9 normally wont allow you to install onto a virtual machine on Virtual PC, it bombs out with the error 486 Processor Detected This Processor is not supported by this release of Solaris
Thanx to this simple hack found on Kernel Thread its possible to install Solaris 9 without a hitch on VPC