Uwe Hermann - openmphttp://www.hermann-uwe.de/taxonomy/term/1816/0
enRebuilding the whole Debian archive using the Open64 compilerhttp://www.hermann-uwe.de/blog/rebuilding-the-whole-debian-archive-using-the-open64-compiler
<p>I got bored recently, so I rebuilt the whole Debian archive on one of my machines. To make this not a <em>completely</em> useless excercise, I used the <a href="http://www.open64.net/">Open64</a> compiler instead of gcc and created <a href="http://people.debian.org/~uwe/">build logs</a> for your perusal.</p>
<p><strong>So what is Open64?</strong></p>
<p>From the <a href="http://en.wikipedia.org/wiki/Open64">Wikipedia page</a>:</p>
<blockquote><p>
Open64 is an open source, state-of-art, optimizing compiler for the Intel IA-64 (Itanium), AMD Opteron and Intel IA-32e architecture. It derives from the SGI compilers for the MIPS R10000 processor. It was released under the GPL in 2000, and now mostly serves as a research platform for compiler and computer architecture research groups. Open64 is licensed under the GPL. Open64 supports Fortran 77/95 and C/C++, as well as the shared memory programming model OpenMP. It can conduct high-quality interprocedural analysis, data flow analysis, data dependence analysis and array region analysis.
</p></blockquote>
<p><strong>Open64 installation</strong></p>
<p>The <a href="http://svn.open64.net/filedetails.php?repname=Open64&amp;path=%2Ftrunk%2FHOWTO-INSTALL-OPEN64&amp;rev=0&amp;sc=0">installation </a> is pretty easy fortunately:</p>
<pre>
$ wget http://ovh.dl.sourceforge.net/sourceforge/open64/open64-4.0-src.tar.bz2
$ tar xfvj open64-4.0-src.tar.bz2
$ cd open64-4.0
$ export TOOLROOT=/opt/open64
$ make
$ make install (as root)
</pre><p>
I think you need gcc-3.4 (gcc 4.x is not yet supported), and for some odd reason you also need <a href="http://packages.debian.org/unstable/shells/csh">csh</a> as one of the install scripts seems to use it.</p>
<p>It would be nice if someone could package Open64 for Debian, I definately don't have the time to maintain such a huge package (a whole maintainer team would probably be good here).</p>
<p><strong>Rebuilding the Debian archive</strong></p>
<p>There are several possible ways (and tools) to rebuild the Debian archive; I've used <a href="http://packages.debian.org/unstable/devel/pbuilder">pbuilder</a>/<a href="http://packages.debian.org/unstable/utils/cowdancer">cowbuilder</a> with the <a href="http://blog.venthur.de/2007/02/20/rebuilding-the-archive-using-cowbuilder/">rebuild scripts from Bastian Venthur</a>, which are now included in pbuilder.</p>
<p>First we need to install the required packages, setup a cowbuilder base chroot, and get the list of packages:</p>
<pre>
$ apt-get install cowdancer grep-dctrl wget devscripts sudo
$ cowbuilder --create --distribution lenny --basepath /var/cache/pbuilder/testing-base.cow
$ cp -r /usr/share/doc/pbuilder/examples/rebuild .
$ cd rebuild
$ ./getlist lenny
</pre><p>
Now we add Open64 into the cowbuilder chroot and fix up the chroot by pointing the gcc/g++ symlinks to Open64:</p>
<pre>
$ cp -a /opt/open64 /var/cache/pbuilder/testing-base.cow/opt
$ chroot /var/cache/pbuilder/testing-base.cow
$ cd /usr/bin
$ mv gcc gcc.orig
$ ln -s /opt/open64/bin/opencc gcc
$ mv g++ g++.orig
$ ln -s /opt/open64/bin/openCC g++
$ exit
</pre><p>
In addition, we set the <code>CC</code> and <code>CXX</code> environment variables to Open64, which will make 90% of all (autoconf-using) packages automatically use Open64. We need a small script for that:</p>
<pre>
$ cat c.cfg:
export CC="/opt/open64/bin/opencc -m32"
export CXX="/opt/open64/bin/openCC -m32"
</pre><p>
Now edit the <code>buildall</code> script. Change the Debian mirror used there (optional) and make it use our <code>c.cfg</code> script by adding the <code>--configfile /path/to/rebuild/c.cfg</code> option in the "pdebuild" line.</p>
<p>We can now finally start building the archive:</p>
<pre>
./buildall list.lenny.i386 lenny
</pre><p>
You can also run multiple <code>buildall</code> instances at once to speed up the archive rebuild on SMP/multicore machines, and you can even abort the command and simply restart it later. The script will continue where it left off.</p>
<p><strong>Results</strong></p>
<p>The whole rebuild (with 2 instances of <code>buildall</code> running at the same time) took ca. 9 days on an AMD64 Athlon64 X2 (dual core, 1.8 GHz each) machine with 1 GB of RAM.</p>
<p>I really should have used something like <a href="http://packages.debian.org/unstable/admin/apt-proxy">apt-proxy</a> to speed up the rebuild and save some bandwidth, but I read about apt-proxy too late...</p>
<p>All log files from my rebuild <a href="http://people.debian.org/~uwe/">are available</a> for detailed analysis if anybody is interested (you can browse the logfiles online or download all of them as tarball). I didn't perform any detailed analysis, just some rough numbers here:</p>
<ul>
<li>Succeeded package builds: 8425</li>
<li>Failed package builds: 2509</li>
<li>Total number of packages rebuilt: 10934</li>
</ul>
<p>If anybody does some more elaborate analysis, please let me know.</p>
http://www.hermann-uwe.de/blog/rebuilding-the-whole-debian-archive-using-the-open64-compiler#commentsarchivecompilerdebianfortranfungccgplopen64openmpMon, 06 Aug 2007 13:40:51 +0200Uwe Hermann1281 at http://www.hermann-uwe.de