I encountered the same problem. I found out that the script always dies when a line ends with an "escaped line feed" (don't know how it is called). Remove the line feed and put all the commands in one line.

I encountered the same problem. I found out that the script always dies when a line ends with an "escaped line feed" (don't know how it is called).

Thanks for that more accurate problem description!

I therefore conclude the origin of the problem must be the following: When copying/pasting the script, whitespace has somehow been added after the backslash which was not there in the original script.

In order to fix things, that white space must be manually removed: The backslash must be the very last character in the line! No spaces/tabs are allowed to follow.

Seems your editor somehow screwed up the pasted text by inserting some whitespace after the backslashes.

COMKEEN wrote:

Remove the line feed and put all the commands in one line.

Originally, this was exactly as the line looked! But as I never allow any line in one of my scripts to get longer than 79 characters, I inserted escaped line feeds in order to break long lines.

For the future, I will try to find a way to avoid the escaped backslashes, in order to make the script even work if an editor unintentionally adds whitespace at the line end.

UPDATE: I have updated the script in the article by an updated version (1.12) of the script which avoids all escaped backslashes. (As an unintended side effect, it looks even more cryptic now. But who cares.)

I encountered the same problem. I found out that the script always dies when a line ends with an "escaped line feed" (don't know how it is called).

Thanks for that more accurate problem description!

I therefore conclude the origin of the problem must be the following: When copying/pasting the script, whitespace has somehow been added after the backslash which was not there in the original script.

In order to fix things, that white space must be manually removed: The backslash must be the very last character in the line! No spaces/tabs are allowed to follow.

Seems your editor somehow screwed up the pasted text by inserting some whitespace after the backslashes.

Actually it is not the text editor's fault, but the browser's fault. I am using Konqueror and when I am copying/pasting your code snippet into a text editor (no matter whether I use a GUI editor like KWrite or a command line editor running in Konsole), there are additional whitespaces inserted at the beginning as well as at the end of each line. When I am copying the script from Firefox, everything seems alright.

To remove the leading and trailing whitespaces, one could use this command:

Hey, what about when two versions of the same package are installed? I have dev-libs/lzo-1.08-r1 and dev-libs/lzo-2.02-r1 installed. I did a "emerge --sync && emerge --update --deep --newuse world && revdep-rebuild" and everything is good there. lzo-2.02-r1 is what is installed with an "emerge -u lzo" and kino needs liblzo.so.1 from lzo-1.08-r1. You can see that they are both installed below.

This must be the method that the script uses to find installed packages although it doesn't really list them all other wise it would have found lzo-1.08-r1 also. Maybe something like the following could be used to generate your list of installed apps, minus the sys-devel/gcc and old sys-kernel/* or what ever other stuff you know doesn't need to be installed again :

Guenther, great job!
Check out my script and how I've made it interface with yours for interruption-free compiling of the entire system (in the correct order)._________________"We must all hang together, or assuredly we shall all hang separately."
-Ben Franklin

But in this case both installed versions were found by your script. I wonder what I might have done wrong to mess up the other case.

My script is based on the output of "emerge --emptytree". It re-orders that output and removes duplicate and unnecessary emerges, but will not add anything to that list.

In your case it seems that Portage thinks both GPG-versions are necessary, but not both of the LZO versions.

This might be possible if you left out the optional "emerge -a --depclean"-step of my guide, which will remove unnecessary packages from your system.

But as emerge --depclean is "known to be broken" (as it's own output says), I did not want to make it a non-optional step of my guide.

So, what can you do?

I suggest the following approach:

Run

Code:

emerge -avuDN world

to be sure everything is in a consistent state from emerge's point of view.

Run

Code:

emerge --ask --depclean

. Make sure no packages are suggested for removal which you obviously don't want to be removed. Accept the removal of suggested packages only if you agree. This should remove any unnecessary packages.

Run

Code:

revdep-rebuild

repeatedly until no more packages can been found containing broken library dependencies. (revdep-rebuild is part of gentoolkit.)

If you are lucky, your old LZO package should have gone by then.

If it's still there, you can use equery depends to check whether any other package actually depends on the old LZO version.

If you don't find any dependent packages, just remove the old version of the LZO package manually using emerge --unmerge.

In order to be sure, use qpkg before the removal and create a binary archive from the LZO version to be removed. If it later turns out the old version is actually required by anyone, you can easily re-install it.

As a general precaution when removing shared libraries, I would also suggest emerging and running sash, the "Stand-Alone Shell", in a different window before removing the library. (A statically-linked busybox can alternatively be used for the same purpose.) sash is a statically-linked shell which will even continue to run if most other tools and shells won't because of shared library problems. In such cases you can use the already-running sash to fix things even if bash won't work any longer.

I have just updated the script and guide as of version 1.15. The script now features interruption-free operation, just as you suggested. Your contribution has been mentioned in the guide.

After the package list has been processed, the script automatically tries to recompile the packages again which failed earlier, hoping it will work then. This process will continue recursively, but care has been taken to avoid an infinite loop.

I hope this new approach will also defeat any issues that might arise from circular dependencies in ebuilds.

This all brings me back here with some things that I have learned that someone may want to take into consideration with this howto. cd to the dir with the recompile-remaining-packages script and do the following:

To list the packages that the recompile-remaining-packages script will not recompile but emerge -e world would:

Code:

diff -bn emerge.tmp recompile.tmp|grep =

For me this gave back:
=sys-devel/gcc-4.1.1
=sys-devel/gcc-config-1.3.13-r3
Pretty much novelty information.

What packages are installed but will NOT be recompiled by "emerge -e world":

Code:

diff -bn emerge.tmp db.tmp|grep =

I got 4 packages listed here that would NOT be recompiled:
=dev-libs/lzo-1.08-r1
=sys-devel/automake-1.4_p6
=sys-devel/automake-1.8.5-r3
=sys-kernel/gentoo-sources-2.6.17-r4
This information is important because these packages are also not recompiled by the recompile-remaining-packages script. Btw, lzo is in that list because of the lack of it's DEPEND in the kino ebuild so I --oneshot it in to keep it out of the world file(as stated above, I already filed a bug about this). I don't know why the other three are not recompiled by emerge -e world. I have 6 versions of automake but only 2 are not recompiled by emerge -e world, and those 2 are not removed by --depclean.

Now I'm off to figure out a one-liner to tell which packages in the world file are dependencies and so may be removed from the world file.

Also, my world file is located at /var/lib/portage/world, not /var/portage/world as this howto has it. Is there a default or even an option to change where the world file is located?

Thank you... and I hope that I'm not being too annoying here._________________AssociateX
Gentoo rocks!

Last edited by AssociateX on Sat Sep 23, 2006 3:46 pm; edited 1 time in total

ABSTRACT: How to recompile your entire system / toolchain with MINIMUM PROCESSING TIME EFFORT.

If you plan to do a major compiler update for your Gentoo box, such as from GCC 3.3 to GCC 3.4, or from GCC 3.3/3.4 to GCC 4, then read on. (This will typically be the case when upgrading your Gentoo installation to the new 2006.1 profile.)

Guenther, you are a star . I have had several trying periods over the last few years with GCC upgrades, and system recompilation, and your script has just made this process much smoother. I am now in the final stages of a full recompilation using your script, and I am sure it has saved me much heartache. Due to my misunderstanding of the changes to the profiles (specifically the need to move from 2006.0 to 2006.1/desktop, not 2006.1) I have had to carry out the compilation operation twice. First time I used the time honoured method, and this time with your script.

Hi
Firstly, thanks for the script!
I'm using the latest version that you uploaded today. However I get an error (unexpected token >) at line 62 of the recompile-remaining-packages script, which is `exec > >('

Any ideas?

Edit:
Also, is it really necessary to emerge world newuse etc before running the script? Surely these are all going to be compiled anyway?

I get an error (unexpected token >) at line 62 of the recompile-remaining-packages script, which is `exec > >('

Perhaps you might have been using a very old version of bash? My script assumes a moderately modern version of bash.

The >(...) expression is a process substitition, which will be replaced by the filename of an ephemeral named pipe, which will be connected to the statements in the parentheses. Those statements will be run as a background process, and will read from the pipe; implementing the log file feature.

That is, the

Code:

exec > >(...)

is actually an

Code:

exec > anonymous_pipe

where anonymous_pipe has been constructed on the fly by bash.

However, process substitution is not a feature that all shells provide. For instance, ash does not.

That's why my script starts with

Code:

#! /bin/bash

and does not contain the usual

Code:

#! /bin/sh

steveL wrote:

Also, is it really necessary to emerge world newuse etc before running the script? Surely these are all going to be compiled anyway?

Yes and no. The problem is, my script assumes a working compiler and tools to be available when it starts its job.

That is, tools like gcc, ld, glibc, make, sed, bison, flex, cp, ar, as, libtool and any other tools involved in the process of an "emerge" must be in a consistent state.

But certainly it will be unnecessary to bring packages like openoffice up to date just for rebuilding the whole system.

However, most people run

Code:

emerge -avuDN world

on a regular basis, which means their systems are close to being up-to-date anyway, and the additional emerge won't hurt.

If you know what you are doing, it is certainly possible to just do a

Code:

emerge --sync

and omit the the emerge -avuDN world step.

But I will not take any responsibility for problems arising from an improperly functioning toolchain, in case the tool chain was in an inconsistent state.

Well, actually they are not really anonymous from the viewpoint of the kernel, but rather temporary: Bash uses mkfifo or something similar internally to create a pipe with a temporary name in /tmp (I guess), then runs the process substitution, and eventually removes the pipe automatically.

Which means the pipe will actually have some name, but from the viewpoint of the shell script it works like an anonymous pipe. It's also fun and easy to use!

...as long as it works. What seems to be a problem with your system configuration.

steveL wrote:

I am running bash-3.1_p17; while I don't know if this is old

My bash --version gives "3.1.16(1)-release (i686-pc-linux-gnu)", from which I conclude your bash should be good enough.

So the problem you encountered must be triggered by something else.

steveL wrote:

Is there anything else that could cause this?

Perhaps it might be easiest to run a few tests manually before going into details how my script works.

My bash --version gives "3.1.16(1)-release (i686-pc-linux-gnu)", from which I conclude your bash should be good enough.
..
Perhaps it might be easiest to run a few tests manually before going into details how my script works.