If want you use such settings, which are quite different from the ones used by compiling the Gentoo CD, you have to recompile everything anyway, so using a stage 3 CD will not help.

If you are happy with the defaults used on the CD, however, I certainly would be much faster to make a stage 3 reinstall. (But I have tweaked and twisted my installation so much, that I won't be happy with what has been compiled on the stage3 CD.)

It turns out that in the world.list file, it was showing that ati-drivers and ati-drivers-extra were masked for some odd reason (probably used ACCEPT_KEYWORDS instead of adding it to my package.keywords file!). I added them to my /etc/portage/package.keywords file and then it generated world.list.

that's what I thought... I'll think a little more before doing this. The problem is that while this is being upgraded it's wise if X is not running, but I need my computer almost every day so I think I'll wait some more time for this. I'll check how much free space do I have for another chroot and how could I grab some stuff from the packages cd.

I strongly recommend running this script from the text mode interface, without an X-server running. There are too many things that can go wrong if trying to replace an X server by a recompiled version while it is running. (But on the other hand, I have never tried it. Perhaps it may actually work. But I would not count on it.)

I've always recompiled xorg from within xorg and I don't remember having any problems - I'm not saying you won't have problems just that any problems I had were so minor I never noticed.

Why is the script more efficient than just doing "emerge -e world" once after upgrading to gcc 4.1.1?

Is it possible, in general, to tell what compiler version a package was compiled with? I already compiled some packages with emerge -e system (as per http://www.gentoo.org/doc/en/gcc-upgrading.xml) before reading about this script, but your script tries to compile these again (at least as far as I can see)... I think it would be useful if it wouldn't.

That's why. The order is not optimal. glibc should not be the 75th package to be compiled, but one of the first ones.

Basically, my script does nothing else than run an emerge -e world, only that it reorders items, and excludes GCC from the list, because GCC has already been installed.

"Is 'emerge -e' flawed then?" one may ask.

No. But "emerge -e" assumes there is absolutely nothing there when it starts its job.

Which is not the reality if you are running an emerge from an installed gentoo System.

For instance, you may already have the new GCC, binutils etc. installed.

Exploiting this fact, you can optimize the order in which "emerge -e" does things, and even skip some of the packages (GCC).

Besides that, my script is identical to "emerge -e" world.

It does nothing what you couldn't do manually. (In fact, I did all the steps manually before writing the script.)

So my script is only by one packet faster than what "emerge -e" does.

However - "emerge -e" has not yet done the full job!

Remember those 75 packages that have been compiled before glibc was updated?

"Just to be sure" one should recompile those 75 packages again after the emerge finishes, so that all of them are linked against the new glibc (although I doubt this will be necessary... but that's another story).

My script compiles glibc as the first package (emerges it as the second package; the first package is only header files), and thus makes any this sort of headache obsolete from the beginning.

Erlend wrote:

Is it possible, in general, to tell what compiler version a package was compiled with? I already compiled some packages with emerge -e system (as per http://www.gentoo.org/doc/en/gcc-upgrading.xml) before reading about this script, but your script tries to compile these again (at least as far as I can see)... I think it would be useful if it wouldn't.

Well, that's the problem.

Actually, it is not even necessary to recompile all packages which have been compiled with the old compiler, because the C ABI is pretty stable (see my posting with the explanations for my guide for more of this).

It would be sufficient to recompile only those source files, which have been written in the C++ language by the old compiler.

Unfortunately, it's hard if not even possible to find that out in a safe, automated way (see also my explanation posting).

So we better recompile everything, just to be sure not to miss one of those C++ sources.

Plus we have the benefit that then everything is compiled using the presumably better code generator of the new compiler.

re system profile and glibc. So, I should recompile glibc after setting my system profile? Also, when I did an eselect profile list, it didn't show one as selected. It also did not show 2006.1/server as a choice, only desktop._________________"Blessed is he who finds happiness in his own foolishness, for he will always be happy".

re system profile and glibc. So, I should recompile glibc after setting my system profile? Also, when I did an eselect profile list, it didn't show one as selected. It also did not show 2006.1/server as a choice, only desktop.

Then your system seems not to be up to date, or something is screwed. When I run

I've always recompiled xorg from within xorg and I don't remember having any problems - I'm not saying you won't have problems just that any problems I had were so minor I never noticed.

That's good news!

But - what Desktop Environment are you using?

I suspect some environments to be more fragile than others when recompiling them while they are running.

I'm currently using Xfce. I previously used Gnome.

Guenther Brunthaler wrote:

Erlend wrote:

Is it possible, in general, to tell what compiler version a package was compiled with? I already compiled some packages with emerge -e system (as per http://www.gentoo.org/doc/en/gcc-upgrading.xml) before reading about this script, but your script tries to compile these again (at least as far as I can see)... I think it would be useful if it wouldn't.

Well, that's the problem.

Actually I think it could be done if you ask the user when they changed their compiler to (say) gcc 4.1.1. Just look up emerge.log for any packages done after this date (although they would probably not be linked against the new glibc if you do this).

Guenther Brunthaler wrote:

Plus we have the benefit that then everything is compiled using the presumably better code generator of the new compiler.

Off topic, but I can't wait until icc works as a drop in replacement for gcc (if ever)!

talk abt postcount++++++++_________________The best argument against democracy is a five-minute conversation with the average voter
Great Britain is a republic, with a hereditary president, while the United States is a monarchy with an elective king

Hmm that's also one of the leaner ones. The odds of "surviving" a recompile without crashing are certainly quite better with such a DTE than for KDE!

Furthermore, can you tell whether Xfce is written in C++ or C?

The maximum damage is done only if C++ Shared Objects are heavily used (as plugin DLLs).

I also assume Gnome to be much less effected by a recompile than KDE for that reason.

Erlend wrote:

Actually I think it could be done if you ask the user when they changed their compiler to (say) gcc 4.1.1. Just look up emerge.log for any packages done after this date (although they would probably not be linked against the new glibc if you do this).

Yes, that might actually work.

And it can rather easily be included within my guide: After the recompilation script has been generated, remove any lines from it which refer to packages found in the output of emerge.log since the new compiler was emerged and was made the system default compiler. (Because otherwise it will not have been used although it was installed.)

A small Perl/Python/Ruby/awk script will do.

Unfortunately, it will only work for people who have enabled portage logging - it's still not the default, I'm afraid. (Is that correct?)

Erlend wrote:

Off topic, but I can't wait until icc works as a drop in replacement for gcc (if ever)!

it took me 2 days, but my kde laptop survived while using it extensively at work. I even hibernated 2 times.

sure kde is plenty of c++, but once a shared lib is loaded, it stays in mem until unused, so you can safely use your computer while emerging world. just be sure not to close c++ programs you wish to use (and start them beforehand of course). then, at the end of the upgrade, restart X, or better even, reboot, so the previous libs are flushed and latest libs reloaded.

I basically didn't noticed it up until I accidentally closed kmail, which then complained about the ABI mismatch. but who cares, I waited next day to check my mail again. just a fine, hassle-free upgrade._________________Moved to using Arch Linux
Life is meant to be lived, not given up...
HOLY COW I'M TOTALLY GOING SO FAST OH F***

just be sure not to close c++ programs you wish to use (and start them beforehand of course). then, at the end of the upgrade, restart X, or better even, reboot, so the previous libs are flushed and latest libs reloaded.

This sounds very promising, indeed.

I have taken the liberty to update my guide accordingly and refer to your post!

do you know about app-portage/genlop?
I didn't use your script, but I thought it would be a nice addition to your script to output the result of 'emerge -p `cat list_of_packages` | genlop -p' (i.e total estimated merge time) before actually running 'emerge `cat list_of_packages`', and have a yes/no question like 'emerge -a'.[/url]_________________Moved to using Arch Linux
Life is meant to be lived, not given up...
HOLY COW I'M TOTALLY GOING SO FAST OH F***

Interesting. But I'd recommend install ccache (make sure it's enabled in FEATURES in make.conf) and do it the usual way: emerge -e system and then world. It's probably about 30min longer than the "optimal" script. But you'll be 100% sure, everything is done correctly.

Interesting. But I'd recommend install ccache (make sure it's enabled in FEATURES in make.conf) and do it the usual way

Without question, ccache is a great tool. I'm using it all the time.

And it works perfectly with my script. (At least I did not encounter any problems. However, I manually cleared the cache of ccache after using gcc-config to set the new compiler. Just to be sure there are no cached object files left originating from the old compiler.)

vicaya wrote:

emerge -e system and then world. It's probably about 30min longer than the "optimal" script. But you'll be 100% sure, everything is done correctly.

So, if you want to be "100 % sure", "emerge -e world" won't provide you with what you want.

My script does the same as an "emerge -e world", only in the correct order for recompiling a system.

Also note that dispite of this, "emerge -e world" is neither flawed nor broken! It's just not intended to be used for recompiling the whole system, but for rebuilding a non-existent system from scratch. Which is not the same!

My script, in contrary, has been designed for recompiling your entire system.

So it does a similar, but not the same job.

Last edited by Guenther Brunthaler on Mon Sep 04, 2006 9:01 pm; edited 2 times in total

Interesting. But I'd recommend install ccache (make sure it's enabled in FEATURES in make.conf) and do it the usual way: emerge -e system and then world.

ccache won't help in this instance because the compiler version has changed.

vicaya wrote:

It's probably about 30min longer than the "optimal" script. But you'll be 100% sure, everything is done correctly.

You won't be 100% sure, because as stated above 70-or-so packages won't be linked against the newly compiled glibc... this is why some people say you should do

Code:

emerge -e system
emerge -e world
emerge -e world

The optimisation this script makes is to place glibc at the beginning of the compile run, so everything thereafter is compiled against it and running "emerge -e world" twice is definitely unnecessary.

Guenther Brunthaler wrote:

Yes, to the environment! Thousands and thousands of Gentoo users, all wasting weeks worth of processing power when following inefficient "recompile entire system"-style compiler upgrade guides, will waste an enormous amount of kilowatt hours of electrical power.

How much more electricity does it use? The processor will go to 100%, and I guess the hard drives will thrash? Hard drives are always spinning though, so I don't think they'll really use significantly more power whem read/writing - it'll just kill the disks. The fans will probably be going quite fast too?

Also note that dispite of this, "emerge -e world" is neither flawed nor broken! It's just not indended to be used for recompiling the whole system, but for rebuilding a non-existent system from scratch. Which is not the same!

There should be some way of doing what your script does from within portage... like emerge --rebuild-upgrade world.

Hey I just happened to spot your script re-emerging =sys-libs/libstdc++-v3-3.3.6 ... surely I don't need that program any more?

Well, my script recompiles any package that is installed at the time it is generated. It doesn't care about whether the packages are up-to-date or outdated. (That's why I want users to run "emerge --update --newuse --deep world" first, before starting my script.)

So it seems your package is simply left from an old compiler and never has been uninstalled.

OK, now it is too late... but as soon as all is finished, run a

emerge --ask --depclean

and see what portage suggests for uninstallation. (And don't forget to run revdep-rebuild in order to fix any packages which might have been broken by those uninstalls.)

Another common problem are old slot installations.

For instance, GCC is not replaced when you emerge a newer version, but installed additionally.

Same for gentoo-sources, by the way.

If you want to see what slots of a package are actually installed, try

Code:

equery list gcc
equery list gentoo-sources

Then use "emerge --unmerge" to uninstall the slot versions you do not need. But be sure to specify the exact version number for the unmerge, or else the newest version will be removed instead!

Hey I just happened to spot your script re-emerging =sys-libs/libstdc++-v3-3.3.6 ... surely I don't need that program any more?

Well, my script recompiles any package that is installed at the time it is generated. It doesn't care about whether the packages are up-to-date or outdated. (That's why I want users to run "emerge --update --newuse --deep world" first, before starting my script.)

So it seems your package is simply left from an old compiler and never has been uninstalled.

Sorry I meant in general... I don't think I need libstdc++v3 any more - I think I emerged it to fix any problems that might occur in the 3.5 -> 3.6 upgrade.

You won't be 100% sure, because as stated above 70-or-so packages won't be linked against the newly compiled glibc...

Yes!

Erlend wrote:

How much more electricity does it use?

It certainly depends on the processor and its power-saving capabilities.

And also on the OS.

Unless disabled, Linux will perform a HLT instruction on the processor when no processes need running.

Then the processor "sleeps" until the next interrupt, requiring only very little power.

But when running under full load, such as recompiling QT libraries - I'm quite sure the processor needs much more power then, in comparison to "sleeping". (You can even hear it work when listening to the variable fan noise.)

And also don't forget: Some people turn off their machines at night! But for the updates they have to let them running for a week or so. Certainly this also uses more power than usual.

Erlend wrote:

and I guess the hard drives will thrash?

At least they will if those horror-stories from GCC 4 are actually true - I've read reports where the machines were running out of swap space, because GCC4 needed more than 2 gigs to compile a single C++ file.

I can only hope those guys were exaggerating a bit... because one of my machines has only 512 megs.

But for now I'll stay with 2006.0 for a while - I have updated my GCC from 3.3 to 3.4 not more than two months ago - it's no fun recomiling everything again so soon ;-(

Erlend wrote:

Hard drives are always spinning though

Yes, but there are 2 motors in a hard drive: The normal one
which makes it spin, and the stepper motor. At least in older hard disks there was one. Newer hard disk have something else (I think its called "actuator" or similarly), but it certainly needs also more power if its moving the heads, than if not moving them.

Erlend wrote:

The fans will probably be going quite fast too?

You can bet on that! Especially on my notebook here - "you can hear the sound"...

Last edited by Guenther Brunthaler on Mon Sep 04, 2006 10:17 pm; edited 1 time in total

At least they will if those horror-stories from GCC 4 are actually true - I've read reports where the machines were running out of swap space, because GCC4 needed more than 2 gigs to compile a single C++ file.

Ah that explains it... my swapfile is going through the roof (and I can only remember 3-4 occassions when it's been used before).

But it's not just the swapping that'll use disk space, there's also the fast creation of hundreds of small files in /var/tmp, although I think PORTAGE_TMPFS="/dev/shm" in make.conf is meant to lower this.