OK, so you are suggesting to do the gcc-config before the emerge --update then.

It should certainly not a be problem changing this in my guide, but first I have to understand what exactly the problem was.

From what I've seen in the link you provided, they were actually getting a "gcc too old"-sort of error message.

But in our case, GCC will be up to date when the glibc will finally be compiled by my script, so I am uncertain on what step of my guide the problem actually occurred in your case!

So far, my the logic of my guide is:

The initial emerge --update brings all packages up-to-date, including glibc and the old GCC as well as all the tools required for rebuilding the system.

Then the new GCC is emerged. Now both GCCs are installed, and still all packages are up-to-date.

Now the used runs gcc-config and sets the new GCC as the system default compiler. But the user does not use it to compile anything with it yet.

Instead the user starts my script, which will generate a package list of all packages in the system, and also determine the correct order in which they should be recompiled. This includes glibc.

The user starts the generated script. glibc should be emerged as the second package of all, and the first one to be actually compiled. This will already be done with the new GCC, which is the system C compiler now.

If there is an error in this flow of logic, please be more specific and tell me at what point of the above list the problem occurs.

I may then be able to realize the origin of the problem. (Let's hope!)

Well, I did mine the hard way. I didn't see this guide before I started. I have ran revdep-rebuild about 3 times now. It wants to rebuild gcc each time and it just goes in circles. Do you have any reason why it does this?? Info:

Hard to say, as it certainly depends on the set of packages you have installed.

But I can acknowledge the problem per se, although I experienced it with some different package. (Don't remember which one.)

However, I still remember that I found the origin of the problem: It was the consequence of a mixture of normal Gentoo packages installed from source code, and prebuilt binary packages.

It was either mozilla-firefox-bin, mozilla-thunderbird-bin or openoffice-bin. Don't remember which one.

After uninstalling the guilty bin package, the circles were gone.

I think the problem were the USE flags.

The faulty bin-package was compiled without some optional feature, which was required because of USE-flag settings by another package.

Because of this missing feature, some library entry point in some shared library from the bin-package was undefined, although the other package required it (as a consequence of the USE-flags).

revdep-rebuild detected the missing library-entry, and thus scheduled the faulty bin-package for re-emerging.

But as re-emerging does not change anything for a bin-package, the problem continued to exist after the re-emerge, and so the next run of revdep-rebuild detected the problem again, emerging the package again... and again... and again...

Thanks for the info. As I said I ran into this before and I usually let it do it twice then I figure the point is pretty much mute. I don't think I have to much binary stuff installed on here though. I even compile OOo on this thing. I may start a seperate thread, as much as I hate to, and see if it is a bug that needs to be dealt with before it does break something for someone else.

I did get your script and I stopped the emerge -e world that I had running. I then ran it and it seems to be working fine. I noticed that it compiles the kernel stuff first and that it said to compile glibc again during the install. I thought hmmm, never saw that before. Then it started emergeing glibc just like it said too. That was sort of funny in a way. It's like the script was reading the things mind.

Anyway, I'll try revdep-rebuild again after your script gets done. If I get the same thing I'll start a thread.

emerge -e system and then world. It's probably about 30min longer than the "optimal" script. But you'll be 100% sure, everything is done correctly.

I that only were so easy ... I would never needed to write my script.

Sorry, what I really meant is that just follow the official gcc upgrade instruction. emerge gcc etc... see http://www.gentoo.org/doc/en/gcc-upgrading.xml code listing 2.1 and then emerge glibc && emerge -e system && emerge -e world. With ccache, it won't be that much slower (considering the total emerging time) than any "optimal" script.

What, you didn't know that linux users are thieves, liars, terrorists, and long haired smellies? Where have you been

The Bad Penguin is a barb at the litigious assholes known as the SCO Group, in particular Darl McBride, who has falsely accused the entire open-source community of being thieves...

Guenther Brunthaler wrote:

But you are not the first one to find a flaw in my guide. Hey, I'm only human! I'm not perfect, everyone can make as mistake and I'm no exception. That's why we discuss ideas like this in the forums, don't we?

For instance, devsk has already pinned down a potential problem related to circular dependencies, although it seems that (by coincidence I must admit) my script will not be affected by it. See his posting for more.

The flaw is not so much in the guide, but in making definitive, authoritative sounding statements that have not been tested or proven. I am just anal that way

Guenther Brunthaler wrote:

I have not even an idea how to prove the correctness of a computer program with as far-reaching consequences as the application of my script. Perhaps if I wrote it in ADA instead as a BASH script...

Perhaps by building it via your method and running the test suites of every package as you go (FEATURES="test")? The only way to be certain is to actually use your method in some different environments and see how many bugs are produced as a result.

Guenther Brunthaler wrote:

Bad Penguin wrote:

As far as your second assertion, that your method is somehow faster, wrong again.

Here we obviously have a simple misunderstanding: When I wrote

Quote:

Compared to other guides which suggest recompiling the entire system up to 6 times

I didn't mean I my guide was faster than all the guides which may exist, only faster than those which recommend excessive recompilation of the system over and over.

Which, by the way, was the primary motivation for writing my guide: 6 times was too much!

Heh heh, I have never seen a guide that suggests to do it 6 times. The Rob Moss method suggests to "emerge -e system && emerge -e system && emerge -e world && emerge -e world", which compiles the system packages four times and the remaining packages (world) twice. This method is not very useful for toolchain updates though, which need to be done in the correct order.

Guenther Brunthaler wrote:

As you have read my argumentation and how my script works, you might have noticed that it builds a list of all packages to be recompiled when the generated build script is run.

This list countains all the packages which will be rebuilt in the correct order, including any dependencies.

An incorrect assumption. If you are merely recompiling every package on your box for fun, or a change in USE flags/CFLAGS, you would be correct. The only reason to ever recompile every package would be a toolchain update, which needs to be done in a specific order, which is not handled by portage correctly.

Guenther Brunthaler wrote:

But an update like the 2006.1 switch is mere overkill for the power of your script, where my simple guide is obviously sufficient for such minor cases.

That depends on which profile and versions of everything you are upgrading from. Not everyone is going to be switching to the 2006.1 profile from a 2006.0 profile. Your script might work in a very small set of cases, in other cases it might turn out to be disastrous. You are leaving one very important factor out of your method - the human factor Bugs happen, people don't always know to go to the gcc/glibc migration guide, which gives out bad advice anyway. Use revdep-rebuild on your system and you will soon find out that you have old .la files laying around that will not be removed by emerge because the files have been modified by an outside process. Who the hell knows what code is actually being statically compiled into your packages as a result.

Guenther Brunthaler wrote:

But why should such a compiler actually be faster than the compiler emerged at the beginning of my guide, which fully honors all MMX/SSE flags and also uses the full optimization settings as defined in /etc/make.conf?

And from then on, my compiler has to recompile all remaining packages in the system exactly once, according to my guide.

During a bootstrap the USE flags are only restricted during the bootstrap phase, when the actual toolchain packages are built. The subsequent emerge -e system uses any USE flags in make.conf. One example of why this is useful is the solitaire game pysol - which requires tk. If all use flags were allowed in the bootstrap phase the compile of python will pull in X to build tk, then you would have to build X all over again during the subsequent emerge -e system.

Guenther Brunthaler wrote:

How can your script, starting with a slower compiler, do the same in a faster way?

So your script/guide has to recompile at least exactly the same number of packages as my guide does.

It does not actually compile anything faster, it does it more efficiently. The important thing is not that your packages compile fast, it is that they compile correctly so you don't have to go back and do it all over again later. But no, my "method" actually only ends up compiling the bootstrap packages twice along with a small set of the bloated needs of portage (python). You start with a cleanly bootstrapped system, the remaining packages are all compiled on a clean toolchain.

Guenther Brunthaler wrote:

The "overhead" of my guide is the initial "update world", and the overhead of your guide is the overhead of a stage1 reinstall.

The "overhead" of your guide is that chances are it will really screw up peoples systems Sadly, there is no way to guarantee that every package on the system is built against current libraries that are built against current libraries... yadda yadda yadda... without a brute force, recompile it multiple times approach. There are just too many circular dependencies and too many different factors when you take USE flags into account. The safest, and fastest way, for the large majority of people, is to start from stage1. This is unique to gentoo because it is a source based distribution... Too bad gentoo has abandoned the build from source approach, instead encouraging people to hack up their boxes... I digress.

Anyway, if you are interested, I will write up some scripts that take a box from a specific state, say portage-20060301, to portage-20060905. One script that does it your way, one that does it my way, and we can add FEATURES="test" to see how it turns out

Well, the script just finished. I have the same problem that I had before the gcc upgrade just with the newer version so I guess all is well. It seems revdep-rebuild just likes to emerge gcc in a circle.

Anyway, if you are interested, I will write up some scripts that take a box from a specific state, say portage-20060301, to portage-20060905. One script that does it your way, one that does it my way, and we can add FEATURES="test" to see how it turns out

Interested, yes, of course.

But I'm afraid my knowledge about Portage and the internals of ebuilds is not yet mature enough to write any "test" scripts myself. Those internals are mostly still black magic for me.

In fact I have not even finished studying the autoconf guide - and I'm even farther from writing my own ebuilds.

After all, I'm relatively a Linux newbie and there are still many aspects of the system which I don't fully understand (or understand at all). Although I am trying hard to learn. But it takes time.

I have read through this thread, I think it will solve my problem but Im not entirely sure,.
I have recently migrated to X Server 7.0, which I sort of got working, (but so much is broken its not useful!).
Some of the new X packages (glibc I think) failed to compile, due to an insistence of using gcc 3.4.6 not 3.3.6, so I switched and tried again. The problem was when I recompiled my kernel, all of the modules refused to load, complaining of magic version different. (the complier version namely), but I did recompile all the modules with the same complier as the kernel. (or at least I think I did )
What I want to know is :
Is there some other setting I need to change other than gcc-config to ensure modules and kernel are compiled using the same version?

Does 2.6.12-gentoo-r4 HAVE to be compiled with 3.3.6?

I got round this issue, by compiling the kernel and modules with 3.3.6 everything else with 3.4.6, but now I have messed up because the ATI fglrx module complains about missing symbols. I pretty sure your script is what I need, but I need to know what else I need to do first, not mad about wasting electricity and all that.

BTW, I'm currently using 2006.0 profile, shall I switch to 2006.1 and just use gcc 4.1 saving time later?

Is there some other setting I need to change other than gcc-config to ensure modules and kernel are compiled using the same version?

No, that should be sufficient.

However, you also need to recompile all the packages compiled with the old compiler, because mixing packages compiled with different major compiler versions can cause troubles.

Unfortunately, portage is not aware about those problems, and will not automatically recommend recompiling all the packages. You have to do this on your own.

Common sense is that all packages should be recompiled at least once, which is what my script does.

There are different points of view whether this is enough; I say "yes". (See my argumentation link in the article.)

Recent major GCC updates which require recompiling the system are: From pre-3.4 to 3.4, and from pre-4.0 to 4.x.

As you are updating from 3.3.6 to 3.4.6 or 4.1 you are affected anyway.

BTW: Just switching back to 3.3.6 won't help if you already compiled some packages with 3.4.6: Either you recompile all those packages again with 3.3.6, or you will end up in a system with mixed versions, which will cause troubles sooner or later for sure.

I thus suggest completely updating your system to either 3.4.6 or 4.1 to have a stable system again. You may use my script for this, or some of the other guides (which take a longer time, but their authors believe it's better. I don't believe this, but you decide! The most pragmatic version is probably using my guide first, because it's fastest under normal circumstances, and if you should be so unlucky to be the first one having problems with it, then you still can use one of the other guides.)

ANewFishMonkey wrote:

Does 2.6.12-gentoo-r4 HAVE to be compiled with 3.3.6?

No. The only known limitation I am aware of is that GCC4 cannot be used to compile a kernel 2.4 or older.

3.3.6 as well as 3.4.6 or 4.1 should not have any problems compiling a 2.5+ kernel.

ANewFishMonkey wrote:

ATI fglrx module complains about missing symbols.

I also had that problem, although with the NVIDIA drivers.

It was not a problem of the compiler, but of the kernel version: They had changed something in the new kernel version, which also required a corresponding change in the NVIDIA code.

When an update was available to the NVIDIA binary drivers, all the problems went away.

In meantime, I was using the open source drivers instead. Although they had inferior 3D speed (= unusable for games), there was no difference for normal X applications.

So I did not play 3D games for a couple of weeks, but otherwise there was no disadvantage using the open source X11 drivers until the new NVIDIA binary drivers were finally released.

I'm pretty sure things will be the same for ATI.

ANewFishMonkey wrote:

BTW, I'm currently using 2006.0 profile, shall I switch to 2006.1 and just use gcc 4.1 saving time later?

Yes, you can. I did it also. Just look into the official Gentoo GCC update guide (but don't follow it if you want to use my guide). They have a paragraph there descibing how to mask GCC4 in the 2006.1 profile, so that GCC 3.4.6 will be used instead.

But - is there a specific reason why you won't be using GCC4? If you recompile your entire system anyway, why not using the best compiler if you can, which certainly is 4.x?

Sure, 4.x works slower than 3.x, which means it takes more time to recompile your system.

But then, 4.x also generates better code, which means the programs it compiles will run (slightly) faster then. GCC4 is slower because it does a better job, which takes more time to complete.

The only problem of GCC4 is its hunger for RAM. If you have a box with less than 512 megs, it may be better to stay with 3.4.6.

But otherwise, I would suggest using 4.x.

Last edited by Guenther Brunthaler on Sat Sep 09, 2006 10:36 pm; edited 1 time in total

While compiling with kde, firefox & al running, I wouldn't call that exactly hunger. it would fit in 256M. (btw, currently compiling OOo203)_________________Moved to using Arch Linux
Life is meant to be lived, not given up...
HOLY COW I'M TOTALLY GOING SO FAST OH F***

I had no problems first time through all the way to completion, 40 hours later. Only failure was mod_gzip fails to compile.
Gzip bug_________________"If the automobile had followed the same development cycle as the computer, a Rolls-Royce would today cost $100, get a million miles per gallon, and explode once a year, killing everyone inside."

Hi, thank you for the script. My system is being compiled right now, as we speak . I have an issue though which I resolved in two ways. I am not sure the first solution hasn't spoiled my system, though.
Namely, during the compilation process the script stopped with an error at a package from an overlay tree. No thinking much I unmerged the package and ran the script again. Of course it didn't work that way, because the package was still on the list of the script. So I removed the files the script generated (script.ar, remaining-packages, entire -system). I recreated the script and started anew. At this point I do not know whether gcc and linux headers were compiled again or not?

the script failed again at qemu-user and qemu-softmmu. This time I thought it would better be that I find a way to skip the compilation of the packages instead of removing them from my ssystem ans start from the beginning. So I found out that I can comment out the troublesome packages in the recompile-remaining-packages script. It worked.

Is this the only and correct way of recovering form a failure of a uncompilable package by commenting it out?

Thank you
Agryppa_________________The first successor of Saint Peter was Linus (a.d. 68-79) - whose namesake became the creator of Linux in our time. Torvalds' middle name is Benedict - the name assumed by the previous Pope who resigned from office.

If I understand correctly Your script recompiles the packages in the exact versions as installed at the moment whith the same use flags (cause you say to do #emerge --ask --update --deep --newuse world before running your script)

So I assume I shouldn't use this script to switch to hardened profile from 2006.0?

Do you think it does make sense to do emerge -avu --deep --newuse world and then run your script?
I am not sure how much packages are affected with use flags change (hardened impicates at least -nptl hardened flags)

First of all, I don't want to offend anybody. I haven't tried this script, so I will say my opinions a priori:

1.- If this script works better than the emerge -e system && emerge -e world stuff that we can see in the official guide, why don't try to contact a developer to talk about this? Maybe the script can become an official part of Gentoo in the future. Have you contact any developer? What he/they say about this? Messing with the system like this is a dangerous thing, I don't want to break anything.

2.- Don't mean to offend you, but reading you post, the way they are writen and the way you explain things, I think that you are acting like a televangelist or an infomercial vendor. This undermines you credibility. If this thing works, talk with some devs in a chat, or something. But trying to "sell your product" with "Because the amazing truth is: It is in fact UNNECESSARY to recompile any package bla bla bla" or other similar quotes is not the best idea.

Personally I would wait to see what devs think about this._________________I cannot write English very well. Please, correct any mistake so that I can improve.

No thinking much I unmerged ... So I removed the files the script generated (script.ar, remaining-packages, entire -system). I recreated the script and started anew.

You did well. This should fix any potential problems.

agrypa1 wrote:

At this point I do not know whether gcc and linux headers were compiled again or not?

"linux-headers" is really just that - a bunch of header files. They will only be downloaded and installed, but nothing will ever be compiled by emerging that package.

Besides that, re-emerging "linux-headers" is the very first thing my script does - so you certainly need not be worried about it.

GCC is another story: It won't be re-emerged by my script, because it is not necessary: If you were following my guide, you did this yourself already.

And as you can see in my related argumentation article ("myth 1 - GCC gets better each time it is re-emerged"), there is no point in re-emerging the same version of GCC ever again. It cannot get any better than it already is, no matter which profile or glibc was current at the time it was emerged.

So you also need not worry about re-emerging GCC too.

(The only case when re-emerging the same version of GCC makes sense is if the CFLAGS in /etc/make.conf or the USE flags for GCC have been changed by you.)

agrypa1 wrote:

Is this the only and correct way of recovering form a failure of a uncompilable package by commenting it out?

No, the lines could be deleted as well. But I preferred instructing the users of my guide to comment the lines out, because that way it's easy to un-comment the lines again in case they did comment out the wrong lines mistakenly.

Also, just commenting out allows looking at the script file contents after the system rebuild is complete, and see which packages were skipped.

Then the users could decide to try re-emerging those missing packages again, hoping it will then work as anything else will already be up-to-date.

If I understand correctly Your script recompiles the packages in the exact versions as installed at the moment whith the same use flags

That's correct. I wanted to use a stable version snapshot at some instance in time, or otherwise an imprudent "emerge --sync" could easily make an ongoing system rebuild operation break.

Using a stable version snapshot of all packages, the chances for an intermittent "emerge --sync" to break things are much lower.

s_wilk wrote:

So I assume I shouldn't use this script to switch to hardened profile from 2006.0?

There should not be a problem if you switch the profile before starting my script, because then the version snapshot will be stable again.

My script generates the package list based on the current system profile at the time the generator script is called.

And as my script will recompile everything anyway, all packages will be compiled according to the settings of the new profile.

With one exception: My script will never recompile GCC, because this will already have been done manually when users were following my guide.

This assumes however, that GCC has been emerged with the correct USE-flags and CFLAGS already set in /etc/make.conf (i. e. after the profile has been changed.)

s_wilk wrote:

Do you think it does make sense to do emerge -avu --deep --newuse world and then run your script?

Well, my guide actually requires the user to do so before invoking my script.

However, in your case this might actually be an overkill, because if the USE-flags changed by the hardended profile were all that changed, you can save some time by using the following approach:

Do the emerge -auvDN world while still being attached to the old profile.

Switch to the hardened profile. But don't do any emerge --update yet.

Re-emerge GCC as

Code:

emerge --ask --update --oneshot --newuse --deep gcc

This will use the new USE-flag for the new GCC, if there are any. Look at the outpout of emerge whether any USE-flags did actually change. If there are none, and CFLAGS in /etc/make.conf also didn't change, there is no point in re-emerging GCC, and you can skip this step.

Run my script. It will recompile anything else, according to the hardened profile.

Update: I have also updated by guide (as of version 1.11) to support your scenario directly.

Last edited by Guenther Brunthaler on Sat Sep 09, 2006 7:14 pm; edited 1 time in total

why don't try to contact a developer to talk about this? Maybe the script can become an official part of Gentoo in the future.

While this may be an option for the future, I want to see much more testing before I will actually consider doing this.

While the script itself has not changed since this thread has been started, the accompanying guide serveral times has.

As more and more users were testing my guide, a couple of them have also suggested various improvements, which in turn led to several revisions of the guide to take those improvements into account.

I want this to get on for some time before I eventually might dare to contact any Gentoo developers.

juantxorena wrote:

Have you contact any developer? What he/they say about this?

I am not related to the Gentoo project in any way except for my commitment to the project. I'm just a normal user like most here in the forums.

juantxorena wrote:

Messing with the system like this is a dangerous thing,

Yes it is. That's also why I consider more testing to be essential.

Although it seems the guide actually worked fine with the people posting here, that says nothing about how well it will work in unusual cases of system configurations.

juantxorena wrote:

I don't want to break anything.

I can understand this.

But there is actually little reason to be afraid of: The script is actually a wrapper around a set of emerge statements, each one emerging another package. The script will not try to interfere with the internals of Portage and won't in fact do anything you were not also able to do manually.

The worst thing that could happen to your system is that the script stops before being finished due to a build error of some package.

If that happens, simply skip that package and let my script continue. Then, after all else has been rebuild, re-emerge the failing package again, hoping the emerge will then work better.

And if it turns out that my script can't continue, because that failing script is an essential dependency for other important packages, you can still revert to the usual

which some people consider to be a better solution. (I doubt it will, but I could be wrong. The only difference one will see for sure, is when looking at the next electricity bill.)

And if that standard method considered "safe" by most will also fail, you still have the last-resort option of doing a complete stage1 system reinstall, just as badpenguin suggests in his guide.

So - no matter whether my guide actually works for you or not, it will never damage your system to a degree that would not allow you to revert to any of the other guides around.

Such a case has not been reported so far, though.

juantxorena wrote:

the way you explain things, I think that you are acting like a televangelist or an infomercial vendor

Since when do televangelists or infomercial vendors explain anything?

They usually want you to believe something, which is definitively not my intention. If it were, I would only post my guide, but not any accompanying guides where I explain the ideas and concepts which eventually led to the creation of my guide.

So, if you felt like listening to an televangelists when reading my guide, I have to apologize: This has not been my intention in any way.

Also, as I am not a native English speaker, I will certainly not always be able to express my thoughts in the same natural way a native speaker would be able to do it in the same situation.

In other words: Please excuse my rather bad English!

juantxorena wrote:

This undermines you credibility.

To be honest: Being still a Linux newbie, I do not have any credibility yet. So there is hardly anything to be undermined.

So I'm doing the best I can to read man pages, HOWTOs, source files, etc and figure out how to do things in the most optimal way.

And when I find a solution to a problem that finally pleases me, but is different from the solutions I have found being suggested by others so far, I will share it with the community by posting an article.

Of course this will be done in the hope it may also be useful for others.

But the other reason is that is allows others to comment on and suggest improvements, which has shown to be a quite effective way for improving the quality of any guide.

juantxorena wrote:

It is in fact UNNECESSARY to recompile any package bla bla bla" or other similar quotes is not the best idea.

Well, perhaps you are right. It may not have been the most polite way to put that statement.

But it truly reflects my personal opinion about the issue, so it's at least not a lie.

Also, except for unusual scenarios where the usual "emerge -e system && emerge -e world" would fail as well, there have not been any serious problems reported with the most current version of my guide so far. (Which does not mean there were no problems reported with older versions of the guide. But I updated the guide in such cases in order to fix it.)

But up to date, my guide is still the fastest way to rebuild an entire system (assuming users have been updating their systems on a regular basis even before considering using my script; my guide will exploit this).

There are known situations in which it will not work, such as updating glibc from version 2.x to 3.x. But I won't expect such a major glibc update to happen within the current decade.

Also, in such a situation, all the other guides based on "emerge -e world" will also fail. You have to do a complete stage1 reinstall in such cases.

But otherwise, the guide seems just to work.

juantxorena wrote:

Personally I would wait to see what devs think about this.

I assume the Gentoo devs have more serious things on their do-to-lists than caring about such minor issues.

After all, my script is a mere optimization. It is nothing new or revolutionary on its own, it will only save you some bucks on your next electricity bill. That's all.