If you like send me an email with the source attached and I'll post it on my webspace. That should help for now. I'll post a link to it here after I do. One thing that I have noticed about the forums is that they strip extra whitespace out of posted comments. I do not think that it does in posted code segments, but there could be bugs messing with it.

I am also interested in incorporating this functionality into porthole. I've used it a few times and I like how it separates the different areas of programs. It should be relatively straightforward to do in porthole. How good is your python skills?_________________Brian
Porthole, the Portage GUI frontend irc@freenode: #gentoo-guis, #porthole, Blog
layman, gentoolkit, CoreBuilder, esearch...

OK, I can send you the source in an attachment. Can you PM me your e-mail address? Thanks!

Quote:

How good is your python skills?

Unfortunately, nonexistent!

I'm pretty well versed in Bash and Perl, but I've never taken the time to learn Python. Mostly, it is because I don't have any need for it---other than emerge/portage. In fact, the engineer in me says that it is wasteful spending time on Bash, since Perl can do everything Bash can do. But, I cannot get away from the fact that knowing Bash is extremely helpful for little one-liner command-line scripts---and I need that since Perl is not a shell! I really like Perl, and I really seem to prefer it to what I have seen/read about Python (probably because I hate OO stuff). I know Python is probably superior for OO programming, but Perl is more than capable for everything I need in a scripting language. If only the emerge/portage stuff were in Perl... _________________BillyBear: I'm scared Poncho.
Poncho: Bullsh!t. You ain't afraid of no man!
BillyBear: There's something out there waiting for us, and it ain't no man. We're all gonna die.
<ominous music>
<enter Hillary Clinton>

maguire : doing emerge -w creates a world file and starts the emerges but in an alphabetical order instead of the tree structure like hielvc was having. I think it could be important to respect the tree structure for some dependancies.

Hiel's script is quite clever and takes care of a number of details like that in a pretty concise way. I think maguire you should be a little careful with the harsh words. I don't find emwrap.sh hard to understand at all, just took some minor logic skills. It's always easier to modify or even re-invent an idea than to make it work in the first place. Maguire, I'm not saying your input isn't welcome, (and it's not for me to say), Hiel has graciously accepted it for that matter. Personally though I don't care if bash hacks win awards for coding etiquette. They're just bash hacks after all. Maguire, maybe you want to put Hiel's name a little higher up in the comments anyway. I suppose if someone really wants to do things right they'll dive into all that python stuff in portage... yuck.

hielvc:
Thank you for posting the link to emwrap on the first page! That was nice!
I figure that only a few people will actually look at emwrap, but I thought it might help some people follow the Bash code better... Thanks again!

__________________________________________________________________

papaours:

Quote:

maguire : doing emerge -w creates a world file and starts the emerges but in an alphabetical order instead of the tree structure like hielvc was having. I think it could be important to respect the tree structure for some dependancies.

papaours, you are confused. I'll explain below.

dagurasu:

Quote:

Hiel's script is quite clever and takes care of a number of details like that in a pretty concise way.

dagurasu, you are a moron. I'll explain below.

Neither one of you are talking about emwrap.sh. You are referring to sys-build.sh and the acompanying text file basesys-build.lst. sysbuild.sh is a different script entirely, in that it is made for a single purpose: to (re)build the system from scratch, using a prescribed order of emerges! If you look at the code for the original emwrap.sh, you will see that there is no reference to eithersys-build.shorbasesys-build.lst, and that there is nothing that controls the order of packages that are emerged.

I only said that emwrap was a source-code cleaned-up version of emwrap.sh, with some---in my opinion---small improvements in functionality and large improvements in readability. In fact, looking back, my exact words were:

I did not say anything that emwrap included any of the sys-build.sh + basesys-build.lst functionality.

As far as
dagurasu:

Quote:

I think maguire you should be a little careful with the harsh words. I don't find emwrap.sh hard to understand at all, just took some minor logic skills. It's always easier to modify or even re-invent an idea than to make it work in the first place. Maguire, I'm not saying your input isn't welcome, (and it's not for me to say), Hiel has graciously accepted it for that matter. Personally though I don't care if bash hacks win awards for coding etiquette. They're just bash hacks after all. Maguire, maybe you want to put Hiel's name a little higher up in the comments anyway.

What I said was that the formatting was horrible, and it is. I never said that emwrap.sh was broken in any way. Obviously, it was tested by quite a few people. Everyone's coding style is a bit different, but some things are agreed upon by the vast majority of programmers. One would be to not mix tabs and spaces in the indentation of the code. I won't waste my time posting examples---you're clearly no coder. As far as your comment about it always being easier to modify or re-invent an idea, I agree! But did you happen to notice that emwrap.shitself was an improvement on tcupdate? Did you tell hielvc that "it's always easier to modify or even re-invent an idea than to make it work in the first place"? You may not care about how "Bash hacks" work, but if the "hack" is going to be (re)installing a bunch of software---and ciritical system tools---on my system, then I want to know how the "hack" works. You've also clearly never coded in Bash. Have you coded at all? I'd like to hear why you think all Bash scripts are "hacks", as opposed to serious programs. Are all Perl scripts "hacks" too? How about "Python" scripts? Is all portage + emerge a "hack"?

You clearly think that I somehow slighted hielvc. (No slight was ever intended!) Maybe you should let hielvc take care of himself? Stop flaming people who actually add something to this thread.

maguire said
"there is nothing that controls the order of packages that are emerged. "

No need for name calling maguire. You are wrong. emwrap.sh gets it's ordering via it's use of emerge (at least last time I looked at it, which I'll admit wasn't in the last few weeks). Emerge knows how to order builds(at least more or less)... and thus so does emwrap.sh. But if that's true or not, all I did was suggest you might want to show a little respect. You seem much better at calling people names. "I'm clearly no coder"... hehehe.... you're funny, you seem to think you know alot about me. I am alot of things including an appreciator of things that work and the people who come up with them. Anyway, relax man, I never said your input wasn't welcome....

Edit:
The funny thing is you use unecesarily harsh words to criticize someone else's code that works just fine and when people correctly point out problems with yours the first thing you do is call them a moron? Even if they are wrong... is that really helpful?

Last edited by dagurasu on Fri Aug 19, 2005 3:46 pm; edited 1 time in total

...in an alphabetical order instead of the tree structure like hielvc was having.

and
dagurasu:

Quote:

Hiel's script is quite clever and takes care of a number of details like that in a pretty concise way.

I had no idea either of you was talking about emerge's ordering of the packages! It sounded like you were both talking about some special ordering done by hielvc's emwrap.sh!

Now, since the most recent post pointed-out that you weren't talking about any emwrap.sh ordering, but rather the ordering of emerge itself, I see what papaours was talking about!

Everything that I've read about emerge on these Tool-Chain-discussion threads says that emerge's ordering appears to be random. I completely forgot about the case where emerge installs needed dependencies for a package. That, obviously, is important!

OK. I think I fixed the problem with the out-of-order emerges. I just e-mailed the file to dol-sen (who is graciously hosting the file on his site ) So give him a while to get the file updated, then try the new version (version 4.1).

In the newer version of emwrap, the order of packages from emerge is kept exactly the same for emwrap -puDsw ('emerge -puD world') and emwrap -puDs ('emerge -puD system'). In the case of updating the files that are in only the WORLD class (not in the SYSTEM class), emwrap -puDw, the order is that same as that of 'emerge -puD world', just with all the files in 'emerge -puD system' removed.

Let me know if I introduced any new problems (or if one remains from the prior version).

...Everything that I've read about emerge on these Tool-Chain-discussion threads says that emerge's ordering appears to be random. I completely forgot about the case where emerge installs needed dependencies for a package. That, obviously, is important!...

I'd like to see if something i've been lazily thinking about for a while gets anybody's juices flowing on this matter...

Firstly, as will soon become painfully apparent, i AM no coder! It doesn't follow, however, that i can't have an enquiring & analytical mind. If it turns out that i'm totally wrong, i'll welcome correction - "moron" might put my nose out of joint, though...

The ordering of packages by 'emerge' does have a defined pattern, & 'the alphabet' isn't a million miles from being a fair description of it...

I originally started wondering if there might be a way to get hielvc's 'sysbuild.sh' to create it's build-list dynamically, as the static one supplied would need periodic updating, & would also benefit from some 'tailoring' to each specific system.

The obvious tool to use for this dynamic list-building is 'emerge' - but if that's the case, then why doesn't it do so already..?

Now, i only extended this investigation as far as the 'system' keyword, originally, with particular emphasis on the '--emptytree' option. Here's what appears to happen if you issue the command :-

Code:

emerge system --emptytree

Portage constructs a list of packages by going through the various leaves of the installation's target-profile - it starts with /usr/portage/profiles/base/packages, & works it's way back up the heirachy adding any further files it finds onto the end of the list, & catering for masking, unmasking, & virtuals files, as well. This behaviour is further illustrated if you customize the 'system-profile' by creating your own additions in /etc/portage/profile/packages.

And it's those package-files that are kept in a more-or-less alphabetical order.

Once this list is complete, portage then goes to it's first package & starts checking dependencies, & this is where the '--emptytree' option, if not others, seems to me to have a bit of a blind-spot...

It seems to check each package's dependencies against the existing build-system, regardless of the fact that everything is going to be either re-emerged or upgraded, anyway! So if package-A's dependence on package-Q is catered for by the build-system... Fuck it, that'll do.

Anyone interested can check this themselves by issuing the command :-

Code:

emerge system -ept

...& comparing the output to the various 'packages' files in their profile-path, & to any specific package's ebuild for it's dependencies.

EDIT: It gets kinda chaotic if you have the java USE-flag enabled globally, & don't block it's use by 'sys-libs/db'.

Without having any python skills, i'm currently ill-equipped to check this more professionally. Plus, it's also occurred to me that the portage-dev's may have already considered this, but, because of circular-dependencies &/or infinite-loops, have plumped for 'the lesser of two evils'.

I'd be intrigued to hear what others think of this theory, & whether or not anything can be done to improve the situation. Also, although most relevant to freshly-built systems, i'd have thought that a correction of this behaviour ought to have positive repercussions for any of the toolchain-insulating script variants, too... _________________"Anyone who goes to see a psychiatrist should have their head examined!"

Im still haveing major x windows problem and work has been busy. Anyway emerge does a pretty good job of the proper build order. One of the reasons why Ive let sysbuild lye is that portage does such a good job and keeps getting better. Im useing links so I cant put yourallss names in but its one bigger prob, my thoughts, is is the "binutils gcc linuxheaders glibc" build order. I would reccomend useing emerges order of building. Thats why I didnt use "sort". This could cause major probs particularly in "emptytree" rebuilds whare you want the correct build order so that the libaries are built in the correct order._________________An A-Z Index of the Linux BASH command line

...When I try emwrap.sh -uDwp , it only shows system packages, but not the TC packages...

I haven't tried maguire's script, but hielvc's version would need either the '-t' or '-b' option in order to include the toolchain components :-

Code:

emwrap.sh -Dutps

...would be "a pretend deep update of ONLY the toolchain-components from the system-file", or :-

Code:

emwrap.sh -Dubps

...would be "a pretend deep update of BOTH the toolchain-components & all other packages from the system-file".

Changing the '-s' option (system) to '-w' (world) shouldn't matter if you just want to update the toolchain-components :-

Code:

emwrap.sh -Dutpw

...but it's been advised previously (like in hiel's first post) to rebuild everything after a major toolchain-overhaul.

EDIT: Maybe maguire's version got thrown by your inclusion of both the 'system' & 'world' options at the same time... _________________"Anyone who goes to see a psychiatrist should have their head examined!"

Last edited by taipan67 on Fri Aug 19, 2005 4:21 pm; edited 1 time in total

I would not be at all surprised if what you say is true. I have found other similar (but sort of reverse) problems with portage. You'll find I have a couple of posts/respones laying around about an oposite problem where checks for satisfying dependancies are made only against the portage tree and not against the system. this means that things that are no longer in portage get upgraded unecesarily. To be fair maybe it's thought that things are removed from portage for a reason, ie so new packages don't have to keep up with if they still work or not, but then why does emerge -u complain but without the -u is perfectly happy with installed dependancies. Anyway, it's off-topic, but the point is, I don't think anyone will try to argue that emerge does things quite perfectly. It is though, alot more likely to work than a pure alphabetical aproach though, and in fact seems to usually work. I can say that emerge -u finds dependancies for new packages that show up, but I'm guessing this is diferent from the issue you describe with previously installed packages using --emptytree. Anyway, reading the protage todo list, it doesn't seem like these types of things are dead issues, so I guess, as hiel says, it is getting better.

@sam_i_am
as for emwrap.sh, as stated it's not supposed to actually do tc updates unless you ask it to, however.... it can be made to still show them by reversing the comments on these two lines. Personally though, I don't like doing that, cause -p should show what will actually happen. I guess there's a reason it's commented out.

Building a list of all EMERGE WORLD packages...
Building a list of all EMERGE SYSTEM packages...

No primary Tool Chain updates are available.

What gives? I thought glibc and linux-headers are TC components. Similar thing happens with emwrap.sh as well. When I try emwrap.sh -uDwp , it only shows system packages, but not the TC packages.

Could you please grab the 4.1 version of emwrap (if that's not the one you used), and could you please post the results of this emerge command:

Code:

$ emerge -pvuD system

and this emwrap command:

Code:

$ emwrap -pvuD system tool

(the "v" is ignored), and, finally, could you post the contents of all the files in your /tmp/emwrap/ directory:

Code:

$ head -n 1000 /tmp/emwrap/*

The emwrap command that I put above should ask for a (pretend) update of the SYSTEM and TOOL (aka Tool Chain or TC) packages (or the equivalent short version: emwrap -pvuDst). Hopefully this will make the problem more clear.

The recommended usage for emwrap.sh (when rebuilding the toolchain) is:

Code:

gcc-config -l # make sure of gcc before beginning
emwrap.sh -set
emwrap.sh -seb # We rebuild the TC and then the rest of the system
emwrap.sh -se # doing the 2nd system emerge minus the TC
emwrap.sh -wet # rebuilds TC against clean system
emwrap.sh -r # first build of the world files against clean TC system
emwrap.sh -r # second world rebuild and done

What's the recommended usage for emwrap to rebuild the whole toolchain? (edit: I guess this question is essentially the same as Master One's)

Also, I have a feature request: When "-p" is included, can you have the emerge output include "-v" so that we can see what use flags are set?

I just downloaded maquire's latest emwrap v4.1, but now I am not 100% sure how to proceed.

If I now just perform a single
Code:
emwrap -etsw
is that already enough, to have the whole job done?

Or do I need to perform
Code:
emwrap -etsw && emrwap -esw
to have the whole system being rebuilt properly?

OK... Proceed with caution...

First of all, that's a good question. And like many good questions, it does not have a simple answer. I am not a developer, or an expert concerning system builds, but I can relate to you what I've read (and it makes logical sense). Please see the Jackass! installation documentation and the 1/3 NPTL 2005.0 installation documentation for a better explanation of all the following (where I got most of it).

First, it is quite critical to build the system tools (the Tool Chain [TC] components) in the specified order, and repeat the builds of some of these packages. That should be taken-care-of by emwrap (although I have not yet tested that functionality!). So, for instance, if gcc is among the packages that will be built, then several of the TC components will be built, with gcc being built twice---with gcc-config being run after the first build, in order to select the new compiler version for all subsequent builds. So, the TC portion of the problem should be handled automatically.

Now, in the event of a full system recompilation (in order to full take advantage of the new Tool Chain), the new self-consistent Tool Chain will be used to build all the packages in world (or just system, if that is what you specified). A problem may then apparently occur where package A has some dependency on package B, but package A gets rebuilt first---against an old package B (at least that's my understanding of the possible issue). Therefore, to be really sure that all packages get built against the latest-TC-built dependencies, it is safer to rebuild everything a second time.

So, the long and the sort of it is that the Tool Chain will be solid (self consistent), but to be safe, if you are going to rebuild your whole system, you should probably build the system and world packages again, like you said:

Code:

$ emwrap -e tools system world && emwrap -e system world

(Note that I'm subtly bragging a bit about the flexibility of the command-line parsing! )
Since the Tool Chain build seems to take so long, you could also break that into three chunks, using:

Code:

$ emwrap -e toolchain

one night, and:

Code:

$ emwrap -e system world

the next, and:

Code:

$ emwrap -e system world

the next. And given the small number of problems that require the second world build, I would think that the final system+world build could be done more slowly in the background (heavily 'nice'd?).

I would recommend that you do just the Tool Chain rebuild first, like I showed above, to make sure that it all goes OK. I have not yet had the opportunity to test the Tool Chain builds using emwrap! I think it will work fine, but I'm waiting for someone to tell me! (You can look-over the TC-update part of the code, too, if you want, it's pretty well documented with long, clear variable and function names.)

The recommended usage for emwrap.sh (when rebuilding the toolchain) is:

Code:

gcc-config -l # make sure of gcc before beginning
emwrap.sh -set
emwrap.sh -seb # We rebuild the TC and then the rest of the system
emwrap.sh -se # doing the 2nd system emerge minus the TC
emwrap.sh -wet # rebuilds TC against clean system
emwrap.sh -r # first build of the world files against clean TC system
emwrap.sh -r # second world rebuild and done

What's the recommended usage for emwrap to rebuild the whole toolchain? (edit: I guess this question is essentially the same as Master One's)

Also, I have a feature request: When "-p" is included, can you have the emerge output include "-v" so that we can see what use flags are set?

Well, first of all, since emwrap, like emwrap.sh, builds the Tool Chain packages such that binutils, gcc, and possibly glibc are already built twice, I don't understand why another TC build is necessary in the second call to emwrap.sh in your code. I gleaned my understanding from the Stage 1/3 NPTL 2005.0 documentation, which really only explains why the second compilation of the TC components is necessary. Why is a third and fourth compilation of the TC packages needed?

Second, in your code above, the fourth call to emwrap.sh rebuilds the TC a fifth and sixth time--with the comment "rebuilds the TC against a clean system". Why would this be needed? What part of the system is the TC dependent upon? It seems to me that these are the basic building blocks, and should not depend on anything but themselves, right? I mean, once you've rebuiltgcc using the newer version of gcc, why would gcc need to be rebuilt again after all your world packages are installed?

If someone can show me where I missed the boat here on TC builds, I can fix any problems in the emwrap code.

Lastly, as far as compiling, recompiling, and recompiling all the system packages (emwrap.sh calls number 2, 3, and 4 above) and then the compile, recompile, and recompile of the world packages (emwrap.sh calls number 4, 5, and 6 above), I kind of covered that in my prior post. Compiling the system twice and then compiling the world-system twice is probably the safest way to be absolutely sure that everything is compiled against the latest version of everything else.

So, unless and until I'm corrected on my TC thinking, I think that my equivalent---but not overkill---recommendation for the ultra-safe rebuild-of-everything would be:

(and by the way that emwrap---not emwrap.sh---defines "world", it is really the emerge world packages with all the emerge system packages removed, and the TC build in emwrapshould do the gcc-config step automatically [at least that's what I wanted it to do].)

So the above incantations of emwrap would build:
1) linux-headers, glibc, binutils-config, gcc-config, portage, binutils, gcc, glibc, binutils, and gcc
2) the emerge system packages not including the Tool Chain packages above
3) the emerge system packages not including the Tool Chain packages above
4) the emerge world packages not including emerge system and not including the Tool Chain packages above
5) the emerge world packages not including emerge system and not including the Tool Chain packages above

Also, I have a feature request: When "-p" is included, can you have the emerge output include "-v" so that we can see what use flags are set?

Sorry, the last post was so long, that I forgot to address your other item!

Yes! I've already added it to the update that I'm working on (I'm combining redundancies in the code for building TC elements vs. everything else). That was just an oversight on my part. I'm going to always have the "verbose" emerge option "on". Why not?