Ah I see what you mean now Geki. I've been working on update today, and just pushed some fixes to stuff that was stopping me upgrading. (I could have got round them with emerge, but that would have meant I lost the test-cases to fix the bugs with.)

I'm glad I took the time to go slowly with the harfbuzz thing: it turned out I needed to change the default cacheDir setting, since portage has been md5-cache only for quite a while. That's what was triggering it looking for the SLOT in the ebuild, instead of the cache, which is where it was picking up the comment from (that's sorted now as well, ofc.) So that should make things a bit quicker for all of us.

I think that just got missed, as I remember spending ages waiting for md5-cache to be the default. Also, there was a problem with blockers, which are slotted, which I hit with php. The ebuild was gone from tree, so I spent a while thinking it was another issue with getSlot. The fixes were pretty small in the end, as usual, but that area has always been hard to debug; so I made it a bit easier for next time.

Anyhow, I've just hit it not allowing me to proceed with toolchain upgrade because I need to upgrade my kernel before I can do kernel-headers. Picking up on --exclude is an option, although i feel loathe to add yet more code, when my inclination is to keep the rigour of the check, as it only kicks in if linux-headers is in the list, and upgrading to it would be unsafe. Other than that, if you want to do something past the envelope, you should be doing it directly with emerge. I'll have a think about it, as I get my machine updated.

Yeah that will work, in terms of it not coming up in the list, though I'm not sure it's something you want on a permanent basis.

There is always the SKIP envvar for temporary situations, or ofc the noCompile array in /etc/update if you never want update to handle a package, unless you tell it to explicitly on the command-line (see: update -h config.) So if you just want to avoid linux-headers for one run, you'd use SKIP=sys-kernel/linux-headers update or just export it in the terminal session you're running update from. It sounds more like you want it in noCompile though, and that should definitely work or it's a bug that needs fixing immediately.

Code:

** Skipping sys-kernel/linux-headers-3.9 - in $SKIP

SKIP's been there since the beginning (as has noCompile), and it could no doubt use some tweaking, so if you find a way it could be better, tell me and I'll try to implement it. It was only ever meant as an emergency measure, so it requires the full category: looking at it, it would be more useful if it could skip on package name as well. Back when we added it, we actually had CATX iirc for category exclusions, though we don't any more, and the idea was to do more on skipping, but then we added the dialog (E) to skip stuff, and left SKIP as a "left-string" match; eg:

Code:

export SKIP='dev-perl/ sys-devel/gcc-4.7. dev-lang/python-2'

BTW if you want to sort things out a bit more quickly at whatever point, login to IRC: chat.freenode.net and /join #friendly-coders; my nick is igli. IME it usually helps to chat real-time about what needs changing, do it there and then, and then rework in collaboration with someone who actually needs the use-case; they tend to know exactly what needs to happen, and what is or isn't needed.

Sometimes it's hard to get that right, when you're just making your best effort to guess what would be useful, and experience leads you to be cautious; you can end up implementing a whole chunk of code that no-one ever uses, which is a waste of CPU time as well as your own. ;)

I'm online most of the time; if not, Griz64 is the QA meister for update, though he's really our infra guru, so he knows it inside out and is always on the latest git version (at least half the bugfixes start with him poking me on IRC) and there's a quite a few Gentoo users in there (the people who started it are all #gentoo-chat regulars.)

For ref, if you need to take something out of the command-line target, edit /root/emerge/target (/root is default dir setting); I just did that to get the SKIP output, though I started with update linux-headers glibc (since SKIP is a glob match, targets specified on the cli, are not considered. I guess we could tweak that to always skip exact package name matches.)

Well, I've finally finished the first cut of --toolchain; it's currently in git (along with a few other fixes.) Oh I also enabled update --perl in line with --python and --haskell, since Griz was on-hand to QA it.

My last --deep world, according to /var/log/emerge.log was on January 06, 2013, (ie 8 months ago!) and I've finally set it off again, after upgrading my kernel, since linux-headers were blocked as above (by design), then the following from update --toolchain

I cancelled because it wasn't uninstalling alsa-headers before trying to emerge linux-headers.
This turned out to be an oversight in toolchain handling, so it was important to fix it. And to be sure it worked across resume, since when we have a toolchain target (that is update --toolchain with no named target, much like we default to world/@system @world in normal usage) there is no package data at all in the usual list files. Again this fixed an issue with previous code, since the custom -e handling uses the same setup.

If you use update -r --toolchain then it will only do the toolchain packages in the saved list. Don't forget you can use a numeric flag ie -[0-9] in wildcard terms, for a temporary output run, with its own set of list files etc, exactly as under your main /root/emerge directory (or whatever you set dir to in the config).

Similarly, if you give it a target with --toolchain, it will resolve that target (or targets or sets) and then do only the toolchain packages in that list. You can see above that the default target, assesses the state of all toolchain packages against the compiler. linux-headers upgrades always pull in glibc (as you've probably seen, since that's been the case for a few years.) In the above case, many things which are important to compilation had not been built for ages, let alone with the current compiler (that's what the dates are about. Yes this may be superfluous for *-config, but it's easier not to do a check than bolt-it in afterward.)

compiler deps (gmp, mpfr, mpc by default for gcc) must also have been in place when the compiler was last built, as well as checking for ABI changes (minor upgrades are simply built without triggering a rebuild of gcc.) Other packages are tested against its first-built time, and the sequence is as we define it in our configuration (cf: update -h config.)

libc deps (default to binutils as seen above) changing ABI trigger a further rebuild of glibc. This is the modern equivalent of the old "build everything twice" metaphor applied only to gcc and glibc. You can set libc_deps to an empty value in /etc/update but note that a compiler rebuild as above will still trigger a glibc rebuild.

Sorry, but if you want to play it unsafe, use emerge at the command-line to do it. KK, press E and skip the last glibc (or anything else;)

Sit back and wait and my toolchain is all nicely consistent. At this point info ld stops working so I take the hint, and reboot.

Next I found a bug in revdep library detection (not handling quotes properly due to a typo, but let's take the opportunity to deal with spaces and pick up revdep libs more consistently), and this led to the circular revdep issue; I tracked that down to revdep-rebuild wanting to rebuild libraries with preserved-rebuild libs in them; these are held-over from the last installation, but counted as part of the current (or next) version of the package: so ofc rebuilding them doesn't work since portage continues to preserve the lib, and revdep continues to think it's a dud.

This only applies when you follow the emerge warning to use revdep-rebuild --library '/usr/lib64/libfoo.so.1' && rm .. for example; I think the manual message is to avoid a preserved-rebuild loop. Luckily the 3 packages with preserved libs were precisely the gcc deps, so when the other two came up as the only consumers of libgmp.so.3 I knew something was up (since they were upgraded after it was rebuilt with new(ish;) gcc.) I've raised it in #gentoo-portage and hopefully it'll be fixed soon (revdep should not treat a preserved lib as being a current object.) The python rewrite will be default from next version, so after that's bedded down hopefully this bug will get sorted.

edit: this isn't standard preserved-rebuild message.

Last edited by steveL on Sat Sep 14, 2013 5:46 pm; edited 1 time in total

So what came next, once I was sure there was no weird linkage going on with toolchain things? (And I'd got some sleep;-) The usual tree-chasing basically, just a few things compressed into one upgrade, including an improvement in USE flag editing.

Standard update -s came up with a couple of blockers. First off, ncurses got a new tinfo flag, and openrc needed to be compiled with it off (I think; I lost the terminal output, so I'm going off emerge.log and eix output for what's there now.) update ncurses took care of that.

texlive-langgerman-2012 was blocking the upgrade to texlive-2013, despite the new version being in the list, so for the time being I simply used emerge -Cq texlive-langgerman (remember: update doesn't have an uninstall option: we don't shield you from emerge, since if you don't know how to use it properly you will not be able to troubleshoot anything at all.) Note that -cq is the normal uninstall switch, which is dependency aware. However since it's dependency-aware, it won't remove a package that's in-use (ie depended on by another package, or in world/system.)

Additionally sysvinit was holding back openrc and udev (iirc) and util-linux needed upgrading for something else, so I simply hit update sysvinit util-linux to get past those. (update does not install to world unless you tell it to with -i.)

Then I tried update -aU system but only 2 packages showed up (note i'd forgotten -u ;) and it was worse with --bdep, so I just used update system and let it skip useless rebuilds, using the old logic. That got the list down from 40 to 16 packages and meant I could proceed knowing my base system was up to date. (Stuff like bash; again a reboot was in order.)

The next bit was me playing with USE-flags, basically, and seeing what new stuff was coming in. I had a slight issue with qtsql, since the KDE desktop profile now sets the mysql flag on for that package. This was a suspect area, since we had logic to detect "default" in UI (dialog) as what the package is currently coming in with, ie the default if we don't change anything, but the logic for deciding what flags to set is separate, and uses IUSE from the ebuild. I'd wrestled with it before, but came to a hackish compromise that allowed me to at least switch off a profile-set flag. Even if I do force this one off globally at some point. Basically we now always accept an off setting from the user if the package is currently going to have the flag on, according to emerge. Previously (as here) the global being off, would be enough for it to assume the setting would be off.

That needs a bit of refinement, as ever, but it is an iterative improvement on what was there before. (or I would have had to set the flag manually, by editing package.use.) I also fixed up error handling under --jobs: if you skip an ebuild then that can cause emerge to error out. In my case, I have firefox in noCompile, so it only gets updated when I use update firefox, not on every upgrade that comes in (since it takes forever and uses over 15G of disk space.) But the installed firefox had a dependency on sqlite with fts3 USE-flag, which sqlite-3.8.0 does not have. So I was getting an error, only I couldn't see it, apart from "0 Jobs of compiled" and then a die message that no packages were compiled. That's fixed; (I actually only noticed it as I was trying to update udev without lvm2, which didn't work either. So that was my testcase; it was only when I got the error output that I saw the firefox/sqlite issue.)

:-)
So my advice is: get the machine to a point where emerge can do update -uU system usually by doing --toolchain first (if it's been a while: that's what it's intended for, as well as ensuring everything is built with the current compiler, headers, libc etc.)
Then see what's stopping the world update and preferably upgrade pieces that are causing conflicts; in the worst case remove a dependency that isn't important (and if you don't know, ask in #gentoo.) You may have to skip an additional package or two: for instance I ended up simply skipping sqlite in addition to firefox, and letting everything else go.

Note skipping does not (usually) mean use of SKIP; it just means deselecting the package from the dialog list. Hit 'e' (or 'E') to see that when it asks if you want to proceed with the build.

And make sure you run dispatch-conf before you reboot, as well as revdep-rebuild. (update makes sure you do that, of course, though you can't always get through revdep-rebuilds, so try to find out what's happening under the hood: man revdep-rebuild. In my case I confirmed the issue with ldd /usr/lib64/libmpfr.so.1 for example, which told me it was the preserved lib that had the linkage to the old gmp lib.)

There are of course things to improve with toolchain handing, especially slotted packages like python, which applies to all types of update. Now we have --perl in place, that's also in toolchain, and they trigger the relevant cleaner/update when they change ABI. ghc is rebuilt when gcc changes ABI. Again, that needs to be added so --haskell will be picked up even when gcc isn't being rebuilt.

Just so you don't think I'm neglecting this (or an ebuild) I am very close to finishing off --toolchain. The recent perl difficulties, led me to wanting this dependency conflict handled automatically, which entailed handling all of them (3 types that I know about so far) generically, including when ABI warnings happen for packages that aren't in the resultant list. This shows the problem:

Note that it's processing the conflicts and flagging perl as a toolchain ABI warning (I added 5.16 and 5.18 to local version). But it's not running the fix yet as perl-5.16 is not in the list. So the perl deps aren't being added either, as emerge is saying it's just going to rebuild 5.12. You might also notice the dates on gcc: it's rebuilding its deps, since they were built just before the last build of gcc, and not with a build of gcc linked with those versions. That was built just after, so this should be the last build of them (for this sequence.) "Confused? You should be." ;)

To actually fix it I need to add support for: clean(virtual/perl-Test-Simple) and perhaps re-resolve; turns out we already had something slightly more complex, for unmerges that happen before a package is installed (which needs to be handled for warning packages as well, ofc.) I didn't write much of this part, but most of it is really crufty, and hasn't been touched for several years. Anyhow I've rewritten some of the old warning code to be more efficient structurally (which I now need to change the rest to using.) It should be more useful for upgrades of old machines as well, once I do, since it will do them in the order in the file, ie chronologically as tree changed.

So not gone, not forgotten :) Error output should use more colour after I committed the setup of them; we were using derived code in shell, so imported it back, and found everything was a tangle of spaghetti so had to clean that up first. The actual ids I've been using for a while, so you should see a few places brighten up (unfortunately mostly errors ofc..:) Once I'm happy with toolchain, I'll switch to working on the installer again.

OFC I'm far behind the tree. I always stop for big fixes like this, that hit my install if I can. It means I'm forced to work on getting the thing working correctly. It also reassures me that update copes with several months between world upgrades, as that's what I usually end up doing.

Ah thanks mate; that's exactly what motivates me to improve it (flattery;) Well it's actually that people find it useful, as that's the biggest compliment in my book.

Quote:

I used my own little script for years but wanted something more polished and complete. This fits the bill nicely.

Many thanks +SteveL

You're very welcome :-)

Quote:

As a stop gap for overlay. Could you possibly make the ebuilds available via git (in app-portage directory)?

Yeah was going to put the ebuild in the git repo, though I've changed it as well as the makefile for prefix support, so wanted to test it a bit first in case I've broken something.

I can easily make an overlay available; it just seemed odd to do it for literally one package, and even then I'm only really happy supporting the git version, unless someone else is happy to deal with the QA process; bug wrangling principally. Decision-making as to what's required right now, and veto over any changes, belongs to Griz64. If we had an overlay and were putting out versions that people reported on, decision-making could move to votes I guess, but not veto. Somehow I doubt enough people care to make that worth doing.

If we were to put it in an overlay, I'd rather use sys-apps, btw, as it was designed to support all 3 package manglers, from the beginning. But with a git ebuild, that's not necessary (which is why I like it;)

Again, thanks for the positive feedback: it's given me the buzz I need to tackle the last bit of toolchain (putting back a function I deleted a while ago so backup of packages we're about to remove, works again.)

I found the script discussed in this thread while trying to find a semi-automated way of keeping my Gentoo/MythTV system updated/upgraded. Could I ask that the author of this script take a look at the thread I created at https://forums.gentoo.org/viewtopic-t-1021528.html?sid=6b54a4ec784b5aa3fe0d4ae52a8571bd and offer input as to whether this script could serve my purposes? Thanks in advance for any input on this: I hope this might provide the answer I'm seeking.

Sorry I didn't reply sooner; this didn't come up in "My Posts", for some reason.

jamtat wrote:

I found the script discussed in this thread while trying to find a semi-automated way of keeping my Gentoo/MythTV system updated/upgraded. Could I ask that the author of this script take a look at the thread I created and offer input as to whether this script could serve my purposes? Thanks in advance for any input on this: I hope this might provide the answer I'm seeking.

Yes it would; it's designed to be used automated, under the control of another script or program:

The -x flag means "automated", wherein it bails out on many things it would ask a human about; also happens if it's being piped, and there's no terminal in sight.
cf: POSIX test: [ -t 0 ] though we use [[ in bash.

The -H flag means "use colours for the terminal, as you're being run for a Human, even though you're being piped: I'm piping your output to a terminal"; it also implies automated mode.
Note it's undocumented, though discussed here before, and deeper semantics are subject to change, since it's meant for use with chroot installation, and was implemented precisely for our installer. The relevant script variable (easily searchable) is useTerminal.
Having said that, it hasn't changed for years; you can always complain if and when something does change with it, though if you do use it, it's likely better to tell me here as well, so I'll post if we do some more work on the installer that requires such a change in semantic. It's unlikely since its effects are covered by -x, apart from forcing colour display.
Oh I think it doesn't ensure certain deps like dialog either, because it's designed for use in early-install.

Additionally, it has specific exit codes, documented in the script, only about 7 or 8 afair, so that the calling script can know about certain outcomes. Haven't added any for ages, since no-one's ever even mentioned using the ones that are there, let alone ask for others.

Talking of dialog, I quite like the -E flag to go straight to Editing the list, which might be useful in certain situations for a calling script (eg retrying, when you know you're interactive, and the user wants to tweak.)

For more info, run update -h and update --help (which is a different option, not aliasing the first.) You might like update -h error as well, if you plan to do much custom-tweaking. Thank drescherjm for that ;) (the base function does what he mentioned there, or is meant to, afair.)
update -h config will always show you the latest config settings (directly from the script), and their defaults.

HTH,
steveL

(I fixed the topic link above for you; you can edit your post and mod it to use the same; select the text, and click the "Topic" button, or type the tags.)

PS: there's a bug with autounmask when you skip the package, in unstable portage, which is much tighter and does not accept full CPVs, only pkg or pkg:slot, for exclusion. It doesn't affect the majority of exclusions, but thought I should mention it; should be corrected within a week or two, depending on when Griz next shouts at me about it. ;p

Just in case anyone tries to register on the trac, sorry but we've closed that for a few days, due to a deluge of fake account spam, till infra sort out a captcha or something they're happy with.

Shame, as I've always hated captchas, but can't see any other real option. In any event, if you need an account (unlikely I know ;) then feel free to pm me, or ask (Monkeh, igli or Griz64) in #friendly-coders on IRC: chat.freenode.net