I have one question.
I want the solution with nfs share (only 2 clients) but i don't know which direcotries i have to share. i want to do a emerge sync only on the server maschine and emerge -u world manually.
Ok there is /usr/portage and /usr/portage/distfiles but something else? Something in /var ?

You see. That is why we have forums. I haven't read that particular man page in a long time. That'll solve part of the problem. Satisfactorily (spelling ...) for now. And probably for a long time to come.

Next question, we maintain some of our own ebuilds, some of these are queued for inclusion or we're pushing to get them in, but some of these will never get included. We have a seperate rsync section for this called gentoo-local-portage to which we can manually rsync. Since we are running the sync's out of cron'ed scripts I suppose I can just add an additional rsync command to those scripts, but it would be nice if portage has some way of doing this for me, ie, if I can ask it to sync /usr/local/portage against rsync://some.server/gentoo-local-portage whenever emerge sync gets called . Any ideas? And this time I promise that I have completely read and understood man 5 make.conf and has not seen such an option._________________There are 10 kinds of people in the world,
those who understand binary and who don't

I have one question.
I want the solution with nfs share (only 2 clients) but i don't know which direcotries i have to share. i want to do a emerge sync only on the server maschine and emerge -u world manually.
Ok there is /usr/portage and /usr/portage/distfiles but something else? Something in /var ?

AFAIK sharing /usr/portage (including all subdirectories) should be fine. With the default setup portage uses /var only for hostspecific (e.g. /var/lib/portage/world) and temporary files (e.g. /var/tmp/portage/*).

@jkroon

My first guess was moving /usr/local/portage to a subdirectory of /usr/portage and then add this to PORTDIR_OVERLAY, but after a small test I had to find out that PORTDIR_OVERLAY isn't working the way make.conf(5) implies.

make.conf(5) wrote:

Defines the directories in which user made ebuilds may be stored and not overwritten when 'emerge --sync' is run. [...]

During 'emerge sync' /var/portage/sys/dummy is deleted with all files in it and at the end emerge complains about PORTDIR_OVERLAY not being a valid path (anymore). Well, at least none of the files has been overwritten .

Nevertheless, /usr/local/portage could still be moved to a subdirectory of /usr/portage, but should be added to the file pointed to by RSYNC_EXCLUDEFROM (actually the name of the subdirectory, not the full path and don't use '/local' as this is excluded in any case). This way 'emerge sync' on the clients will include your own ebuilds, but won't touch them on the server. No need for a 'gentoo-local-portage', though the solution you already mentioned might be the better one.

Basically, the problem is that emerge can sync with one server/module only, unless SYNC (and PORTDIR) is changed every time.

I think I'll just go with the additional rsync command in our nightly scripts - it is imho the cleaner solution. Also make it easier to distinguish which is custom ebuilds and which are portage ebuilds. It doesn't solves the problem for all but a small number of desktops, which isn't that big a problem - we can probably just write a small script that does both emerge sync and rsync -rav --delete rsync://out.rsync.server/gentoo-local-portage /usr/local/portage in one go._________________There are 10 kinds of people in the world,
those who understand binary and who don't

Ok, torpage 0.2.0 is available for testing. It still lacks ACLs but who cares? For most of us this is simply a caching solution and EXCLUDEFROM can be used to exclude games-* (which is what I wanted to have ACLs for).

The new version instructs the server to fetch a specific file for a specific package and version, the server then manually parses the appropriate digest files to check whether these files are actually in those packages, and then does some nifty parsing and sourcing of files to determine the upstream mirrors that portage would have fetched from and then manually fetches and does the checksumming. I do this by hand since portage cannot fetch a particular file for me at this point in time - nor is the emerge command available on non-gentoo servers (and yes, I do have the need that torpage run on non-gentoo systems on the server side).

Additionally torpage doesn't take over your fetching entirely any more, it now actually specifies a seperate torpage:// protocol, so you can have multiple upstream torpage servers (ok, that is just leetness factor and doesn't imho have much real use) along with other standard ftp:// and http:// servers and torpage won't interfere with these. This allows for fallback should torpage fail for some reason or another. I've even managed to configure it to use different upstream torpage servers depending on the network I'm connected to, or to silently fail if I'm not connected to one of these networks.

Anyway, please take a look and let me know of any shortcommings (both in my code and in documentation).

Posted: Tue Apr 05, 2005 4:50 am Post subject: How do I get back my original emerge cache ?

HI,
Is there any way I could get back my original emerge cache/ portage tree, after doing emerge --sync
Sorry, if this question is too silly to be replied to, by all the GURUs out there. I recently installed Gentoo from the universal CD (install-x86-universal-2005.0.iso) and then installed KDE, GNOME, etc from the package CD (packages-x86-2005.0.iso) using emerge --usepkg <package-name> This installed KDE 3.3.2.
Well, then I did emerge --sync. Now I can't install any kde-3.3.2 package since all the entries in the emerge cache/portage tree point to kde-3.4. Thus I can't use the package CD. I want to be able to install packages from the Package CD which has got kde-3.3.2. So, is there a way by which I could get back my original emerge cache or portage tree. I was thinking emerge --regen. Would that do it ?
Also I am not very sure whats the difference (if any) between emerge cache and portage tree.

Something along those lines might work - and if not, emerge --sync will get you back where you are now ._________________There are 10 kinds of people in the world,
those who understand binary and who don't

If you have several machines as well with different configurations, and several flagship machines that do compiling, etc... then you might want to consider something like this...

Server
/ \
/ \
/ \
PC(0) [....] PC(n)

Where PC(0) through PC(n) have NFS access and are trusted for authentication. Given that each and every machine can mount and access the distfiles NFS share, you can setup a simple cron command where the trusted machines will 'sync' with the server by running cp /usr/portage/distfiles/*. Then you can run rsync to your heart's content. Why not just use NFS? Because NFS requires an additional service for all the non-flagship machines and rsync doesn't require any additional kernel modules.

Portage uses two methods to keep an updated Portage tree and retrieve current Gentoo packages (distfiles). Rsync (rsync) is used for the tree and wget is used for the packages. Im not aware of the motivations behind these protocol choices but they work very well.

Great article, but wget is not a protocol.

Last edited by jasperbg on Tue Jun 07, 2005 6:00 am; edited 1 time in total

You really don't want to be doing any kind of NFS unless your _entire_ network can be trusted. It is simply too insecure. Take for example a university (Yes, I'm a sysad at a university) where I'll need to export /usr/portage rw to the entire campus (well, a large part thereof anyway) to make your trick work. Not something I'm willing to do since this means some cracker can go and put a new baselayout along with a trojanned archive on the server and everybody would install it.

Nope, not a good idea imho, plus there is locking issues even if you do trust the entire network._________________There are 10 kinds of people in the world,
those who understand binary and who don't

Yes, and that's why something should be done for authentication dealing with trusted keys and so forth like kerberos auth.

Ok, we're going off topic, but OpenAFS seems to be the only viable alternative at this point in time, and for heavens sake, don't use the version in portage. Anyhow, kerberos itself has some severe flaws in as well with regards to the way it works making replay attacks possible. But much better than NFS. NFSv4 (kerborised NFS) might also be an alternative if it should get stable at some point.

Anyway, with regards to a central repository a read-only NFS export of /usr/portage (And $DISTDIR if different from the default /usr/portage/distfiles) and then mounting that and using something like torpage is about the best solution I've managed to come up with. Using it both at home and at work and it's like a charm, it just works. Well, at work I'm only using it as a on-demand mirror (ie, download from outside once, download to servers/workstations from there). At home I've got a bit of a mixture atm (need to get everything back to ro NFS)._________________There are 10 kinds of people in the world,
those who understand binary and who don't

First off, thanks to everyone for the well-written, thoughtful, and useful comments.

Here at work we've got something of a similar situation: five recycled Ultra 5's do most of the dirty server work, with one of them being the designated binary package build machine and portage sync'er. Because they are all very similar hardware (only processor speed and RAM differ), they all use the same make.conf and general filesystem layout. Actually, I've got a neat rsync setup that lets the other machines pull this and some other files automatically from the server.
At first, I setup the machines to use binary package downloading using emerge -g, but this had some drawbacks:

1. It's slow: every time I wanted to emerge a package, some 'metadata pickle' had to be downloaded and parsed, and this took ages.
2. It required the setup of an http server on the package host, which was one more thing to worry about that I didn't want.

I also had that host serve as the rsync mirror, which had another few disadvantages:

1. It came off of an actual hard drive, which meant that it was probably 5-10x slower than syncing to an official mirror (which act like they're on RAM drives.)
2. It caused a ton of grinding each time the tree was sync'd! This worries me because I want these servers to go the next 5 years with minimal adjustments on the administrator's part (I'll be leaving soon), and all that grinding * 4 for each machine that had to sync just gave me a bad feeling.

Solution? Share /usr/portage read-only over NFS to the local network. The other machines automount /usr/portage, and I can use emerge --buildpkg (actually, I have it set in FEATURES in /etc/make.conf to build binary packages) to build packages on the main host, and then install them on the others using emerge --usepkg.

Actually installing packages has worked great, but I'm worried because emerge goes through a huge round of 'quarterly updates' when I first automounted /usr/portage, and sometimes it acts like it wants to write into those directories:

I've submitted a patch that removes the need for +w to ${DISTDIR}, but then you need some other method to fetch files (like torpage - http://www.kroon.co.za/torpage.php works nicely). I like the --usepkg and --buildpkg idea, I quess one can even use the same idea that torpage uses for fetching distfiles for building packages ... (aka, packages gets built on demand whilst the client waits)._________________There are 10 kinds of people in the world,
those who understand binary and who don't

I am having a problem setting this up as described in Setup #3. I have set up rsync exactly as described and setup my client accordingly. However, whenever I do an emerge --sync, I get the following error:

Well, I figured out my problem, I was making changes to the /etc/rsync/rsyncd.conf file instead of the /etc/rsyncd.conf file. However, I now have a new problem. What I would like to happen when I emerge something (ie. emerge gdesklets-core), have my portage server download the ebuild for that package all and the dependencies if its not there and cache it in its portage directory, and then serve it to the client that requested it. However, thats not what is happening. When I do a 'emerge gdesklets-core' here's what I get:

Code:

linuxclient etc # emerge gdesklets-core
Calculating dependencies ...done!
>>> emerge (1 of 5) dev-python/pyorbit-2.0.1 to /
>>> Downloading http://distfiles.gentoo.org/distfiles/pyorbit-2.0.1.tar.bz2
link_stat "pyorbit-2.0.1.tar.bz2" (in gentoo-packages) failed: No such file or directory
client: nothing to do: perhaps you need to specify some filenames or the --recursive option?
rsync error: some files could not be transferred (code 23) at main.c(653)
>>> Downloading http://distro.ibiblio.org/pub/Linux/distributions/gentoo/distfiles/pyorbit-2.0.1.tar.bz2
link_stat "pyorbit-2.0.1.tar.bz2" (in gentoo-packages) failed: No such file or directory
client: nothing to do: perhaps you need to specify some filenames or the --recursive option?
rsync error: some files could not be transferred (code 23) at main.c(653)
>>> Downloading ftp://ftp.sunet.se/pub/X11/GNOME/sources/pyorbit/2.0/pyorbit-2.0.1.tar.bz2
link_stat "pyorbit-2.0.1.tar.bz2" (in gentoo-packages) failed: No such file or directory
client: nothing to do: perhaps you need to specify some filenames or the --recursive option?
rsync error: some files could not be transferred (code 23) at main.c(653)
>>> Downloading ftp://archive.progeny.com/GNOME/sources/pyorbit/2.0/pyorbit-2.0.1.tar.bz2
link_stat "pyorbit-2.0.1.tar.bz2" (in gentoo-packages) failed: No such file or directory
client: nothing to do: perhaps you need to specify some filenames or the --recursive option?
rsync error: some files could not be transferred (code 23) at main.c(653)
>>> Downloading ftp://ftp.gnome.org/pub/gnome/sources/pyorbit/2.0/pyorbit-2.0.1.tar.bz2
link_stat "pyorbit-2.0.1.tar.bz2" (in gentoo-packages) failed: No such file or directory
client: nothing to do: perhaps you need to specify some filenames or the --recursive option?
rsync error: some files could not be transferred (code 23) at main.c(653)
>>> Downloading http://ftp.gnome.org/pub/gnome/sources/pyorbit/2.0/pyorbit-2.0.1.tar.bz2
link_stat "pyorbit-2.0.1.tar.bz2" (in gentoo-packages) failed: No such file or directory
client: nothing to do: perhaps you need to specify some filenames or the --recursive option?
rsync error: some files could not be transferred (code 23) at main.c(653)
>>> Downloading ftp://ftp.no.gnome.org/pub/GNOME/sources/pyorbit/2.0/pyorbit-2.0.1.tar.bz2
link_stat "pyorbit-2.0.1.tar.bz2" (in gentoo-packages) failed: No such file or directory
client: nothing to do: perhaps you need to specify some filenames or the --recursive option?
rsync error: some files could not be transferred (code 23) at main.c(653)
>>> Downloading ftp://ftp.gnome.org/pub/gnome/2.0.0/sources/pyorbit/2.0/pyorbit-2.0.1.tar.bz2
link_stat "pyorbit-2.0.1.tar.bz2" (in gentoo-packages) failed: No such file or directory
client: nothing to do: perhaps you need to specify some filenames or the --recursive option?
rsync error: some files could not be transferred (code 23) at main.c(653)
!!! Couldn't download pyorbit-2.0.1.tar.bz2. Aborting.

As you can see, its not even attempting to connect to my portage server. Here is what my /etc/rsyncd.conf file looks like:

Like I mentioned before, what I would like to have happen is if I try to emerge something that is not cached on the portage server, have the portage server download the ebuild for that package and all the dependencies and cache it locally, then serve it to the client that requested it for the emerge. Any ideas what I need to change to make that happen with my setup? Right now, when I do an 'emerge sync' on my client, that works fine. Just not emerge <package>...

That is your problem right there. You need some mechanism for the client to notify the server what to fetch. Some solutions make use of squid, but squid usually don't cache overly large files (yes, it actually has an option to not cache files bigger than x bytes - argument: it will expell too many other pages from the cache). Enters torpage. To the best of my knowledge the configuration for torpage is well documented, and can be obtained from http://www.kroon.co.za/torpage.php - it was developed to exactly what you are now attemting._________________There are 10 kinds of people in the world,
those who understand binary and who don't

Ok, this may sound like a stupid question, but do I need to install this on the server or the client or both? According to the readme on the link you provided, it sounds like both. I just want to make sure so I get this right.

Ok, well I installed torpage onto the server and client anyway. So, disregard my previous post. However, when I try to emerge something it doesn't pull it from my portage server. And my portage server doesn't go out and retrieve it. Here's what I get when I try to emerge gdesklets-core: