Sync 1-2 times per day, maximum. There's being on the bleeding edge, and there is being just plain silly. Analysis of rsync logs show that a few discourteous users syncing 10, 15 or even 25 times per day are using a disproportionate amount of rsync mirror resources. Rsync mirror maintainers have been encouraged to use iptables rules to limit people who are abusing the system.

sometime u get blocked from using a mirror with a message that u used more then (x times) and u actually using it for the first time in that day or maybe even in a while , that kinda sxxxs
just a thought

Sync 1-2 times per day, maximum. There's being on the bleeding edge, and there is being just plain silly. Analysis of rsync logs show that a few discourteous users syncing 10, 15 or even 25 times per day are using a disproportionate amount of rsync mirror resources. Rsync mirror maintainers have been encouraged to use iptables rules to limit people who are abusing the system.

sometime u get blocked from using a mirror with a message that u used more then (x times) and u actually using it for the first time in that day or maybe even in a while , that kinda sxxxs
just a thought

This is a perfect illustration of the proverb that says that a few rotten apples spoil the barrel. It goes without saying that setting up iptables rules to prevent users from repeatedly connecting to an rsync server is going to result in non-offenders getting screened off as well. I do not rsync more than twice in a week (barring the possibility of a series of vital security updates in a short span), but I will not be surprised to see myself getting an occasional refused connection as the recommendation gets adopted more.

The only solution that will please everyone is for the offenders to stop what they're doing. Oh wait, maybe it won't please them... oh well, I guess we all get to suffer because of the selfishness of a few. _________________Vogon poetry is of course the third worst in the Universe. -- Douglas Adams, The Hitchhiker's Guide to the Galaxy

I have a local rsync mirror that resyncs hourly. Is this considered against rsync etiquette? It's serving 8 Gentoo boxen at the moment.

I don't suppose I speak for anyone but myself, but in my opinion, yes, you are breaching rsync etiquette. What do you expect to gain by rsyncing on an hourly basis? _________________Vogon poetry is of course the third worst in the Universe. -- Douglas Adams, The Hitchhiker's Guide to the Galaxy

The main advantage is that if I need to get a brand-new ebuild (especially for GLSA's), and the server hasn't synced yet, I can't get it until my server syncs. I think I'll increase the time to once every three hours anyways, though.

The main advantage is that if I need to get a brand-new ebuild (especially for GLSA's), and the server hasn't synced yet, I can't get it until my server syncs. I think I'll increase the time to once every three hours anyways, though.

Exactly how difficult would it be for you to make a policy of having the server do a daily rsync, and then manually perform it when advisories are released after the daily one? Eight times a day to save a little effort still seems excessive to me. Not to mention that there have been dry spells on the order of nearly a week between GLSA announcements in the past._________________Vogon poetry is of course the third worst in the Universe. -- Douglas Adams, The Hitchhiker's Guide to the Galaxy

I think it's safe to say there's no need to be rsync'ing more than twice a day and I can only hope that those people do it due to being unaware of the situation. Hopefully this will help raise awareness.

I see people on here who incessantly `emerge sync && emerge -u world` without realizing that they're wasting resources.

Gentoo's already bandwidth intensive enough without people hammering the servers...

For anyone interested in further reducing bandwidth on Gentoo servers, you might want to take a peek at this thread a couple weeks ago:

I agree that to be rsyncing more than twice a day on a regular basis is probably excessive. Realy, what advantage do you get by automatically rsyncing more than once a day. Like the message in the first post said,

Quote:

There's being on the bleeding edge, and there is being just plain silly.

You can always rsync manually when there is a glsa, not like it's hard.

One other thing I see is, judging from conversations on these forums, people are a little too quick to nuke their distfiles in my opinion. Just because it is safe to do doesn't make it a good idea, it's certainly not courteous. By all means, clean out the cruft, but why load the distfiles servers excessively by downloading the same package more than once just because you were lazy. I like the distfile diffs idea. But I think we as users could do a lot better with the current system. And please be very quick to share your distfiles among machines if you have more than one. And copy them over if starting a new install. Little things like that, if we all do it , would help a lot I think. Am I completely off my rocker?

This thread is all about cleaning out stale distfiles, which means that the distfiles that are still currently "in use" remain on your system unless they aren't used by any packages. This prevents re-downloading of the same files.

Again, like I've been saying to anyone who will listen, I think the key is to keep these distfiles on the harddrive and use small, incremental patches to patch them to newer versions.

Gentoo can be MUCH more bandwidth friendly (while still being source-based) than it is right now. I hope that the developers are keeping this in mind for the future.

I have setup one of my boxes to serve via NSF its /usr/portage dir. It wasn't that hard and seems to work quite well. I just have to finnish some changes to my newer box and all will be using NFS to one /usr/portage. I rsync from 3 to 6 times per week depending what I'm doing or if I'm looking for some update to come thru._________________Brian
Porthole, the Portage GUI frontend irc@freenode: #gentoo-guis, #porthole, Blog
layman, gentoolkit, CoreBuilder, esearch...

Hell, I break those guidelines daily at times.. but I'm not happy about it

Problems:

a) broken ebuilds
- Sometimes we have broken ebuilds in unstable, and those of us who act as willing testers for Gentoos unstable branch actually need to do emerge rsync a few times a day (also connected to d) at times.

b) incomplete rsyncing
- Rsync fails all the time for me, stalling halfway, thus I hit another server this can't be helped.

c) important updates
- if I see a GLSA notice I rsync... it's that simple.

d) mitration speed
- rsync is SLOW at migrating changes, so if an important fix has hit the mirrors, I keep hitting the servers once an hour untill I have it - like the XFree update a few days ago.

I can only conclude that rsync needs to die, and it has already be discussed in a previous GWN that we need to replace rsync with a less server demanding protocol - cvsup fx. but nothing has happened in the matter AFAIK. I'm known not to be rsync' greatest cheerleader.

Maybe we could get an option to pay for rsync services if we needed regular access, and then give the paying users prior access to mirrors (I'm guessing this can be done using iptables rules and some sort of authentication) - I would, for one, really hate to get the message, "You have reached your daily limit", and then just be kicked off the network.

#1, updating through patching is already being done in other source distributions like sorcerer/sourcery (dunno if sourcemage is doing it), so it can't be impossible (again, I discussed possible implementation strategies, feel free to brainstorm in that thread, I might try to hack a prototype model when I'm finished this semester).

#2, if everyone kept their rsync usage to a reasonable level there'd probably be no need to get denied, or have to pay. I think raising awareness through the GWN, forums, and even on the mirrors is a valid way to get this message across.

1) I know - but that don't change the fact that it's hard to do.. I've been looking at how to go about it, and I have a few promising ideas, but the best option might be to simply enforce the use of a "base package" and then release everything from there as diffs, the kernel fx. already does this.

2) I also know that - I would have no problem paying because I know I use the mirrors more than most people.. I was talking to a few people a while back about replacing and Gentoo transfers with BitTorrent to surcome this saturation issue and I think it would be an exellent approach for the file transfers but the rsyncing is a bit trickier to do, a sidebenefit here would be that with bittorrent it would be DEAD easy to setup a mirror, all you would have to do was to start bittorrent and point it to the tracker, as soon as you were done downloading you would automatically become a mirror, and would stay that way untill bittorrent was closed.

on point #2, I think it would be great if some kind of secure, transparent bit-torrent method could be used, but bittorrent's only as good as the popularity of the file. Of course this would work well for new-releases (Gnome 2.2, KDE 3.1.1, XFree86 4.3).

All in all, I think anything to take some of the strain of the mirrors would be good. I'd hate to think of Gentoo-users as bad netizens due to their high bandwidth requirements.

one thing about reporting problems with broken mirrors:
i once connected to a rsync mirror that downgraded my portage tree to a >1 week old version, but i did not report this as a bug.
reason: i did not know which mirror it was as this information hat scrolled out of the terminal (and out of its buffer) in the meanwhile. since then i usually try to take a look at the mirror. maybe emerge sync should print the server information at the end and/or log it to /var/log/emerge.log, too.

There was some discussion about a p2p file sharing system (volunteers only ofc), so a gentoo install could share one or two files, so most of the main distfile servers could be turned into rsync servers which would help the load somewhat...

Maybe it could be adapted for a p2p rsync, a rsync server on a standard gentoo install that would only allow one connection at a time..._________________95% of all computer errors occur between chair and keyboard (TM)

These are guidelines, not rules. We are asking for your help in keeping our rsync mirror system sustainable for the future. That said, if you occasionally have to sync a third time in a 24 hour period, you're not going to have jack-booted thugs show up on your doorstep with a cluebat...

Only the most aggregious users will be rate limited. Syncing more than twice a day won't automatically get you rate limited. Syncing against one particular server, 50 times per day almost certainly will get you rate limited. Don't be stupid and you don't have to worry about it.

Don't single out particular servers I can't stress this one enough. Not only is it poor etiquette, it's stupid as well -- you draw attention to yourself because your IP address hits the logs of that server over and over. If you use rsync.<continent>.gentoo.org or rsync.<country>.gentoo.org, you'll spread your usage over many servers. This is a Good Thing.

This is a very real problem, folks. One of our German mirror admins ran a script against his logs and found that the top 5 users were consuming over 50% of the CPU resources going to rsync on that machine. (Those users have all been rate limited, btw.) We're not trying to be jerks about this -- but when I have multiple rsync mirror admins complaining to me about the amount of resources we're taking up, we have to do something about it. With your cooperation and assistance, this GWN article will be all we have to do.

One final point to Lovechild's question/suggestion on paid access to rsync mirrors. That's something we're also looking at doing. The issue comes down to building out the payment processing infrastructure. It's on the radar and something that I'd like to see us do. I just don't know when we'll have the time or resources to do so.

--kurt_________________The problem with political jokes is that they get elected

Last edited by klieber on Mon May 05, 2003 12:36 pm; edited 1 time in total

d) mitration speed
- rsync is SLOW at migrating changes, so if an important fix has hit the mirrors, I keep hitting the servers once an hour untill I have it - like the XFree update a few days ago.

Have you considered waiting 4 or even (gasp) 6 hours to let the changes propogate naturally? All the rsync mirrors sync every 30 minutes against one master rsync mirror (rsync1.us.gentoo.org) so all mirrors should have new updates within 30 minutes of them hitting the tree.

That said, I fail to see how waiting a few hours to let the changes work their way down is really that onerous of a request. IMO, the rest of your points were valid -- this one is bogus.

--kurt_________________The problem with political jokes is that they get elected

Perhaps it would help to have emerge check (by default) how many rsyncs have been done in a 24 hour period and print out a message after the fourth or fifth time. Part of the problem may be that users just aren't THINKING about what they are doing.

on point #2, I think it would be great if some kind of secure, transparent bit-torrent method could be used, but bittorrent's only as good as the popularity of the file. Of course this would work well for new-releases (Gnome 2.2, KDE 3.1.1, XFree86 4.3).

All in all, I think anything to take some of the strain of the mirrors would be good. I'd hate to think of Gentoo-users as bad netizens due to their high bandwidth requirements.

Ah, but all the mirrors are still running bittorrent, so the bw. is actually better even when it's an "unpopular" file... like emacs or something

Bittorrent would make it a lot easier to cope with sudden rushs - like when the new versions of KDE, xfree and GNOME are released - big popular packages.

I know I can set GENTOO_MIRRORS in make.conf - but I thought rsync already pointed to a master hub that spread the load to a random non busy mirror.

Anyways if this is not the case, then what's the syntax we need to add, and where's the "official" list of rsync mirrors (I assume we have one someplace) so we can change it to aid the cause.

-edit-

I'm a bit confused, but the answer to my question seems to be, don't mess with it, since that should use the official rotation, hardcoding according to GWN would be bad - yet in this thread it's mentioned as "a good thing".. hrmm

I'm a bit confused, but the answer to my question seems to be, don't mess with it, since that should use the official rotation, hardcoding according to GWN would be bad - yet in this thread it's mentioned as "a good thing".. hrmm

See this Tips and Tricks. You can change your SYNC variable in /etc/make.conf to something that is geographically closer to you. Personally, I encourage all users to use the continent-level rsync rotations.

--kurt_________________The problem with political jokes is that they get elected