Package mirroring / Jisc

Package mirroring / Jisc

Jisc, which is the consortium operator and one of the founding board members, has offered to work this year on moving the packages into their Azure cloud, which will mirror things in a few locations around the world in one step. I don't have an ETA on that happening.

We're still willing to rsync or lftp the files to any members or anybody willing to offer open access, and there are a couple of offers for that, nothing in place yet.

Separately, just FYI the data center where our server lives has been having network issues all morning. [1]

Jisc, which is the consortium operator and one of the founding board members, has offered to work this year on moving the packages into their Azure cloud, which will mirror things in a few locations around the world in one step. I don't have an ETA on that happening.

We're still willing to rsync or lftp the files to any members or anybody willing to offer open access, and there are a couple of offers for that, nothing in place yet.

Separately, just FYI the data center where our server lives has been having network issues all morning. [1]

Re: Package mirroring / Jisc

* Cantor, Scott <[hidden email]> [2018-03-15 15:43]:
> I posted all the relevant details at [1] excepting that we're happy
> to use lftp instead of rsync, since it's generally easier for the
> host if you're familiar with it.

Re: Package mirroring / Jisc

On 14 Mar 2018, at 14:46, Cantor, Scott <[hidden email]> wrote:
>
> Jisc, which is the consortium operator and one of the founding board members, has offered to work this year on moving the packages into their Azure cloud, which will mirror things in a few locations around the world in one step. I don't have an ETA on that happening.

Thanks to Peter for working with me on this. We now have a relatively workable system that will be relying on a centrally hosted mirror list that I've put together, so I've updated the repo generation script [1] to produce an indirected reference to that mirror set, which currently only has the Austrian mirror in it.

I decided to omit the opensuse mirror for the simple reason that it just doesn't work, but if this is a problem for Peter's server before we get more mirrors online, I'll adjust it.

> This should be permanent going forward and we'll add mirrors as they come
> online, so it should work as we'd all expect. I will update the docs in the wiki
> as time permits.

An early tester found a missing package from the new mirror and that's been corrected along with the underlying cause. We'll also be clarifying, please just file bugs at https://issues.shibboleth.net if any problems with a mirror occur, so we can triage and reach out as needed, no need to pester the mirror owner as they probably aren't at fault (and unlike OBS, we actually care about the mirrors working, so we'll deal with it).

> I was wondering if you could give me some statistics on the current
> usage (i.e bandwidth requirements, space requirements etc) and I am
> sure we can get some server space setup as a mirror.

Bandwidth: The first 10 weeks of running such a mirror have caused
~860 MiB of incoming traffic (likely including minor package uploads)
and less than 13 GiB outgoing, according to vnstat.
Each new mirror would reduce that even further, of course.

Disk size: The current YUM archive is ~11 GB on disk. So nothing to
lose sleep over, either.