I've been asked sometimes about the hardware behind mirror.rts-informatique.fr, the server load, the sync process and even the Cherokee theme May this also be a little view into Linux Mint Debian and it's growing.

As of today, mirror.rts-informatique.fr is a DNS round-robin on IPv4 and IPv6 for 2 servers: "ganymede" and "callisto". Yes there will be an "io" server a little before UP6 (even a "jupiter"), but "no europa" (2001 A Space Odyssey and the following books, this quote is from 2010 )Both servers are 100-megabit OVH dedicated servers, located in Roubaix (France). To preserve bandwidth, ganymede syncs daily with debian.linuxmint.com (download), and then syncs with callisto (upload). The hardware of the two servers is a bit different, but the config is the same: Ubuntu 12.04 servers with autoupgrade and stuff.

When connecting to mirror.rts-informatique.fr, you end up randomly on ganymede or callisto. Between downloads, your computer may swap from one to another. So actually this gives quite accurate logs for unique IPs on both of them.

Here we go:mirror.rts-informatique is used as APT mirror for at least 2500 IPs resulting in about 2700 sources.list (counting multiple PCs at home or office, virtualboxes,..).Peak activity is at 17:00 UTC, lowest being at 05:00 UTC (two-third less).8% of all connections are IPv6Each server has an average weekly output of ~20 Mb/sTop countries are:

FranceSpainUKNetherlandsGermany

[/list]100% uptime (had to reset the graph since Pingdom had buggy monitors not so long ago)The average user download speed is at 6 Mb/s, the average user latency is at 110 msAverage system load of both servers is at around 0.08, quite steadydisk usage: 91 GB for ISO, 13 GB for Packages, 223 GB for Debian latest (that's around 450 GB with the two Debian pools)

Summary:Right now the load is low enough to cut down to 1 server, but with UP6, the pack size and the user count, we'll need to set up a third one with same config ("io"), and a 1000-megabit server (which can't be found low-cost at OVH, so we're looking at a Dedibox DC for "jupiter"). 4 servers up and running with over 2 TB traffic expected, this will be stressy

Update pack releases are a great real-world practice for heavy load infrastructures for me, I've learn many things with UP5 and I'm happy to share this so if you want to set up a repository mirror*, feel free to ask me advice

* yes I really don't know how many people are using LMDE right now, but with 3 repository mirrors all located in Europe this isn't good: we really could need mirrors in the US/Canada, somewhere in Asia like South Korea, and in Australia. Yes it takes a lot more disk space than "packages" so there won't be as many mirrors, but please, 1 TB servers can be found nowadays and the project really needs it before UP6 gets out

Last edited by ketoth on Mon Nov 12, 2012 8:15 am, edited 12 times in total.

Quick announcement: for the next few days, I gonna do some tests. Because it involves DNS manipulation, the mirror may be unavailable. I gonna progress safely so that your cached DNS always point to valid addresses, but who knows if your cache just happens to record between a NS migration. Everything should be ok within 24 hours whatever happens.

What I gonna do ? Test some cache & CDN infrastructures. Right now and for improved DNS refreshing, the domain runs on Cloudflare. That won't be a long term solution, since they disallow "non-web" content in big amounts (while the mirror contains 600 GB of ISOs and DEBs ). Amazon Cloudfront is an option too, but their billing system scares me off (keeping record of credit cards, pouaaah). Microsoft Azure ? LOL. Akamai ? No.

I just signed to the beta of OVH CDN. This will be active in the upcoming days, with pools in Europe, USA+Canada, and Asia. Since it's beta, bugs can happen. I do prefer now than just while an update pack is released If you have any bugs or questions, feel free to post

Objective: rely on only 1 dedicated server behind a global CDN for the mirror, to cut down costs while providing top-level speed & availability especially on update pack release. More soon

Cloudflare, leave my DNS zone alone whaaaaaaaaaaaa... Screw you and your "browser cache enhancements". Oh, and screw you too, Google DNS. When the TTL you show is 290, you should refresh every 290 seconds like the clients, you moron. By the way OVH has 600 as TTL.Ok, OVH CDN is so much of beta right now that the server headers return a "accept-range: bytes" flood that -of course- lead to an "entity too long" error. AND: APT listings are modified so that the signature doesn't match. Oh yeah sometimes, out of the blue, Cloudflare pops up alerting you that -of course- the OVH CDN IPs are blacklisted. Results: some stuff is cached, some stuff buggy OVH CDN side, some stuff disallowed by Cloudflare, some stuff in DNS chaos by Cloudflare, some stuff in the internet void by Google DNS, and some stuff in cosmic /dev/null. Don't try this at home.

Long things short:- kick Cloudflare out and remove domain info, don't touch my zone with your filthy scripts- NS for rts-informatique/./fr back to OVH (hell yeah it didn't even break the DNSSEC)- mirror.rts-informatique/./fr points to "europa" server (previously known as callisto, because yes that's the one I gonna keep)- mirror.cdn.rts-informatique/./fr points to the global OVH CDN frontend (don't use: that's for testing purposes), behind of which is "europa"(ganymede is left alone, but still up and running in case I need it)

Gonna ask OVH if they plan IPv6 support for their CDN, I love IPv6. Gonna wait a bit while DNS are refreshed, then gonna jump off the cliff adding "mirror.cdn" in my sources.list and try again. You can do it too, but report any problems you may encounter. Merry DNS-mas

ps: I'm so glad I did not start this kind of stuff just while an update pack is released..

EDIT: oh dear, thanks for the 34 min downtime pingdom, you got yourself a bad DNS.Anyway according to my today's testing, it seems ready to rumble. Let's bring everything through the CDN this tuesday.

EDIT2 30/10: meeeeeh. Last night I was trying several hours to track down an error, that corrupts the package listings (APT tells about bad checksum). Not sure about it but there was a cache mess: package listing was cached by the server or the CDN (or both) just before the XChat update was released. So while the general listing is ok, some edition listings were old and therefor mismatching checksum. Sorry for the APT panic

Right now "mirror" points directly to the server. For further testing, added "mirror.cdn0" that points to the CDN. Currently updating a freshly installed LMDE, seems to work fine. Any caching and compression was disabled from Cherokee, to make sure the CDN gets it correctly. Later today gonna plug "mirror" behind the CDN again to see if listings aren't obsolete/corrupted any more... and that the CDN does have effect on server load. Also, only *.deb and *.gz will be cached (that's the point, reduce network load).

EDIT3 31/10: No, it ain't funny. Seems like OVH CDN did record Cherokee's expire header and so still giving the checksum error. Expire header that was set to 1 week, that means november 6th. Next wednesday I can test once again the mirror through CDN. If that doesn't work either, well.. I just gonna tell Clem that a new LMDE mirror appeared, with another subdomain name (since "mirror.cdn0" works just fine). Oh BTW: the CDN can cache files up to 20 MB. Gonna check if this is relevant with UP5.

EDIT4: oh yeah, only a handful packages of UP5 weight more than 20 MB. On average the packages weight 0.7 MB I hope OVH CDN stays beta until UP6, because I think it won't be cheap but it could be really helpful to handle the huge network load. As said, I will try to implement the CDN on the mirror the november 7th, with 48 hours cache.

CDN activated. Some users (well.. everybody ?) are experiencing high latency and low download rate. I'm investigating this further, if this can't be mitigated quickly I guess I will stick to my traditional 2-server load balancing and plug the CDN off. Thank you for your patience

Ok, the data collectors and cachers are now working better. An almost full basic MATE/Cinnamon UP5 is already cached, and other data is fetched quickly. Here, in France, I attain full speed.So I could need feedback from people especially from the USA, from Australia, from Russia, from Brazil and from either India, Japan, China, Korea or Singapore. What to do:

- use a fresh installed LMDE (virtualbox can be used) with UP4- run "apt update": it should take less than 30 seconds the first time, and less than 10 the next times- run "apt dist-upgrade -y": a single download should start in less than a second, and the average download speed should be at least 300 kB/s (if your internet access allows it // ideally a 80%+ of your max speed would be great)

After UP5 is installed, feel free to install whatever software you like to install.- files more than 20 MB in size aren't cached: the CDN only works as proxy then. "apt install 0ad" Here again, interesting information is time for the download to start, and download speed.- install software few to no people use, where there's no chance it was cached before- "apt install", "apt remove", "apt clean", "apt install".. software multiple times to check the caching

Post results with your geographic location. You can also post the latest valid hop when you run a "traceroute", gives a hint on what relay you're on. Thanks for your cooperation

I have my own squid cache on home server, so I get about 250kB/s at first get (probably VirtualBox test dist-upgrade) and 6-10MB/s at subsequent gets .So I'm trying to save your bandwidth And it really helps to test update packs on poor 2Mbit connection.

Ok, time for some conclusions. According to the CDN stats, about 11 TB of data has been passed through worldwide (which appears too much to me, maybe they only count GET requests + file sizes instead of actual traffic).I just unpointed the CDN from the DNS record, why: 404 errors aren't smartly handled, the CDN relays may request each other and retry multiple times on the server, result is a good 3 second delay for each request. And since APT is trying to get translations from every sources line, "apt update" takes about 2 minutes to complete.

This issue didn't improve in time. Therefor, I'm planing to activate the CDN only in update pack releases, because the users know downloading and installing will take more than a hour and can afford to wait a minute to have their lists updated.For now the mirror points directly on "europa", next week I will renew "europa" renting for 3 months but not "pukab". One 100 Mb server is enough for everyday traffic, and there are more LMDE mirrors to share the load

UP6 is approaching, it's currently tested in the "incoming" repo. The situation looks better than UP5 with new high-bandwidth mirrors available in the UK and Australia. But we still need to keep an eye on the servers to be able to react quickly when needed. "mirror.rts-informatique.fr" points to DNS round-robin IPv4/IPv6 with 2 servers: "europa" and "oldie". Both have 800 kB/s per IP capping. I'm doing a 48-hour logging to have more precise statistics of how many users are on this mirror (will edit post with results). Knowing the average weight of UP6 (about 600 MB), I'll know if I have to keep the CDN ready to take over in case of overload.

Took me more time than expected to get the stats, obviously dynamic IPs are problematic All I know is that in a week-long timeframe, around 8000 unique IPs were spotted either downloading ISOs or using APT for mainline or Debian.Also host is capping 100 Mb/s per account, so having two servers doesn't double the bandwidth. Won't renew "oldie", just took it off the DNS records.

Yes the output is at ~95 Mb/s, but with reasonable active connections so it doesn't appear too slow to the users. Again thanks to the mirrors the load can be efficiently shared. No need for the CDN

Well, no I did some stats and logging, found out there's a few complete upgrades every day being cast. Daily average traffic is at ~1.5 MB/s (that's megabytes, not megabits) meaning there's always somebody with DSL line downloading at full speed a whole update pack.Of course that's nowhere close to the kind of traffic when a thousand people want their 500 MB of updates at the same time. Unfortunately, too rare update packs end up being bloated masses of data which are a pain to sync and send.