Have something to say?

Ready to be published? LXer is read by around 350,000 individuals each month, and is an excellent place for you to publish your ideas, thoughts, reviews, complaints, etc. Do you have something to say to the Linux community?

My own contribution to apt download efficiency

A long time ago, I opened a feature request bug report for what “apt-fast” does.

/etc/apt/sources.list can be populated with multiple archives, added by hand after using apt-spy to find the nearest/fastest archives. So long as “ftp.debian.org” is the last entry in the sources.list, it will be used only if the other archives haven’t updated to the latest package yet. Otherwise, apt will use the first entry in the list that has the package available.

This also works with different archives, such as deb-multimedia.org, where packages are unique to the archive. Apt will launch as many downloads in parallel as their are unique packages available on different archives. So one will be pulling from the top of my sources.list entries, and one from deb-multimedia.org, and one from the google chrome archive.

But if I’m running an update that wants to download, say, 30 packages, and all 30 have been replicated amongst the 5 main-stream Debian archive entries in my sources.list. Apt now will launch only one download at a time, pulling all 30 packages from the same archive, the first one on my list.

Effectively, apt _ignores_ duplicate entries.

What I wanted in my feature request was for apt to pull from all 5 “duplicate” entries at once. Not what aria is doing, pulling pieces of the same package from different places, just distribute the downloading of different packages from duplicate archive sites.

So of those 30 packages in my example, the first 5 of those packages will be pulled in parallel, one from each of the 5 entries in sources.list. When one completes, the next package that is desired will be pulled from that archive, and so on until all 30 are retrieved.

One of those 5 archives might provide only one big package, while another might provide a dozen smaller packages, but regardless the bottleneck becomes my own download speed and the load for file serving is distributed amongst the servers.

This uses no new configuration what so ever. Apt already keeps records of what files are available in what archive in sources.list.

Thanks for listening.

Posting in this forum is limited to members of the group: [ForumMods, SITEADMINS, MEMBERS.]