Re: pacserve - easily share Pacman packages between computers

Server pacman.conf file:

[root@testserver ~]# cat /etc/pacman.conf
#
# /etc/pacman.conf
#
# See the pacman.conf(5) manpage for option and repository directives
#
# GENERAL OPTIONS
#
[options]
# The following paths are commented out with their default values listed.
# If you wish to use different paths, uncomment and update the paths.
#RootDir = /
#DBPath = /var/lib/pacman/
#CacheDir = /var/cache/pacman/pkg/
#LogFile = /var/log/pacman.log
HoldPkg = pacman glibc
# If upgrades are available for these packages they will be asked for first
SyncFirst = pacman
XferCommand = /usr/bin/wget --passive-ftp -c -O %o %u
#XferCommand = /usr/bin/curl -C - -f %u > %o
#CleanMethod = KeepInstalled
Architecture = auto
# Pacman won't upgrade packages listed in IgnorePkg and members of IgnoreGroup
#IgnorePkg = yajl
#IgnoreGroup =
#NoUpgrade =
#NoExtract =
# Misc options (all disabled by default)
#UseSyslog
#ShowSize
#UseDelta
#TotalDownload
#CheckSpace
#
# REPOSITORIES
# - can be defined here or included from another file
# - pacman will search repositories in the order defined here
# - local/custom mirrors can be added here or in separate files
# - repositories listed first will take precedence when packages
# have identical names, regardless of version number
# - URLs will have $repo replaced by the name of the current repo
# - URLs will have $arch replaced by the name of the architecture
#
# Repository entries are of the format:
# [repo-name]
# Server = ServerName
# Include = IncludePath
#
# The header [repo-name] is crucial - it must be present and
# uncommented to enable the repo.
#
# The testing repositories are disabled by default. To enable, uncomment the
# repo name header and Include lines. You can add preferred servers immediately
# after the header, and they will be used before the default mirrors.
#[testing]
#Include = /etc/pacman.d/mirrorlist
[core]
Include = /etc/pacman.d/mirrorlist
[extra]
Include = /etc/pacman.d/mirrorlist
#[community-testing]
#Include = /etc/pacman.d/mirrorlist
[community]
Include = /etc/pacman.d/mirrorlist
# An example of a custom package repository. See the pacman manpage for
# tips on creating your own repositories.
#[custom]
#Server = file:///home/custompkgs
[root@testserver ~]#

Workstation pacman.conf file:

[root@testworkstation ~]# cat /etc/pacman.conf
#
# /etc/pacman.conf
#
# See the pacman.conf(5) manpage for option and repository directives
#
# GENERAL OPTIONS
#
[options]
# The following paths are commented out with their default values listed.
# If you wish to use different paths, uncomment and update the paths.
#RootDir = /
#DBPath = /var/lib/pacman/
#CacheDir = /var/cache/pacman/pkg/
#LogFile = /var/log/pacman.log
HoldPkg = pacman glibc
# If upgrades are available for these packages they will be asked for first
SyncFirst = pacman
XferCommand = /usr/bin/wget --passive-ftp -c -O %o %u
#XferCommand = /usr/bin/curl -C - %u > %o
#CleanMethod = KeepInstalled
Architecture = auto
# Pacman won't upgrade packages listed in IgnorePkg and members of IgnoreGroup
IgnorePkg = intel-dri xf86-video-intel libgl
#IgnoreGroup =
#NoUpgrade =
#NoExtract =
# Misc options (all disabled by default)
#UseSyslog
#ShowSize
#UseDelta
#TotalDownload
#
# REPOSITORIES
# - can be defined here or included from another file
# - pacman will search repositories in the order defined here
# - local/custom mirrors can be added here or in separate files
# - repositories listed first will take precedence when packages
# have identical names, regardless of version number
# - URLs will have $repo replaced by the name of the current repo
# - URLs will have $arch replaced by the name of the architecture
#
# Repository entries are of the format:
# [repo-name]
# Server = ServerName
# Include = IncludePath
#
# The header [repo-name] is crucial - it must be present and
# uncommented to enable the repo.
#
# The testing repositories are disabled by default. To enable, uncomment the
# repo name header and Include lines. You can add preferred servers immediately
# after the header, and they will be used before the default mirrors.
#[testing]
#Include = /etc/pacman.d/mirrorlist
[core]
Include = /etc/pacman.d/mirrorlist
[extra]
Include = /etc/pacman.d/mirrorlist
#[community-testing]
#Include = /etc/pacman.d/mirrorlist
[community]
Include = /etc/pacman.d/mirrorlist
# An example of a custom package repository. See the pacman manpage for
# tips on creating your own repositories.
#[custom]
#Server = file:///home/custompkgs
[root@testworkstation ~]#

Xyne wrote:

1) Curl isn't following the redirects. You can try to use the following XferCommand in pacman.conf:

XferCommand = /usr/bin/curl -L -C - %u > %

Run "pacman -Syy" after you edit the file, then check the output of "pacman -Si". If you see the expected output, try to upgrade.

[root@testworkstation ~]# pacman -Syy
:: Synchronizing package databases...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 3422 0 0 8509 0 --:--:-- --:--:-- --:--:-- 8509
error: could not open /var/lib/pacman/core.db.tar.gz: Failed to open '/var/lib/pacman/core.db.tar.gz'
error: failed to update core (unexpected system error)
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 3424 0 0 16359 0 --:--:-- --:--:-- --:--:-- 16359
error: could not open /var/lib/pacman/extra.db.tar.gz: Failed to open '/var/lib/pacman/extra.db.tar.gz'
error: failed to update extra (unexpected system error)
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 3432 100 3432 0 0 16293 0 --:--:-- --:--:-- --:--:-- 16293
error: could not open /var/lib/pacman/community.db.tar.gz: Failed to open '/var/lib/pacman/community.db.tar.gz'
error: failed to update community (unexpected system error)
error: failed to synchronize any databases
[root@testworkstation ~]# pacman -Si pacman firefox thunderbird
error: package 'pacman' was not found
error: package 'firefox' was not found
error: package 'thunderbird' was not found
[root@testworkstation ~]#

Xyne wrote:

2) If that doesn't work, then maybe something is wrong with the mirror or the redirection. Remove pacserve from the mirrorlist, and try to upgrade directly with "pacman -Syy", then check the output of "pacman -Si" again.

[root@testworkstation ~]# pacman -Syy
:: Synchronizing package databases...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
error: could not open /var/lib/pacman/core.db.tar.gz: Failed to open '/var/lib/pacman/core.db.tar.gz'
error: failed to update core (unexpected system error)
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
error: could not open /var/lib/pacman/extra.db.tar.gz: Failed to open '/var/lib/pacman/extra.db.tar.gz'
error: failed to update extra (unexpected system error)
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
error: could not open /var/lib/pacman/community.db.tar.gz: Failed to open '/var/lib/pacman/community.db.tar.gz'
error: failed to update community (unexpected system error)
error: failed to synchronize any databases
[root@testworkstation ~]#

Xyne wrote:

If that doesn't do what you want, then you'll need to look into setting up a proxy on the server to get internet access on the workstation.

I may add a proxy function to pacserve later, but I don't have time to do it right now.

Don't worry. I'll be awaiting this function. A lot of thanks by your interest, patience and help

Re: pacserve - easily share Pacman packages between computers

Logging: Log messages are printed to STDERR by default. You can easily redirect these to a file by appending "2>/path/to/file" to the end of your quickserve command, e.g. "quickserve -p 8000 /path/to/shared/dir 2>/tmp/quickserve_log".

You may want to change the help section to reflect pacserve rather than quickserve.

Re: pacserve - easily share Pacman packages between computers

Re: pacserve - easily share Pacman packages between computers

First i want to say: Congratulations, great tool, great job!

BUT:I've got a problem with pacserve with multicast-option. I've have the following set-up:

* a home-server* a desktop-pc* a netbook

each running with their own instance of pacserve --multicast.There is only one problem: If a package exists on the netbook, the other two computers cannot query it via pacserve.The other way round is possible, and even direct fetching via wget succeeds.This only happens if the package is on the netbook and the other two trying to get it; downloading a package from the other two to the netbook works fine.

Notice following example with "screen" (I've disabled the other pacman-mirrors to make it more clear):

Re: pacserve - easily share Pacman packages between computers

mrman wrote:

Logging: Log messages are printed to STDERR by default. You can easily redirect these to a file by appending "2>/path/to/file" to the end of your quickserve command, e.g. "quickserve -p 8000 /path/to/shared/dir 2>/tmp/quickserve_log".

You may want to change the help section to reflect pacserve rather than quickserve.

I don't remember correcting this but I can't find any mention of quickserve where it should mention pacserve. If I haven't fixed it, can you point me to exactly where that message is?

madjar wrote:

Neat ! Do you plan on adding an init script to the aur package, as it is supposed to be continuously running ?

Adding a daemon is on my todo list. It's not that high up but it's on there. I need to give some thought to configuration first (e.g. a bash config file in /etc/ that the daemon can source for options).

fnoos wrote:

First i want to say: Congratulations, great tool, great job!

Thanks.

fnoos wrote:

[EDIT]Nevermind, everything just snapped into place

I enjoyed the suspense, but please edit the top of your post next time. Until I reached the end, I thought you had discovered some obscure bug and I got worried because I couldn't think of anything that would cause it.

What actually happened? It might be useful to know for future reference.

Re: pacserve - easily share Pacman packages between computers

Hello Xyne, thanks for your great tol and efforts in supporting the community.

I’ve just started a major pacman -Syu on my old Thinkpad X30 which my girlfriend uses from time to time and wanted it to sync — if possible — packages from my brand new X220i. The update is currently running (some several 100 megs)

Although there were some pkgname-pkgver-any packages downloaded yet, and the pacserve servers recognized each other, none of those packages that should be common to both have been downloaded from my x220i. Is that because the x30 is i686 and the x220i is x86_64?

Re: pacserve - easily share Pacman packages between computers

jakob wrote:

Although there were some pkgname-pkgver-any packages downloaded yet, and the pacserve servers recognized each other, none of those packages that should be common to both have been downloaded from my x220i. Is that because the x30 is i686 and the x220i is x86_64?

Yes. If the package architecture is not any, then the package architecture must match the system architecture. That is why pacserve only shared the any packages.

Re: pacserve - easily share Pacman packages between computers

Xyne wrote:

jakob wrote:

Although there were some pkgname-pkgver-any packages downloaded yet, and the pacserve servers recognized each other, none of those packages that should be common to both have been downloaded from my x220i. Is that because the x30 is i686 and the x220i is x86_64?

Yes. If the package architecture is not any, then the package architecture must match the system architecture. That is why pacserve only shared the any packages.

Ah, that was what I wanted to convey: The -any packages were _not_ shared via pacserve in yesterday’s instance of using it. But they should have. Hm hm. Then I don’t know what I did wrong… both pacserves recognized each other.

But after all, the big update has been done and now I think there is no further need to use pacserve from my part.

Re: pacserve - easily share Pacman packages between computers

jakob wrote:

Ah, that was what I wanted to convey: The -any packages were _not_ shared via pacserve in yesterday’s instance of using it. But they should have. Hm hm. Then I don’t know what I did wrong… both pacserves recognized each other.

Maybe you only opened the multicast ports and not the server ports in your firewall. I've just tested pacserve here on different architectures and it shares the any packages as expected. It would have been useful to see the log messages to know what was going on.

Re: pacserve - easily share Pacman packages between computers

Awesome, I've been cutting it real close with my ISP bandwidth cap recently, this is sure to help a little Thanks for making this, I have 4 different machines that can now share the packages, super useful

Re: pacserve - easily share Pacman packages between computers

I've been using it for some times now, but I had never tried the multicast feature, which looks quite awesome (and which is the best solution when the number of machines grows)

However, using pacserve 2011.06.24.1, I get a weird behaviour.

First of all, auto-detection of neighboring pacserves does not always work : sometimes, the two instances (one on the server, the other one on a laptop) do not see each other; sometimes it works one way (A adds B to its server pool but B does not see A); sometimes it works as expected. I can't give a more accurate account as these situations seem to happen at random.

I eventually managed to get the two pacserves seeing each other, but then, running `pacman -Syu' is extremely slow and eats up a lot of CPU.

You'll notice what looks like timeouts (10 seconds between each query). However, the mirror I'm using works fine if pacserve is not running.

Also, on one run, pacserve got completely stuck and ended up eating 800% of the CPU on the laptop. It might be a query loop, but there wasn't any output in the logs. I haven't been able to reproduce this one.

Hope that helps, if you need more log/testing, I'll be happy to provide

Re: pacserve - easily share Pacman packages between computers

@zorunIf you have a firewall, are you using the recommended settings from the project page? Multicast uses UDP packets, which are sent from random ports, so the source port must not be restricted. Firewall settings could also explain the timeouts.

I have no idea what could be causing the high CPU usage (or how 800% CPU usage is even possible). There are multiple checks to prevent loops in the code, and all queries and multicasts should be preceded by a log message. Of course I may have missed something but I can't find it right now.

It might be useful to see pacman.conf and the mirrorlist on both systems. It might also be useful if the system clocks were synchronized. It shouldn't have any effect on pacserve itself, but it would make comparisons of the logs clearer (e.g. the multicast detection occurs at 55:16 on the first system but at 57:31 on the second).

It does look like a timeout (which could be a firewall issue), but it should default to querying external mirrors. It's obviously detected them because it correctly redirected the database requests.

I've added more information to the "querying" message so see exactly what it's querying. Update to the latest version and post the new log. Please include the corresponding pacman output too.

You also mentioned that you have been using pacserve for a while without trying multicast. I assume that means that you had no issues without multicast. Can you test your old settings and confirm that they still work as expected?

I intend to rewrite pacserve, quickserve, etc in Python 3 relatively soon, and I will clean up the code when I do. If all else fails right now then you may just have to wait for that.

Re: pacserve - easily share Pacman packages between computers

I have absolutely no firewall on any of the two machines, and they communicate through a simple switch (which is also connected to a router that provides internet access, but that shouldn't intervene)

800% CPU usage is what conky has been reporting me (htop showed that 4 python processes were actively eating CPU, and it's a dual core system -- don't know how else to explain that weird 800%)

I just tested with the old setup (archminux [server] running "pacserve" without argument, and client having "Server = http://archminux:15678/request/$repo/$arch" as the first entry in the mirrorlist), it works fine when installing a package on the client.

So, first experiment: launching pacserve on both machines with --multicast, then trying, from the laptop, to pacman -S a brand new package that wasn't in any of the caches. Pacman hangs with ":: Retrieving packages from community..."http://paste.pocoo.org/show/441403/ -- log from pacserve (archminux aka server)http://paste.pocoo.org/show/441404/ -- log from pacserve (tuxmachine aka laptop)

The weird thing is that while pacman was stuck, there was something eating up download bandwidth on the laptop (iftop told me the traffic was coming from mir1.archlinux.fr, my mirror). It is weird because I don't see what it could be downloading (there isn't any partially-downloaded package in pacman's cache, for instance, and there was nobody reporting me that it was downloading -- remember, pacman was stuck).

Second experiment, with new instances of "pacserve --multicast". From the laptop, pacman -S a package that is only in the laptop's cache (the server has never seen that package). Pacman does not request anything to pacserve, it uses the local cache directly. So far, so good.

Re: pacserve - easily share Pacman packages between computers

I still don't know what's causing this but I've found and fixed a bug that might be related. Basically, I had an unintentional catch-all case that was querying mirrors when it shouldn't, and returning unexpected data to other pacservers.

I've pushed the updated package to the repo on my site (but not the AUR yet). Please test it and let me know if it has any effect on the problem.

Re: pacserve - easily share Pacman packages between computers

@zorunThanks, I had forgotten about that.

Usually I do update the AUR at the same time as my repo, but when I make experimental changes I prefer to wait a few days to see if any issues arise. I'll eventually create a testing repo at which point I'll keep the normal repo and the AUR in sync automatically.

Re: pacserve - easily share Pacman packages between computers

Re: pacserve - easily share Pacman packages between computers

Hi Xyne and first of all thank you for your great work.

I'm trying to setup pacserve on a small home server that would mirror packages for my home laptops.There's no multicast in my setup as only the small server is intended to be the mirror for the laptops, no cross queries.ICould you please clarify something for me : In the case where a laptop doesn't find a package in its local cache or on the home server, will it download it directly from the internet or will pacserve add it to the home server and THEN push it to the laptop ?

Second point, I'm really missing the daemon feature here, I tried to code a dirty init script but it doesn't seem to work (I'm really poor at bash scripting) :

Re: pacserve - easily share Pacman packages between computers

@Lone_GunmanThe laptop will download the package directly in that case. I've considered supporting a central repository model as you describe but that isn't really the purpose of Pacserve and it would add some complexity (the central server would then need to be architecture and repository aware, additional cache management would be necessary to prevent "pacman -Sc" on the central server from removing "active" packages, there would need to be checks to prevent malicious or unintentional requests from filling up the central server's disk space, etc).

The idea really is to let all of the systems on the network swap packages with each other. You can still do that without multicast by making it the master and then having the laptops register as slaves. In that case, all queries would go through the central server. If a package is found then it would however redirect the requesting laptop to the laptop with the package.

The idea of a central server has crossed my mind a few times so I will give it some more thought when I have time. The daemon is also on my todo list but I'm just too busy right now to work on it.

Why do you not want to allow package swapping between the laptops? Do you have untrusted laptops on your network?