Re: powerpill reborn: pacman wrapper for faster downloads

Whenever I attempt to use yaourt/powerpill together, I get the error in the below output. However, if I use yaourt by itself or powerpill by itself, there are no problems. Has anyone ran into this and know how to fix it? I'm currently using the default configs, aside from changing the pacman binary to powerpill in /etc/yaourtrc. Maybe I'm doing something wrong?

Re: powerpill reborn: pacman wrapper for faster downloads

Re: powerpill reborn: pacman wrapper for faster downloads

Hi Xyne,I'm glad of your Powerpill, it's the best that I found.The only annoyance is the way aria2c logging on the console, excuse my paranoia.I wrote a python3 script that I filter the aria2 output by setting aria2 as daemon and querying it for results, via rpc calls.I kill aria2 after use, but someone would like to keep it alive, which will take some correction

Now I'd like to make a class/function for that filtering, but I'm rather confused about which implementation to use.Maybe it is easy to add as verbosity option in Powerpill.

URL= re.compile('.*(ht|f)tp://.*',re.IGNORECASE)
#Some wrong indentation, just posted as example to show what it would
#be loaded into a dictionary.
for aria_single in aria_multi:
#aria_single is a list of uris
for pkg in aria_single:
# any error it will stop and track it down
if not URL.match(pkg):
sys.exit(EOL+ 'Getting trouble to guess the url >'+ pkg)
#options is a dict that is like CLI options without hyphen(s)
# like: DIR = '/some/path/dir'
# options={'dir': DIR, 'max-overall-download-limit': max_speed}
DL_ID= aria2.addUri(aria_single, options)
pkg_D[DL_ID]=''; before = time.time()

Re: powerpill reborn: pacman wrapper for faster downloads

This is not something that I would want to include directly in Powerpill because I do not want to have to update the code if/when aria2c output changes. I think the best way to do this would be to write a wrapper around aria2c that accepts all of the same command-line arguments. You can then set your wrapper as the aria2c executable in /etc/powerpill/powerpill.json. You can even pass in custom arguments to your script in that case via the "args" list of the "aria2" section of that file.

Incidentally, I recommend using argument lists for command invocation instead of single strings such as

Re: powerpill reborn: pacman wrapper for faster downloads

Xyne wrote:

This is not something that I would want to include directly in Powerpill because I do not want to have to update the code if/when aria2c output changes

I'm using this kind of program across 3 or 4 aria2 upgrades, and I found just one issue around aria2 version 1.2.x. However that's the risk when calling third party programs.Clearly the wrapper would be in a separate module, callable by a CLI option. Failure may drop the option instantly.Basically I would just share some idea, if you feel uncomfortable with that, I'll try it on my own, but I need to know more things to do so. Then at completion one can fill free to try it out.

Xyne wrote:

I think the best way to do this would be to write a wrapper around aria2c that accepts all of the same command line arguments. You can then set your wrapper as the aria2c executable in /etc/powerpill/powerpill.json.

The manual express that we can pass almost all the commands via rpc calls like those command line arguments (CLAs), except no hyphens as separators and we may modify some option on the fly, while aria2 is running. Options are some kind of JSON-RPC/XML-RPC structure build. What I shown here are just a figure how it would do the filter. Furthermore I mentioned snippets, which means a long away to be working.I thought to implement the aria2 wrapper as small server or pipe, see multiprocessing documentations. I'm completely stuck to find how to write the functions.

Xyne wrote:

You can even pass in custom arguments to your script in that case via the "args" list of the "aria2" section of that file.

I read and modified that file. Just I wonder why it is send as CLA. Even an aria2 configuration file should be possible in the same manner, as per man page

Re: powerpill reborn: pacman wrapper for faster downloads

Sorry, after posting another long post in another thread, I'm not in the mood for selective quoting.

internal vs external invocationGiven that the aria2 executable is called at most twice in a given powerpill invocation and that the download itself is the bottleneck by several orders of magnitude, there really isn't much of a reason to include this in the powerpill codebase itself. A sensible wrapper, especially one that uses an aria2 daemon via the rpc interface, would have general utility beyond powerpill anyway, so I think it's a good idea to make a separate, stand-alone wrapper.

CLA vs RPC interface vs configuration fileCLA allows powepill to easily add more arguments as necessary for different invocations. There is no inherent risk in this vs calls to an RPC interface of passing a configuration file. The RPC interface would require a daemon to be running though, which most users neither need nor want, and using a configuration file would require extra disk IO, or it would be passed in via STDIN with the subprocess call, which ultimately is not much different than using CLAs to begin with.

subprocess.getstatusoutput()The call is deprecated and should no longer be used (it's in the "17.5.5. Legacy Shell Invocation Functions" section of the current official documentation), in part because of what I said about the need for shell escapes. Use subprocess.Popen and the "communicate" method instead, or some other supported method from the subprocess module. As for the risk, if you are controlling the arguments via your program then there is no inherent risk. Besides, any user who can run that command locally can run any malicious commands directly themselves.

Re: powerpill reborn: pacman wrapper for faster downloads

OK, final.

aria2 as daemon, it's one's taste. If I'll get the wrapper done, i think it won't matter how many calls are sent to aria2.Mostly I'll setup a class which can talk in "parallel" to aria2 (multiprocessing.Managers). The overhead of setting up a new deamon it won't harm much the system (as aria2 the aria2 author mention) and at the powerpill completion it would be easy to tell aria2 to shutdown.

CLAs, Your preference having the most priority, for powerpill. I just pose an idea, but as you said would make too muck impact on powerpill.

getstatusoutput(), Maybe you can help me on a couple of things about pyalpm, because I'd like to remove these calls from my program, as far as it concern with pacman. I'm choosing pyalpm. The aria2 startup, I think I will put into bash script follow by my program or change the subprocess call, as you suggest.For these points, you may PM to me, if you like.

Re: powerpill reborn: pacman wrapper for faster downloads

TheSaint wrote:

aria2 as daemon, it's one's taste. If I'll get the wrapper done, i think it won't matter how many calls are sent to aria2.Mostly I'll setup a class which can talk in "parallel" to aria2 (multiprocessing.Managers). The overhead of setting up a new deamon it won't harm much the system (as aria2 the aria2 author mention) and at the powerpill completion it would be easy to tell aria2 to shutdown.

I don't think there is any point in running a daemon for a single operation. If you write your wrapper then you should have the daemon running all the time for it to make sense. The daemon would run in the background and the script would add files to it's input queue. Personally I think the greatest utility of this would be for those that use aria2c to manage torrents as it will always be running then.

TheSaint wrote:

getstatusoutput(), Maybe you can help me on a couple of things about pyalpm, because I'd like to remove these calls from my program, as far as it concern with pacman. I'm choosing pyalpm. The aria2 startup, I think I will put into bash script follow by my program or change the subprocess call, as you suggest.For these points, you may PM to me, if you like.

I don't understand. Do you want me to PM you so that we can discuss pyalpm?

Re: powerpill reborn: pacman wrapper for faster downloads

Aria2 as daemon is like any other program in multitasking as you put threading in reflector. In a particular case, it's possible to instruct aria2 to quit when the parent process disappear.Have it background or foreground in both case will do what we want aria2 to do. For whom using it as long term jobs, just leave it alive, else we can tell go away when the jobs are finish, like it does with powerpill.In deep it's nothing wrong on your great work, the problem is from aria2. Actually I stop all its output, but it's hard to guess at which point is the process.

If you fill fine, I'll wait for your PM. What I'm studying now, is to get a list from pyalpm, like it does "pacman -Sup --print-format %r %l %s" (%l or %n).

Re: powerpill reborn: pacman wrapper for faster downloads

zsync author wrote:

zsync is only useful if people offer zsync downloads

Until mirrors start offering zsync downloads there is not much point in trying to support it. When the time comes, I will try to implement some sort of (pseudo-)plugin system for supporting third-party downloaders.