Starting from version 1.2.0, a new client is supported in parallel-ssh which offers much greater performance and reduced overhead than the current default client.

The new client is based on libssh2 via the ssh2-python extension library and supports non-blocking mode natively. Binary wheel packages with libssh2 included are provided for Linux, OSX and Windows platforms and all supported Python versions.

Once either standard output is iterated on to completion, or client.join(output) is called, exit codes become available in host output. Iteration ends only when remote command has completed, though it may be interrupted and resumed at any point.

forhostinoutput:print(output[host].exit_code)

Output:

00

The client’s join function can be used to wait for all commands in output object to finish:

client.join(output)

Similarly, output and exit codes are available after client.join is called:

parallel-ssh’s design goals and motivation are to provide a library for running non-blocking asynchronous SSH commands in parallel with little to no load induced on the system by doing so with the intended usage being completely programmatic and non-interactive.

To meet these goals, API driven solutions are preferred first and foremost. This frees up developers to drive the library via any method desired, be that environment variables, CI driven tasks, command line tools, existing OpenSSH or new configuration files, from within an application et al.

There are not many alternatives for SSH libraries in Python. Of the few that do exist, here is how they compare with parallel-ssh.

As always, it is best to use a tool that is suited to the task at hand. parallel-ssh is a library for programmatic and non-interactive use - see Design And Goals. If requirements do not match what it provides then it best not be used. Same applies for the tools described below.

Pure Python code, while having native extensions as dependencies, with poor performance and numerous bugs compared to both OpenSSH binaries and the libssh2 based native clients in parallel-ssh1.2.x and above. Recent versions have regressed in performance and have blocker issues.

It does not support non-blocking mode, so to make it non-blocking monkey patching must be used which affects all other uses of the Python standard library. However, some functionality like Kerberos (GSS-API) authentication is not provided by other libraries.

Port of Capistrano from Ruby to Python. Intended for command line use and is heavily systems administration oriented rather than non-interactive library. Same maintainer as Paramiko.

Uses Paramiko and suffers from the same limitations. More over, uses threads for parallelisation, while not being thread safe, and exhibits very poor performance and extremely high CPU usage even for limited number of hosts - 1 to 10 - with scaling limited to one core.

Library API is non-standard, poorly documented and with numerous issues as API use is not intended.

Again similar to Fabric, its intended and documented use is interactive via command line rather than library API based. It may, however, be an option if Ansible is already being used for automation purposes with existing playbooks, the number of hosts is small, and when the use case is interactive via command line.

parallel-ssh is, on the other hand, a suitable option for Ansible as an SSH client that would improve its parallel SSH performance significantly.

Wrapper to libssh2 C library. Used by parallel-ssh as of 1.2.0 and is by same author.

Does not do parallelisation out of the box but can be made parallel via Python’s threading library relatively easily and as it is a wrapper to a native library that releases Python’s GIL, can scale to multiple cores.

parallel-ssh uses ssh2-python in its native non-blocking mode with event loop and co-operative sockets provided by gevent for an extremely high performance library without the side-effects of monkey patching - see benchmarks.

In addition, parallel-ssh uses native threads to offload CPU blocked tasks like authentication in order to scale to multiple cores while still remaining non-blocking for network I/O.

pssh.clients.native.SSHClient is a single host natively non-blocking client for users that do not need parallel capabilities but still want a non-blocking client with native code performance.

Out of all the available Python SSH libraries, libssh2 and ssh2-python have been shown, see benchmarks above, to perform the best with the least resource utilisation and ironically for a native code extension the least amount of dependencies. Only libssh2 C library and its dependencies which are included in binary wheels.

In general, long lived commands with little or no output gathering will scale better. Pool sizes in the multiple thousands have been used successfully with little CPU overhead in the single thread running them in these use cases.

Conversely, many short lived commands with output gathering will not scale as well. In this use case, smaller pool sizes in the hundreds are likely to perform better with regards to CPU overhead in the event loop.

Multiple Python native threads, each of which can get its own event loop, may be used to scale this use case further as number of CPU cores allows. Note that parallel-ssh imports must be done within the target function of the newly started thread for it to receive its own event loop. gevent.get_hub() may be used to confirm that the worker thread event loop differs from the main thread.

Gathering is highlighted here as output generation does not affect scaling. Only when output is gathered either over multiple still running commands, or while more commands are being triggered, is overhead increased.

To understand why this is, consider that in co-operative multi tasking, which is being used in this project via the gevent library, a co-routine (greenlet) needs to yield the event loop to allow others to execute - co-operation. When one co-routine is constantly grabbing the event loop in order to gather output, or when co-routines are constantly trying to start new short-lived commands, it causes contention with other co-routines that also want to use the event loop.

This manifests itself as increased CPU usage in the process running the event loop and reduced performance with regards to scaling improvements from increasing pool size.

On the other end of the spectrum, long lived remote commands that generate no output only need the event loop at the start, when they are establishing connections, and at the end, when they are finished and need to gather exit codes, which results in practically zero CPU overhead at any time other than start or end of command execution.

Output generation is done remotely and has no effect on the event loop until output is gathered - output buffers are iterated on. Only at that point does the event loop need to be held.