I have heard that pssh and clusterssh are two popular ones, but I thought I would open it to discussion here and see what the community's experiences with these tools were? What are the gotchas? Any decent hacks or use cases?

I should also add- depending on what you're doing there could be many gotchyas using tools like pssh and clusterssh. eg- editing or deleting files. very easy to make a mistake. As a general rule of thumb I only use pssh for status queries. For anything more complex I use puppet because it lets me test out changes in my testing environment before sending it out to production servers.
–
user7321Jun 2 '09 at 17:28

Sounds like a good best practice. That was exactly what I was looking for thanks!
–
faultyserverJun 2 '09 at 17:37

1

For something that offers the best of both worlds you may want to take a look at salt (saltstack.org) it can do full on configuration management but will also allow you to issue a command to multiple computers at once.
–
kaptk2Jan 4 '12 at 2:25

My favorite parallel SSH tool is PuSSH ... IMO the syntax is more intuitive than any other similar packages available, and it doesn't actually need installation as an RPM, compilation or anything else. It just downloads as a couple of scripts, and just works.

Its important to know what you want to do. If you want to run 'apt-get update' on lots of servers, clusterssh is reportedly an easy-to use and effective tool. See Kyle Rankin's article on clusterssh at the Linux Journal.

If your work on different servers will require logic loops, putting, getting or otherwise interacting with each host (or possibly having to interact with a host because of an error condition) a tool such as Fabric (which has just hit its 1.0 release) is invaluable.
You'll probably enjoy most using Fabric if you are a pythonista.

In either case one is probably expecting to do fairly simple tasks on the remote servers affected, and dropping in specially on any problem hosts. However Fabric provides more options for dealing with variation.

One thing that is generally not handled well is dealing with multiple hosts behind a gateway host, requiring one to connect first to the gateway and then port forwarding from there. As ssh itself does this quite simply, and one can set one's .ssh/config file to something like

In order to do the equivalent of ssh dc1mail 'cmd' && ssh dc1phones 'cmd' one has to first bring up ssh gateway. It would be nice if tools like clusterssh and Fabric would permit this to be done as part of their tools. Naturally, you can do this yourself by hand.

The SSH Power Tool (sshpt) was designed for parallel SSH without requiring that the user setup pre-shared SSH keys. It is superior to pssh and clusterssh in that it supports executions via sudo and can also copy files and execute them afterwards (optionally, via sudo as well). By default it outputs results in CSV format but sshpt.py doubles as an importable Python module so you can use it in your own programs (I used to use it as a back-end behind a a custom-built web-based reporting tool at my former employer).

Mussh is a good alternative, it is already included in many Linux distros.

Mussh is a shell script that allows you to execute a command or
script over ssh on multiple hosts with
one command. When possible mussh will
use ssh-agent and RSA/DSA keys to
minimize the need to enter your
password more than once.

I use cssh and it works like a charm. You can edit the same file on several servers easily and then check only one window and change for instance value of particular variable. Repeat that for every window, hit :wq in vim and voila! Work is done ;)