Run Command Parallel on Multiple Hosts using PDSH Tool

PDSH is a very smart little tool that enables you to issue the same command on multiple hosts at once, and see the output. It is a high-performance and parallel remote shell utility. It can run multiple remote commands in parallel and uses a "sliding window" (or fanout) of threads to conserve resources on the initiating host while allowing some connections to time out. In this article we explain how to install pdsh tool and show few examples.

Note :- Puts the binaries into /usr/local/, which is fine for testing purposes. For production work, I would put it in /opt or something like that – just be sure it's in your path.

You might notice that I used the --without-rsh option in the configure command. By default, pdsh uses rsh, which is not really secure, so I choose to exclude it from the configuration. To override rsh and make ssh the default, you just add the following line to your .bashrc file:

Be sure to "source" your .bashrc file (i.e., source .bashrc) to set the environment variable. You can also log out and log back in. If, for some reason, you see the following when you try running pdsh,

Requirment for executing pdsh command on multiple node

a) SSH keys

In this short series of blog posts I’m going to take a look at a few very useful tools that can make your life as the sysadmin of a cluster of Linux machines easier. This may be a Hadoop cluster, or just a plain simple set of ‘normal’ machines on which you want to run the same commands and monitoring.

To start with, we’re going to use the ever-awesome ssh keys to manage security on the cluster. After that we’ll look at executing the same command across multiple machines at the same time using PDSH.

In a nutshell, ssh keys enable us to do password-less authentication in a secure way. Working with SSH keys involves taking the public key from a pair, and adding that to another machine in order to allow the owner of the pair’s private key to access that machine. What we’re going to do here is generate a unique key pair that will be used as the identity across the cluster. So each node will have a copy of the private key, in order to be able to authenticate to any other node, which will be holding a copy of the public key (as well as, in turn, the same private key).

We’ve several ways we could implement the SSH keys. Because it’s a purely sandbox cluster, I could use the same SSH key pair that I generate for the cluster on my machine too, so the same public/private key pair is distributed thus:

Now we’ll prepare the authorized_keys file which is where the public SSH key of any identity permitted to access the machine is stored. Note that each user on a machine has their own authorized_keys file, in ~/.ssh/. So for example, the root user has the file in /root/.ssh/authorized_keys and any public key listed in that file will be able to connect to the server as the root user. Be aware the American [mis-]spelling of “authorized” – spell it [correctly] as “authorised” and you’ll not get any obvious errors, but the ssh key login won’t work either.

Distributing the SSH artifacts

Now we’re going to push this set of SSH files out to the .ssh folder of the target user on each node, which in this case is the root user. From a security point of view it’s probably better to use a non-root user for login and then sudo as required, but we’re keeping things simple (and less secure) to start with here. So the files in our folder are:

id_rsa – the private key of the key pair
id_rsa.pub – the public key of the key pair.

Strictly speaking this doesn’t need distributing to all nodes, but it’s conventional and handy to hold it alongside the private key.

authorized_keys – this is the file that the sshd daemon on each node will look at to validate an incoming login request’s offered private key, and so needs to hold the public key of anyone who is allowed to access the machine as this user.

At this point you’ll need to enter the password for the target user, but rejoice! This is the last time you’ll need to enter it as subsequent logins will be authenticated using the ssh keys that you’re now configuring.

The -w option means I am specifying the node(s) that will run the command. In this case, I specified the IP address of the node (ec2-52-59-121-138). After the list of nodes, I add the command I want to run, which is uname -r in this case. Notice that pdsh starts the output line by identifying the node name.

1) Single Node Command Execution

To combine commands together that you send to each host or single host you can use the standard bash operator semicolon ;

A very common way of using pdsh is to set the environment variable WCOLL to point to the file that contains the list of hosts you want to use in the pdsh command. For example, I created a subdirectory PDSH where I create a file hosts that lists the hosts I want to use:

I'm only using two nodes: ec2-52-58-254-227.eu-central-1.compute.amazonaws.com and ec2-52-59-121-138.eu-central-1.compute.amazonaws.com . The first is my test system (like a cluster head node), and the second is my test compute node. You can put hosts in the file as you would on the command line separated by commas. Be sure not to put a blank line at the end of the file because pdsh will try to connect to it. You can put the environment variable WCOLL in your .bashrc file:

export WCOLL=/home/shaha/PDSH/hosts

As before, you can source your .bashrc file, or you can log out and log back in.

3) Specifying Hosts Command Execution

I won't list all the several other ways to specify a list of nodes. The simplest way is to specify the nodes on the command line is to use the -w option:

Now I can shift into second gear and try some fancier pdsh tricks. First, I want to run a more complicated command on all of the nodes . Notice that I put the entire command in quotes. This means the entire command is run on each node, including the first (cat /proc/cpuinfo) and second (grep bogomips , model ,cpu) parts.

Note the use of the quotation marks to enclose the entire command string. Without them the bash interpreter will take the ; as the delimiter of the local commands, and try to run the subsequent commands locally:

Related Posts

The bad blocks in a storage device are the portions of the device that are not readable for some reason. These bad blocks can be recovered if we are able to reach the exact location of that block. The SMART [...]

In some occasions where you want to check by how many years someone older than you, how old you are (in days, years or months), the countdown to an event or the next flash sale. There is a python-based command [...]

Previously we learned how we can restrict or allow a particular country using GeoIP but in this article, we'll cover how we can block large IP ranges using ipset module with iptables. IPset is a command line based utility which [...]