parallel_design man page

This document describes design decisions made in the development of GNU parallel and the reasoning behind them. It will give an overview of why some of the code looks the way it does, and will help new maintainers understand the code better.

GNU parallel is a Perl script in a single file. It is object oriented, but contrary to normal Perl scripts each class is not in its own file. This is due to user experience: The goal is that in a pinch the user will be able to get GNU parallel working simply by copying a single file: No need to mess around with environment variables like PERL5LIB.

GNU parallel busy waits. This is because the reason why a job is not started may be due to load average (when using --load), and thus it will not make sense to wait for a job to finish. Instead the load average must be checked again. Load average is not the only reason: --timeout has a similar problem.

To not burn up too much CPU GNU parallel sleeps exponentially longer and longer if nothing happens, maxing out at 1 second.

env_parallel is a dummy shell script that will run if env_parallel is not an alias or a function and tell the user how to activate the alias/function for the supported shells.

The alias or function will copy the current environment and run the command with GNU parallel in the copy of the environment.

The problem is that you cannot access all of the current environment inside Perl. E.g. aliases, functions and unexported shell variables.

The idea is therefore to take the environment and put it in $PARALLEL_ENV which GNU parallel prepends to every command.

The only way to have access to the environment is directly from the shell, so the program must be written in a shell script that will be sourced and there has to deal with the dialect of the relevant shell.

env_parallel.*

These are the files that implements the alias or function env_parallel for a given shell. It could be argued that these should be put in some obscure place under /usr/lib, but by putting them in your path it becomes trivial to find the path to them and source them:

source `which env_parallel.foo`

The beauty is that they can be put anywhere in the path without the user having to know the location. So if the user's path includes /afs/bin/i386_fc5 or /usr/pkg/parallel/bin or /usr/local/parallel/20161222/sunos5.6/bin the files can be put in the dir that makes most sense for the sysadmin.

parset is a shell function. This is the reason why parset can set variables: It runs in the shell which is calling it.

It is also the reason why parset does not work, when data is piped into it: ... | parset ... makes parset start in a subshell, and any changes in environment can therefore not make it back to the calling shell.

The easiest way to explain what GNU parallel does is to assume that there are a number of job slots, and when a slot becomes available a job from the queue will be run in that slot. But originally GNU parallel did not model job slots in the code. Job slots have been added to make it possible to use {%} as a replacement string.

While the job sequence number can be computed in advance, the job slot can only be computed the moment a slot becomes available. So it has been implemented as a stack with lazy evaluation: Draw one from an empty stack and the stack is extended by one. When a job is done, push the available job slot back on the stack.

This implementation also means that if you re-run the same jobs, you cannot assume jobs will get the same slots. And if you use remote executions, you cannot assume that a given job slot will remain on the same remote server. This goes double since number of job slots can be adjusted on the fly (by giving --jobs a file name).

rsync 3.1.x uses protocol 31 which is unsupported by version 2.5.7. That means that you cannot push a file to a remote system using rsync protocol 31, if the remote system uses 2.5.7. rsync does not automatically downgrade to protocol 30.

GNU parallel does not require protocol 31, so if the rsync version is >= 3.1.0 then --protocol 30 is added to force newer rsyncs to talk to version 2.5.7.

GNU parallel buffers output in temporary files. --compress compresses the buffered data. This is a bit tricky because there should be no files to clean up if GNU parallel is killed by a power outage.

GNU parallel first selects a compression program. If the user has not selected one, the first of these that is in $PATH is used: pzstd lbzip2 pbzip2 zstd pixz lz4 pigz lzop plzip lzip gzip lrz pxz bzip2 lzma xz clzip. They are sorted by speed on a 128 core machine.

The setup is duplicated for both standard output (stdout) and standard error (stderr).

GNU parallel pipes output from the command run into the compression program which saves to a tmpfile. GNU parallel records the pid of the compress program. At the same time a small perl script (called cattail above) is started: It basically does cat followed by tail -f, but it also removes the tmpfile as soon as the first byte is read, and it continously checks if the pid of the compression program is dead. If the compress program is dead, cattail reads the rest of tmpfile and exits.

As most compression programs write out a header when they start, the tmpfile in practice is removed by cattail after around 40 ms.

{} is set to $PARALLEL_TMP which is a tmpfile. The Perl script saves the exit value, unlinks the tmpfile, and returns the exit value - no matter if the shell is bash/ksh/zsh (using $?) or *csh/fish (using $status).

First a FIFO is made (.tmx). It is used for communicating exit value. Next a new tmux session is made. This may fail if there is already a session, so the output is ignored. If all job slots finish at the same time, then tmux will close the session. A temporary socket is made (.tms) to avoid a race condition in tmux. It is cleaned up when GNU parallel finishes.

The input is used as the name of the windows in tmux. When the job inside tmux finishes, the exit value is printed to the FIFO (.tmx). This FIFO is opened by perl outside tmux, and perl then removes the FIFO. Perl blocks until the first value is read from the FIFO, and this value is used as exit value.

To make it compatible with csh and bash the exit value is printed as: $?h/$status and this is parsed by perl.

There is a bug that makes it necessary to print the exit value 3 times.

Another bug in tmux requires the length of the tmux title and command to not have certain limits. When inside these limits, 75 '\ ' are added to the title to force it to be outside the limits.

It is not impossible, but by using --nice GNU parallel will do the right thing for you. Similarly when transferring files: It starts to get hard when the file names contain space, :, `, *, or other special characters.

To run the commands in a tmux session you basically just need to quote the command. For simple commands that is easy, but when commands contain special characters, it gets much harder to get right.

--compress not only compresses standard output (stdout) but also standard error (stderr); and it does so into files, that are open but deleted, so a crash will not leave these files around.

--cat and --fifo are easy to do by hand, until you want to clean up the tmpfile and keep the exit code of the command.

The real killer comes when you try to combine several of these: Doing that correctly for all corner cases is next to impossible to do by hand.

The shell shock bug in bash did not affect GNU parallel, but the solutions did. bash first introduced functions in variables named: BASH_FUNC_myfunc() and later changed that to BASH_FUNC_myfunc%%. When transferring functions GNU parallel reads off the function and changes that into a function definition, which is copied to the remote system and executed before the actual command is executed. Therefore GNU parallel needs to know how to read the function.

From version 20150122 GNU parallel tries both the ()-version and the %%-version, and the function definition works on both pre- and post-shellshock versions of bash.

The remote system wrapper does some initialization before starting the command on the remote system.

Ctrl-C and standard error (stderr)

If the user presses Ctrl-C the user expects jobs to stop. This works out of the box if the jobs are run locally. Unfortunately it is not so simple if the jobs are run remotely.

If remote jobs are run in a tty using ssh -tt, then Ctrl-C works, but all output to standard error (stderr) is sent to standard output (stdout). This is not what the user expects.

If remote jobs are run without a tty using ssh (without -tt), then output to standard error (stderr) is kept on stderr, but Ctrl-C does not kill remote jobs. This is not what the user expects.

So what is needed is a way to have both. It seems the reason why Ctrl-C does not kill the remote jobs is because the shell does not propagate the hang-up signal from sshd. But when sshd dies, the parent of the login shell becomes init (process id 1). So by exec'ing a Perl wrapper to monitor the parent pid and kill the child if the parent pid becomes 1, then Ctrl-C works and stderr is kept on stderr.

To be able to kill all (grand)*children a new process group is started.

Transferring of variables and functions given by --env is done by running a Perl script remotely that calls the actual command. The Perl script sets $ENV{variable} to the correct value before exec'ing a shell that runs the function definition followed by the actual command.

The function env_parallel copies the full current environment into the environment variable PARALLEL_ENV. This variable is picked up by GNU parallel and used to create the Perl script mentioned above.

csh limits words of commands to 1024 chars. This is often too little when GNU parallel encodes environment variables and wraps the command with different templates. All of these are combined and quoted into one single word, which often is longer than 1024 chars.

When the line to run is > 1000 chars, GNU parallel therefore encodes the line to run. The encoding bzip2s the line to run, converts this to base64, splits the base64 into 1000 char blocks (so csh does not fail), and prepends it with this Perl script that decodes, decompresses and evals the line.

Different shells behave differently. A command that works in tcsh may not work in bash. It is therefore important that the correct shell is used when GNU parallel executes commands.

GNU parallel tries hard to use the right shell. If GNU parallel is called from tcsh it will use tcsh. If it is called from bash it will use bash. It does this by looking at the (grand)*parent process: If the (grand)*parent process is a shell, use this shell; otherwise look at the parent of this (grand)*parent. If none of the (grand)*parents are shells, then $SHELL is used.

This will do the right thing if called from:

an interactive shell

a shell script

a Perl script in `` or using system if called as a single string.

While these cover most cases, there are situations where it will fail:

When run using exec.

When run as the last command using -c from another shell (because some shells use exec):

When run in a Perl script using system with parallel as the first string:

#!/usr/bin/perl
system("parallel",'setenv a {}; echo $a',":::",2);

Here it depends on which shell is used to call the Perl script. If the Perl script is called from tcsh it will work just fine, but if it is called from bash it will fail, because the command setenv is not known to bash.

If GNU parallel guesses wrong in these situation, set the shell using $PARALLEL_SHELL.

Quoting depends on the shell. For most shells \ is used for all special chars and ' is used for newline. Whether a char is special depends on the shell and the context. Luckily quoting a bit too many chars does not break things.

It is fast, but has the distinct disadvantage that if a string needs to be quoted multiple times, the \'s double every time - increasing the string length exponentially.

While --pipe and --pipepart look much the same to the user, they are implemented very differently.

With --pipe GNU parallel reads the blocks from standard input (stdin), which is then given to the command on standard input (stdin); so every block is being processed by GNU parallel itself. This is the reason why --pipe maxes out at around 500 MB/sec.

--pipepart, on the other hand, first identifies at which byte positions blocks start and how long they are. It does that by seeking into the file by the size of a block and then reading until it meets end of a block. The seeking explains why GNU parallel does not know the line number and why -L/-l and -N do not work.

With a reasonable block and file size this seeking is more than 1000 time faster than reading the full file. The byte positions are then given to a small script that reads from position X to Y and sends output to standard output (stdout). This small script is prepended to the command and the full command is executed just as if GNU parallel had been in its normal mode. The script looks like this:

Every time GNU parallel detects a record bigger than --block-size it increases the block size by 30%. A small --block-size gives very poor performance; by exponentially increasing the block size performance will not suffer.

GNU parallel will waste CPU power if --block-size does not contain a full record, because it tries to find a full record and will fail to do so. The recommendation is therefore to use a --block-size > 2 records, so you always get at least one full record when you read one block.

If you use -N then --block-size should be big enough to contain N+1 records.

With --pipepart GNU parallel can compute the --block-size automatically. A --block-size of -1 will use a block size so that each jobslot will receive approximately 1 block. --block -2 will pass 2 blocks to each jobslot and -n will pass n blocks to each jobslot.

This can be done because --pipepart reads from files, and we can compute the total size of the input.

When running the same commands on many servers what should --jobs signify? Is it the number of servers to run on in parallel? Is it the number of jobs run in parallel on each server?

GNU parallel lets --jobs represent the number of servers to run on in parallel. This is to make it possible to run a sequence of commands (that cannot be parallelized) on each server, but run the same sequence on multiple servers.

When using --shuf to shuffle the jobs, all jobs are read, then they are shuffled, and finally executed. When using SQL this makes the --sqlmaster be the part that shuffles the jobs. The --sqlworkers simply executes according to Seq number.

GNU parallel buffers output, because if output is not buffered you have to be ridiculously careful on sizes to avoid mixing of outputs (see excellent example on https://catern.com/posts/pipes.html).

GNU parallel buffers on disk in $TMPDIR using files, that are removed as soon as they are created, but which are kept open. So even if GNU parallel is killed by a power outage, there will be no files to clean up afterwards. Another advantage is that the file system is aware that these files will be lost in case of a crash, so it does not need to sync them to disk.

It gives the odd situation that a disk can be fully used, but there are no visible files on it.

Partly buffering in memory

When using output formats SQL and CSV then GNU Parallel has to read the whole output into memory. When run normally it will only read the output from a single job. But when using --linebuffer every line printed will also be buffered in memory - for all jobs currently running.

If memory is tight, then do not use the output format SQL/CSV with --linebuffer.

Comparing to buffering in memory

gargs is a parallelizing tool that buffers in memory. It is therefore a useful way of comparing the advantages and disadvantages of buffering in memory to buffering on disk.

On an system with 6 GB RAM free and 6 GB free swap these were tested with different sizes:

GNU parallel is pretty much limited by the speed of the disk: Up to 6 GB data is written to disk but cached, so reading is fast. Above 6 GB data are both written and read from disk. When the 30000MB job is running, the disk system is slow, but usable: If you are not using the disk, you almost do not feel it.

gargs has a speed advantage up until 2500M where it hits a wall. Then the system starts swapping like crazy and is completely unusable. At 5000M it goes out of memory.

You can make GNU parallel behave similar to gargs if you point $TMPDIR to a tmpfs-filesystem: It will be faster for small outputs, but may kill your system for larger outputs and cause you to lose output.

GNU parallel buffers on disk. If the disk is full, data may be lost. To check if the disk is full GNU parallel writes a 8193 byte file every second. If this file is written successfully, it is removed immediately. If it is not written successfully, the disk is full. The size 8193 was chosen because 8192 gave wrong result on some file systems, whereas 8193 did the correct thing on all tested filesystems.

Normally GNU parallel will use around 17 MB RAM constantly - no matter how many jobs or how much output there is. There are a few things that cause the memory usage to rise:

Multiple input sources. GNU parallel reads an input source only once. This is by design, as an input source can be a stream (e.g. FIFO, pipe, standard input (stdin)) which cannot be rewound and read again. When reading a single input source, the memory is freed as soon as the job is done - thus keeping the memory usage constant.

But when reading multiple input sources GNU parallel keeps the already read values for generating all combinations with other input sources.

Computing the number of jobs. --bar, --eta, and --halt xx% use total_jobs() to compute the total number of jobs. It does this by generating the data structures for all jobs. All these job data structures will be stored in memory and take up around 400 bytes/job.

Buffering a full line. --linebuffer will read a full line per running job. A very long output line (say 1 GB without \n) will increase RAM usage temporarily: From when the beginning of the line is read till the line is printed.

Buffering the full output of a single job. This happens when using --results *.csv/*.tsv or --sql*. Here GNU parallel will read the whole output of a single job and save it as csv/tsv or SQL.

The shorthands for replacement strings make a command look more cryptic. Different users will need different replacement strings. Instead of inventing more shorthands you get more flexible replacement strings if they can be programmed by the user.

The language Perl was chosen because GNU parallel is written in Perl and it was easy and reasonably fast to run the code given by the user.

If a user needs the same programmed replacement string again and again, the user may want to make his own shorthand for it. This is what --rpl is for. It works so well, that even GNU parallel's own shorthands are implemented using --rpl.

In Perl code the bigrams {= and =} rarely exist. They look like a matching pair and can be entered on all keyboards. This made them good candidates for enclosing the Perl expression in the replacement strings. Another candidate ,, and ,, was rejected because they do not look like a matching pair. --parens was made, so that the users can still use ,, and ,, if they like: --parens ,,,,

Internally, however, the {= and =} are replaced by \257< and \257>. This is to make it simpler to make regular expressions. You only need to look one character ahead, and never have to look behind.

GNU parallel uses its own testing framework. This is mostly due to historical reasons. It deals reasonably well with tests that are dependent on how long a given test runs (e.g. more than 10 secs is a pass, but less is a fail). It parallelizes most tests, but it is easy to force a test to run as the single test (which may be important for timing issues). It deals reasonably well with tests that fail intermittently. It detects which tests failed and pushes these to the top, so when running the test suite again, the tests that failed most recently are run first.

If GNU parallel should adopt a real testing framework then those elements would be important.

Since many tests are dependent on which hardware it is running on, these tests break when run on a different hardware than what the test was written for.

When most bugs are fixed a test is added, so this bug will not reappear. It is, however, sometimes hard to create the environment in which the bug shows up - especially if the bug only shows up sometimes. One of the harder problems was to make a machine start swapping without forcing it to its knees.

Using a percentage for --timeout causes GNU parallel to compute the median run time of a job. The median is a better indicator of the expected run time than average, because there will often be outliers taking way longer than the normal run time.

To avoid keeping all run times in memory, an implementation of remedian was made (Rousseeuw et al).

Contrary to the obvious --load does not use load average. This is due to load average rising too slowly. Instead it uses ps to list the number of threads in running or blocked state (state D, O or R). This gives an instant load.

As remote calculation of load can be slow, a process is spawned to run ps and put the result in a file, which is then used next time.

GNU parallel kills jobs. It can be due to --memfree, --halt, or when GNU parallel meets a condition from which it cannot recover. Every job is started as its own process group. This way any (grand)*children will get killed, too. The process group is killed with the specification mentioned in --termseq.

GNU parallel uses the DBURL from GNU sql to give database software, username, password, host, port, database, and table in a single string.

The DBURL must point to a table name. The table will be dropped and created. The reason for not reusing an exising table is that the user may have added more input sources which would require more columns in the table. By prepending '+' to the DBURL the table will not be dropped.

The table columns are similar to joblog with the addition of V1 .. Vn which are values from the input sources, and Stdout and Stderr which are the output from standard output and standard error, respectively.

The Signal column has been renamed to _Signal due to Signal being a reserved word in MySQL.

Funding a free software project is hard. GNU parallel is no exception. On top of that it seems the less visible a project is, the harder it is to get funding. And the nature of GNU parallel is that it will never be seen by "the guy with the checkbook", but only by the people doing the actual work.

There is no doubt that this is not an ideal solution, but no one has so far come up with an ideal solution - neither for funding GNU parallel nor other free software.

If you believe you have the perfect solution, you should try it out, and if it works, you should post it on the email list. Ideas that will cost work and which have not been tested are, however, unlikely to be prioritized.

--tollef was included to make GNU parallel switch compatible with the parallel from moreutils (which is made by Tollef Fog Heen). This was done so that users of that parallel easily could port their use to GNU parallel: Simply set PARALLEL="--tollef" and that would be it.

But several distributions chose to make --tollef global (by putting it into /etc/parallel/config) without making the users aware of this, and that caused much confusion when people tried out the examples from GNU parallel's man page and these did not work. The users became frustrated because the distribution did not make it clear to them that it has made --tollef global.

So to lessen the frustration and the resulting support, --tollef was obsoleted 20130222 and removed one year later.

Until 20150122 variables and functions were transferred by looking at $SHELL to see whether the shell was a *csh shell. If so the variables would be set using setenv. Otherwise they would be set using =. This caused the content of the variable to be repeated: