History of GNU Parallel

GNU parallel was originally two
tools: xxargs and parallel.

xxargs was a tool that had the most basic features
of xargs namely -0, -n, -x, and {}. In addition it
also had -1 which is called --halt-on-error 1 in GNU
parallel. xxargs was developed during 2002-2005 and
was based on work by Cameron Simpson. Cameron
sums up the reason for developing xxargs very well in
a posting from 2001-12-20:

Having pretty much had it with xargs, which is a busted piece of crap due
to its quoting/whitespace problems, here is a less featured but more robust one.
- Cameron Simpson <cs@zip.com.au>

parallel dates back to around the same time. It was
originally a wrapper that generated a makefile and used
make -j to do the parallelization. The only usage
was: cat commandlines | parallel -j number. It did
not support grouping of output or running jobs on multiple computers,
and the most common reason for bugs was quoting the string incorrectly
in the Makefile. It also stressed make's dependency
calculation: at this point make's initialization would
take O(n^2) time on n independant targets, which was a real problem if
you had 1000000 commandlines to run. Chopping the commandlines up into
blocks of 5000 lines minimized the problem.

In 2005 it occurred to me that I often used xxargs
and parallel together and I saw no reason why they
could not merge into a single tool. The single tool was
called parallel and the development
of xxargs stopped. At this point
both xxargs and parallel were used
in production.

GNU parallel thus did not have one but two main
objectives: replace xargs and run commands in
parallel.

In the years after 2005 I noticed that other people still had problems
with xargs' treatment of quotes and whitespace, so I
tried getting parallel accepted into GNU findutils.
It was not accepted as it was written in Perl and the team did not
want GNU findutils to depend on Perl.

In 2007 unittests were added to parallel to make sure
old problems would not creep back in while developing new features. The
hosting was moved to git hosted on savannah.nongnu.org.

In February 2009 I tried getting parallel added to
the package moreutils. The author never replied to the email or the
two reminders, but in June 2009 moreutils chose to add another program
called parallel. This choice leads to some confusion even today.

In 2010 parallel was adopted as an official GNU tool
and the name was changed to GNU parallel. As GNU
already had a tool for running jobs on remote computers
(called pexec) it was a hard decision to include GNU
parallel as well. I believe the decision was mostly
based on GNU parallel having a more familiar user
interface - behaving very much like xargs. Shortly
after the release as GNU tool remote execution was added and all
missing options from xargs were added to make it
possible to use GNU parallel as a drop in replacement
for xargs.