Note: If things are not working as described below check the version information for parallel.

parallel --version
WARNING: YOU ARE USING --tollef. IF THINGS ARE ACTING WEIRD USE --gnu.
...

Either add --gnu to all your parallel commands, or edit /etc/parallel/config and make sure it lists --gnu and not --tollef.

Most Linux utilities are single threaded, and when dealing with really large files a single core can be a severe bottleneck. One way to work around this limitation is the GNU parallel command, which I'm still playing around with and finding uses for -- but looks like a promising way to speed up a lot of things I do on a daily basis.

A common time sink with big computers is big logs, and waiting forever to grep them.

Let's break down the command a bit. We cat the contents of the file to the parallel command, which we give the --pipe option which splits STDIN to multiple copies of the command we supplied. (--pipe is the same as --spreadstdin, which is more readable but longer)

But you'll notice is actually took longer, and it chewed up a lot more CPU. What's going on here? By default parallel splits the input into 1 megabyte blocks and hands each to a single instance of the command, with the number of simultaneous instances set to the number of cores by default. Launching grep 102 times to grep 1 meg is pretty wasteful. Let's tune things a little bit:

Another common command to throw large files at is awk. However, depending on what you’re trying to accomplish splitting the input may prevent you from accomplishing certain tasks — like adding all the values of a field. To work around this you can rewrite your commands to perform the same task in two steps. The first step to process blocks of the input, the second step to combine the results.

The caveat being that because it is parallel-izing these commands sometimes the results aren't as reproducible as they would inherently be on a single core. You've got to be really careful with the options to parallel, and the assumptions you make when assembling your commands. When in doubt use the -k (--keep-order) to return the output in the same order as the input, as opposed to as soon as the command running on a particular block of input finishes.

In the above command set delimiter is set to spaces to split between each of the input numbers, in the first one output order is maintained (-k), and limited to 4 cores (-j4). The input numbers are inserted in place of the {} in the sleep\; echo {} command.

By maintaining output order parallel can be used to bzip2 files as well: