Moreutils For Your Linux Shell Tool Set

I have this book -- a Spanish edition of Stephen Coffin's seminal manual, Unix System V Release 4: The Complete Reference. You can open it on any of its 700+ pages and bet your bottom dollar that the commands on the page will work in a modern-day Linux. Well, except where teletypes and tape storage are involved.

Said like that, you may think the *NIX command line hasn't changed a lot since the early 1990s. This is not entirely true.

Take for instance the moreutils collection. You can install it on most distros with your regular package manager. On Ubuntu and Ubuntu-based distros, do

Shiny new tools

Moreutils provides you with a set of new tools that are not part of the standard Linux set but probably should be. For example, vidir provides an editor interface for modifying files and directories. Despite its name, vidir uses the default command-line editor, so if you have vi(m), sure, it will show a vim interface (Figure 1); but if you have nano or emacs configured, it'll show nano or emacs interface (Figure 2).

You use your regular key combinations to modify your directories and files. If you're using a vi-like interface, press ito modify directory and file names; press d[number]d to delete files or directories -- note that vidir has a security feature built in, which won't let you erase non-empty directories; press u to undo changes, and so on.

Soaking it all up: Sponge

According to its manpage, sponge reads standard input and writes it out to the specified file. Unlike a shell redirect, sponge soaks up all its input before writing the output file.

Now, that's useful. To appreciate how much, try this: Create a plain-text file containing the names of the knights of the round table, or the days of the week, or whatever. Basically any file with a list of items *not* in alphabetical order:

ArthurLancelotGawainGeraintPercivalBors the YoungerLamorakKayGarethBedivereGaherisGalahadTristan

Save it as knights.txt.

If you wanted to sort the names alphabetically, what would you do? You'd probably try something like this:

you would start overwriting the file before you had finished reading from it, ruining it in the process. So, you need to mess around with that intermediate file,knights_sorted.txt. And you still have that original, unsorted file hanging around, too, which you now have to erase before renaming the sorted file, hence the long unwieldy chain of instructions. With sponge, however, you can do this:

Thanks to sponge, you can grab the content from a text file, do all the chained processes on it using things like sort, unique, sed, grep, and tr. Sponge will soak it all up, wait until all the lines have been processed, and then wring it all out to the same file, all in one blast.

Talking of soaking, let's discuss pee. Despite its name, pee has *nothing* to do with bodily fluids. In fact, the name is a combination of pipe (as in the way you pass the output from one command onto another in *NIX systems) and tee (as in the tee *NIX command line instruction).

While tee re-routes the output from a command to files (cat knights.txt | tee k1 k2 k3 creates files k1, k2, and k3containing the content cat'ed from knights.txt), pee pipes the output into a list of commands:

In the example above, using the output from the original and unorderedknights.txt, you pipe it first to sortto get an ordered list; then to wc (word count), which, with the -loption, returns the number of lines (13); and finally to grep, which then uses a simple regular expression pattern to print out only the lines that start with a capital "G".

Pipe dream

Getting back to editors but staying with pipes that push stuff hither and thither, you must try vipe. Again, this is a twist on your default shell editor. Plunk it into your chain of piped commands and it will open up with the output from the prior instructions. For example:

cat knights.txt | pee sort "wc -l" "grep - -e ^G.*" | vipe | unique

will show all the outputs we saw in the previous example in a (in my case) vi-like editor. You can now edit the output to your heart's content, removing, adding, and modifying lines. When you're done, save and quit, and your edited output will be passed on to the next command in the chain.

Pretty cool, no?

Moreutils has more...

Moreutils comes with many more goodies. The combineutility merges the lines from two files using Boolean operations; ts adds a handy, human readable timestamp to each line taken from an input;ifdata makes extracting data from a network interface supereasy -- very useful for scripting, and so on. Check out the project's web page and the man pages for each command to see how they all work.

Although it is true you could emulate many of the behaviors of these new commands with a bit of command-line fu, tools like sponge, pee, and vipe just make working on the shell easier and more of a pleasure.

The Moreutilspackage is always evolving, and new stuff gets added from time to time. I, for one, am excited to see what gets included next.