12.4. Text Processing Commands

Commands affecting text and
text files

sort

File sorter, often used as a filter in a pipe. This
command sorts a text stream or file forwards or backwards,
or according to various keys or character positions. Using
the -m option, it merges presorted input
files. The info page lists its many
capabilities and options. See Example 10-9,
Example 10-10, and Example A-8.

tsort

Topological sort, reading in pairs of
whitespace-separated strings and sorting according to
input patterns.

uniq

This filter removes duplicate lines from a sorted
file. It is often seen in a pipe coupled with
sort.

The useful -c option prefixes each line of
the input file with its number of occurrences.

bash$ cat testfileThis line occurs only once.
This line occurs twice.
This line occurs twice.
This line occurs three times.
This line occurs three times.
This line occurs three times.bash$ uniq -c testfile 1 This line occurs only once.
2 This line occurs twice.
3 This line occurs three times.bash$ sort testfile | uniq -c | sort -nr 3 This line occurs three times.
2 This line occurs twice.
1 This line occurs only once.

The sort INPUTFILE | uniq -c | sort -nr
command string produces a frequency
of occurrence listing on the
INPUTFILE file (the
-nr options to sort
cause a reverse numerical sort). This template finds
use in analysis of log files and dictionary lists, and
wherever the lexical structure of a document needs to
be examined.

bash$ cat testfileThis line occurs only once.
This line occurs twice.
This line occurs twice.
This line occurs three times.
This line occurs three times.
This line occurs three times.bash$ ./wf.sh testfile 6 this
6 occurs
6 line
3 times
3 three
2 twice
1 only
1 once

expand, unexpand

The expand filter converts tabs to
spaces. It is often used in a pipe.

The unexpand filter
converts spaces to tabs. This reverses the effect of
expand.

cut

A tool for extracting fields from files. It is similar to the
print $N command set in awk, but more limited. It may be
simpler to use cut in a script than
awk. Particularly important are the
-d (delimiter) and -f
(field specifier) options.

Tool for merging together different files into a single,
multi-column file. In combination with
cut, useful for creating system log
files.

join

Consider this a special-purpose cousin of
paste. This powerful utility allows
merging two files in a meaningful fashion, which essentially
creates a simple version of a relational database.

The join command operates on
exactly two files, but pastes together only those lines
with a common tagged field (usually a numerical label),
and writes the result to stdout.
The files to be joined should be sorted according to the
tagged field for the matchups to work properly.

lists the beginning of a file to
stdout (the default is
10 lines, but this can be changed). It
has a number of interesting options.

Example 12-12. Which files are scripts?

#!/bin/bash
# script-detector.sh: Detects scripts within a directory.
TESTCHARS=2 # Test first 2 characters.
SHABANG='#!' # Scripts begin with a "sha-bang."
for file in * # Traverse all the files in current directory.
do
if [[ `head -c$TESTCHARS "$file"` = "$SHABANG" ]]
# head -c2 #!
# The '-c' option to "head" outputs a specified
#+ number of characters, rather than lines (the default).
then
echo "File \"$file\" is a script."
else
echo "File \"$file\" is *not* a script."
fi
done
exit 0
# Exercises:
# ---------
# 1) Modify this script to take as an optional argument
#+ the directory to scan for scripts
#+ (rather than just the current working directory).
#
# 2) As it stands, this script gives "false positives" for
#+ Perl, awk, and other scripting language scripts.
# Correct this.

Example 12-13. Generating 10-digit random numbers

#!/bin/bash
# rnd.sh: Outputs a 10-digit random number
# Script by Stephane Chazelas.
head -c4 /dev/urandom | od -N4 -tu4 | sed -ne '1s/.* //p'
# =================================================================== #
# Analysis
# --------
# head:
# -c4 option takes first 4 bytes.
# od:
# -N4 option limits output to 4 bytes.
# -tu4 option selects unsigned decimal format for output.
# sed:
# -n option, in combination with "p" flag to the "s" command,
# outputs only matched lines.
# The author of this script explains the action of 'sed', as follows.
# head -c4 /dev/urandom | od -N4 -tu4 | sed -ne '1s/.* //p'
# ----------------------------------> |
# Assume output up to "sed" --------> |
# is 0000000 1198195154\n
# sed begins reading characters: 0000000 1198195154\n.
# Here it finds a newline character,
#+ so it is ready to process the first line (0000000 1198195154).
# It looks at its <range><action>s. The first and only one is
# range action
# 1 s/.* //p
# The line number is in the range, so it executes the action:
#+ tries to substitute the longest string ending with a space in the line
# ("0000000 ") with nothing (//), and if it succeeds, prints the result
# ("p" is a flag to the "s" command here, this is different from the "p" command).
# sed is now ready to continue reading its input. (Note that before
#+ continuing, if -n option had not been passed, sed would have printed
#+ the line once again).
# Now, sed reads the remainder of the characters, and finds the end of the file.
# It is now ready to process its 2nd line (which is also numbered '$' as
# it's the last one).
# It sees it is not matched by any <range>, so its job is done.
# In few word this sed commmand means:
# "On the first line only, remove any character up to the right-most space,
#+ then print it."
# A better way to do this would have been:
# sed -e 's/.* //;q'
# Here, two <range><action>s (could have been written
# sed -e 's/.* //' -e q):
# range action
# nothing (matches line) s/.* //
# nothing (matches line) q (quit)
# Here, sed only reads its first line of input.
# It performs both actions, and prints the line (substituted) before quitting
#+ (because of the "q" action) since the "-n" option is not passed.
# =================================================================== #
# An even simpler altenative to the above one-line script would be:
# head -c4 /dev/urandom| od -An -tu4
exit 0

lists the end of a file to stdout
(the default is 10 lines). Commonly used
to keep track of changes to a system logfile, using the
-f option, which outputs lines appended
to the file.

Example 12-14. Using tail to monitor the system log

#!/bin/bash
filename=sys.log
cat /dev/null > $filename; echo "Creating / cleaning out file."
# Creates file if it does not already exist,
#+ and truncates it to zero length if it does.
# : > filename and > filename also work.
tail /var/log/messages > $filename
# /var/log/messages must have world read permission for this to work.
echo "$filename contains tail end of system log."
exit 0

To list a specific line of a text file,
pipe the output of
head to tail -1.
For example head -8 database.txt | tail
-1 lists the 8th line of the file
database.txt.

To set a variable to a given block of a text file:

var=$(head -$m $filename | tail -$n)
# filename = name of file
# m = from beginning of file, number of lines to end of block
# n = number of lines to set variable to (trim from end of block)

bash$ grep file tstfile# Filename: tstfile
This is a sample file.
This is an ordinary text file.
This file does not contain any unusual text.
This file is not unusual.bash$ grep file tstfile | grep textThis is an ordinary text file.
This file does not contain any unusual text.

--

egrep
- extended grep - is the same
as grep -E. This uses a somewhat
different, extended set of Regular
Expressions, which can make the search a bit more
flexible.

fgrep - fast grep
- is the same as grep -F. It does
a literal string search (no Regular Expressions), which
usually speeds things up a bit.

On some Linux distros, egrep and
fgrep are symbolic links to, or aliases for
grep, but invoked with the
-E and -F options,
respectively.

Example 12-16. Looking up definitions in Webster's 1913 Dictionary

#!/bin/bash
# dict-lookup.sh
# This script looks up definitions in the 1913 Webster's Dictionary.
# This Public Domain dictionary is available for download
#+ from various sites, including
#+ Project Gutenberg (http://www.gutenberg.org/etext/247).
#
# Convert it from DOS to UNIX format (only LF at end of line)
#+ before using it with this script.
# Store the file in plain, uncompressed ASCII.
# Set DEFAULT_DICTFILE variable below to path/filename.
E_BADARGS=65
MAXCONTEXTLINES=50 # Maximum number of lines to show.
DEFAULT_DICTFILE="/usr/share/dict/webster1913-dict.txt"
# Default dictionary file pathname.
# Change this as necessary.
# Note:
# ----
# This particular edition of the 1913 Webster's
#+ begins each entry with an uppercase letter
#+ (lowercase for the remaining characters).
# Only the *very first line* of an entry begins this way,
#+ and that's why the search algorithm below works.
if [[ -z $(echo "$1" | sed -n '/^[A-Z]/p') ]]
# Must at least specify word to look up, and
#+ it must start with an uppercase letter.
then
echo "Usage: `basename $0` Word-to-define [dictionary-file]"
echo
echo "Note: Word to look up must start with capital letter,"
echo "with the rest of the word in lowercase."
echo "--------------------------------------------"
echo "Examples: Abandon, Dictionary, Marking, etc."
exit $E_BADARGS
fi
if [ -z "$2" ] # May specify different dictionary
#+ as an argument to this script.
then
dictfile=$DEFAULT_DICTFILE
else
dictfile="$2"
fi
# ---------------------------------------------------------
Definition=$(fgrep -A $MAXCONTEXTLINES "$1 \\" "$dictfile")
# Definitions in form "Word \..."
#
# And, yes, "fgrep" is fast enough
#+ to search even a very large text file.
# Now, snip out just the definition block.
echo "$Definition" |
sed -n '1,/^[A-Z]/p' |
# Print from first line of output
#+ to the first line of the next entry.
sed '$d' | sed '$d'
# Delete last two lines of output
#+ (blank line and first line of next entry).
# ---------------------------------------------------------
exit 0
# Exercises:
# ---------
# 1) Modify the script to accept any type of alphabetic input
# + (uppercase, lowercase, mixed case), and convert it
# + to an acceptable format for processing.
#
# 2) Convert the script to a GUI application,
# + using something like "gdialog" . . .
# The script will then no longer take its argument(s)
# + from the command line.
#
# 3) Modify the script to parse one of the other available
# + Public Domain Dictionaries, such as the U.S. Census Bureau Gazetteer.

agrep (approximate
grep) extends the capabilities of
grep to approximate matching. The search
string may differ by a specified number of characters
from the resulting matches. This utility is not part of
the core Linux distribution.

To search compressed files, use
zgrep, zegrep, or
zfgrep. These also work on non-compressed
files, though slower than plain grep,
egrep, fgrep.
They are handy for searching through a mixed set of files,
some compressed, some not.

Must use quoting and/or
brackets, as appropriate. Quotes prevent the
shell from reinterpreting the special characters in
tr command sequences. Brackets should be
quoted to prevent expansion by the shell.

Either tr "A-Z" "*" <filename
or tr A-Z \* <filename changes
all the uppercase letters in filename
to asterisks (writes to stdout).
On some systems this may not work, but tr A-Z
'[**]' will.

#!/bin/bash
# crypto-quote.sh: Encrypt quotes
# Will encrypt famous quotes in a simple monoalphabetic substitution.
# The result is similar to the "Crypto Quote" puzzles
#+ seen in the Op Ed pages of the Sunday paper.
key=ETAOINSHRDLUBCFGJMQPVWZYXK
# The "key" is nothing more than a scrambled alphabet.
# Changing the "key" changes the encryption.
# The 'cat "$@"' construction gets input either from stdin or from files.
# If using stdin, terminate input with a Control-D.
# Otherwise, specify filename as command-line parameter.
cat "$@" | tr "a-z" "A-Z" | tr "A-Z" "$key"
# | to uppercase | encrypt
# Will work on lowercase, uppercase, or mixed-case quotes.
# Passes non-alphabetic characters through unchanged.
# Try this script with something like:
# "Nothing so needs reforming as other people's habits."
# --Mark Twain
#
# Output is:
# "CFPHRCS QF CIIOQ MINFMBRCS EQ FPHIM GIFGUI'Q HETRPQ."
# --BEML PZERC
# To reverse the encryption:
# cat "$@" | tr "$key" "A-Z"
# This simple-minded cipher can be broken by an average 12-year old
#+ using only pencil and paper.
exit 0
# Exercise:
# --------
# Modify the script so that it will either encrypt or decrypt,
#+ depending on command-line argument(s).

tr variants

The tr utility has two historic
variants. The BSD version does not use brackets
(tr a-z A-Z), but the SysV one does
(tr '[a-z]' '[A-Z]'). The GNU version
of tr resembles the BSD one, so quoting
letter ranges within brackets is mandatory.

fold

A filter that wraps lines of input to a specified width.
This is especially useful with the -s
option, which breaks lines at word spaces (see Example 12-23 and Example A-1).

fmt

Simple-minded file formatter, used as a filter in a
pipe to "wrap" long lines of text
output.

This deceptively named filter removes reverse line feeds
from an input stream. It also attempts to replace
whitespace with equivalent tabs. The chief use of
col is in filtering the output
from certain text processing utilities, such as
groff and tbl.

#!/bin/bash
# This is a slight modification of the example file in the "column" man page.
(printf "PERMISSIONS LINKS OWNER GROUP SIZE MONTH DAY HH:MM PROG-NAME\n" \
; ls -l | sed 1d) | column -t
# The "sed 1d" in the pipe deletes the first line of output,
#+ which would be "total N",
#+ where "N" is the total number of files found by "ls -l".
# The -t option to "column" pretty-prints a table.
exit 0

colrm

Column removal filter. This removes columns (characters)
from a file and writes the file, lacking the range of
specified columns, back to stdout.
colrm 2 4 <filename removes the
second through fourth characters from each line of the
text file filename.

If the file contains tabs or nonprintable
characters, this may cause unpredictable
behavior. In such cases, consider using
expand and
unexpand in a pipe preceding
colrm.

nl

Line numbering filter. nl filename
lists filename to
stdout, but inserts consecutive
numbers at the beginning of each non-blank line. If
filename omitted, operates on
stdin.

The output of nl is very similar to
cat -n, however, by default
nl does not list blank lines.

Example 12-25. nl: A self-numbering script.

#!/bin/bash
# line-number.sh
# This script echoes itself twice to stdout with its lines numbered.
# 'nl' sees this as line 4 since it does not number blank lines.
# 'cat -n' sees the above line as number 6.
nl `basename $0`
echo; echo # Now, let's try it with 'cat -n'
cat -n `basename $0`
# The difference is that 'cat -n' numbers the blank lines.
# Note that 'nl -ba' will also do so.
exit 0
# -----------------------------------------------------------------

The GNU gettext package is a set of
utilities for localizing
and translating the text output of programs into foreign
languages. While originally intended for C programs, it
now supports quite a number of programming and scripting
languages.

The gettextprogram works on shell scripts. See
the info page.

msgfmt

A program for generating binary
message catalogs. It is used for localization.

iconv

A utility for converting file(s) to a different encoding
(character set). Its chief use is for localization.