Friday, August 3, 2012

The tips and tricks below originally appeared as one of Google's "Testing on the Toilet" (TOTT) episodes. This is a revised and augmented version.

Safer Scripting

I start every bash script with the following prolog:

#!/bin/bashset -o nounsetset -o errexit

This will take care of two very common errors:

Referencing undefined variables (which default to "")

Ignoring failing commands

The two settings also have shorthands (“-u” and “-e”) but the longer versions are more readable.If a failing command is to be tolerated use this idiom:

if ! <possible failing command> ; then echo "failure ignored" fi

Note that some Linux commands have options which as a side-effect suppress some failures, e.g.“mkdir -p” and “rm -f”.Also note, that the “errexit” mode, while a valuable first line of defense, does not catch all failures, i.e. under certain circumstances failing commands will go undetected.(For more info, have a look at this thread.)

A reader suggested the additional use of "set -o pipefail"

Functions

Bash lets you define functions which behave like other commands -- use them liberally; it will give your bash scripts a much needed boost in readability:

ExtractBashComments() { egrep "^#"}

cat myscript.sh | ExtractBashComments | wc

comments=$(ExtractBashComments < myscript.sh)

Some more instructive examples:

SumLines() { # iterating over stdin - similar to awk local sum=0 local line=”” while read line ; do sum=$((${sum} + ${line})) done echo ${sum}}

Avoiding Temporary Files

Some commands expect filenames as parameters so straightforward pipelining does not work.
This is where <() operator comes in handy as it takes a command and transforms it into something
which can be used as a filename:

Built-In Variables

For reference

$0 name of the script
$n positional parameters to script/function
$$ PID of the script
$!PID of the last command executed (and run in the background)
$?exit status of the last command (${PIPESTATUS} for pipelined commands)
$#number of parameters to script/function
$@ all parameters to script/function (sees arguments as separate word)
$* all parameters to script/function (sees arguments as single word)

Note

$* is rarely the right choice.
$@ handles empty parameter list and white-space within parameters correctly
$@ should usually be quoted like so "$@"

Debugging

To perform a syntax check/dry run of your bash script run:

bash -n myscript.sh

To produce a trace of every command executed run:

bash -v myscripts.sh

To produce a trace of the expanded command use:

bash -x myscript.sh

-v and -x can also be made permanent by adding
set -o verbose and set -o xtrace to the script prolog.
This might be useful if the script is run on a remote machine, e.g.
a build-bot and you are logging the output for remote inspection.

Signs you should not be using a bash script

your script is longer than a few hundred lines of code

you need data structures beyond simple arrays

you have a hard time working around quoting issues

you do a lot of string manipulation

you do not have much need for invoking other programs or pipe-lining them