Piping to Awk

The key is to pipe the output to another tool that reads everything from standard input, and prints the same thing but with a prefix added.

What tool do we use? Awk is a perfect candidate because it’s installed pretty much everywhere. Awk is a tool for processing streams of texts.

Usage in single commands

Here’s an example involving a single command:

command | awk '{ print strftime("[%Y-%m-%d %H:%M:%S]"), $0 }'

However, note that the above only adds timestamp prefixes to text written to standard output. If you want text written to standard error to be timestamp-prefixed too, then make sure you redirect standard error to standard output, like this:

command 2>&1 | awk '{ print strftime("[%Y-%m-%d %H:%M:%S]"), $0 }'

The magic part is 2>&1. It means: redirect file descriptor number 2 (standard error) to file descriptor number 1 (standard output).

Caveat: this is Bash, not sh

sh is not Bash. sh is a shell whose history is older than Bash. Although /bin/sh may point to the same executable as Bash, when Bash is run as sh it will execute in a sort of “legacy mode” and disable some features. So if your shell scripts use the trick described in this blog post, make sure your shebang line is #!/bin/bash and not #!/bin/sh.

Caveat: block buffering

Commands behave a little bit differently if their output is redirected to a pipe.

Normally, commands use line buffering, which means that every time they’ve printed a complete line, that line will be written to the terminal.

But when the output is redirected to something which is not a terminal, then commands will use block buffering, which means that they don’t actually write out what they intend to print; until a threshold as been reached, or until the command exits. This threshold is around 16 KB. You can see block buffering in action here:

This Python command prints 15 KB of data to standard output, then sleeps for 10 seconds. The output is piped to cat. Note that during the 10-second sleep, nothing is printed. Only after the 10 seconds does it print something.

If you increase 15 to 16, then you’ll see the output immediately.

If this behavior is problematic for you, then there are various ways to force line buffering:

On Linux and FreeBSD, you can use stdbuf. Prefix it to any command for which you want to force line buffering:

For Python scripts, you can set the environment variable PYTHONUNBUFFERED=1, which tells Python to disable any sort of output buffering entirely.

Caveats of, alternatives to Awk

That there are multiple implementations of Awk, with different features. On Linux, you would usually be using the GNU implementation. On macOS, you would be using a BSD implementation. If you intent to write shell scripts that run on more than just one OS, then be sure to test things on all platforms you intend to support.

In fact, the Awk implementation on macOS does not support the strftime function!

ABOUT FULLSTAQ

We use our wide expertise to help customers tackle their complex technology problems. We thrive to go beyond the service most traditional infrastructure vendors offer by working closely together with your software development teams and integrating the DevOps mentality, toolstack and best practices.