Work the Shell - Breaking Numbers Down

A kilo of information on how to represent even giga numbers in a mega-useful way.

Last month, we continued our journey into the dark caverns of Apache Web
logs, examining how relatively simple shell scripts can be utilized to
produce useful and important data. The specific script we created searched
a log file for traffic that occurred the previous day, summarizing the
number of bytes transmitted.

That's all well and good, but as with many shell scripts, there's a bit
of a problem with this one, which was immediately obvious when my busy
site produced an estimated monthly data transfer rate of 2346990660 bytes.

Clearly that's a very human-unfriendly number, and doubly so without
any commas to break it up into thousands, millions and so on. More
important, when talking about data transfer, we're used to thinking
in terms of powers of two, so 1 kilobyte is 1024 bytes of data, not 1000
bytes of data, and 1 megabyte is 1024 kilobytes of data, and so on.

Unfortunately, the expr command that we're using for the mathematical
calculations doesn't have the ability to work with these powers of two,
so we're going to have to do the work ourselves, converting massive
numbers into more readable KB, MB or GB values, as appropriate.

We don't want zero values; we want to see the fractional decimal values,
which means not only that we can't use the built-in mathematical
capabilities of the shell, but we also can't use expr. Instead, we need
to move into the crufty, ancient world of bc, the binary calculator.

Now, bc isn't for the faint of heart, but to save you from reading the
man page, here's how you can force four digits after the decimal point
on the result of a division that results in a value less than 1.0:

$ echo "scale=2 ; 3000 / 30001" | bc
.0999

Can you see how to put these together? Here's a new, far-improved way
to calculate kilo, mega and giga:

The debug output from the -x option is getting a bit confusing here,
I admit, but you now can see that kilo is set to 4.88 when given the
initial value of 5000 bytes, and that both mega and giga are zero.

Let's try again (and I'll clean up some of the spurious debug output
from this point on, for clarity) with the initial really big value:

Cool. Now we can finally see that we're talking about 2.18GB of data
being transferred off the site each month—far more coherent than the
huge value shown earlier.

Now, let's figure out how to show always the most logical of these values,
rather than all of them.

Displaying the Simplest Answer Only

The easiest way to figure out which value is best is simply to ascertain
where the value drops below 1.0. In the case of 5000 bytes, that'd be best
displayed as 4.88KB, and in the case of the bignum value, that's 2.18GB.

To figure out when the value drops below zero, we'd love to have a
floating-point numeric comparison, but sadly, the shell can't manage
it. If you try it, you'll just get the error “integer expression
expected”.

There are a number of ways to get the “floor” of the value,
but I use
bc again here to do the job by calculating the division once more, this time
without any scale value at all:

kiloint=$( echo "$value/1024" | bc)"

Doing this gets just the integer portion of the $kilo value, and that can indeed
be tested in a conditional statement:

The final step is to make it a function so we can include it in other
shell scripts and access it as desired. This is done within the Bourne
Shell by giving it a unique name and then wrapping the functional code
in braces:

kmg()
{
code for function goes here, params are $1, $2, etc.
}

This can then be invoked within a shell script by name (k=kilo, m=mega,
g=giga):

kmg 500000

More important, you can embed it within a line by using a subshell
notation, so given the kmg() function, the following two-line script
works splendidly:

echo given value is $1
echo which converts to $(kmg $1)

That's nice and short, and if the kmg function is dropped into its own
file, you also can use the . command to include another file in the
shell script, meaning that the entire test script is now:

#!/bin/sh
. kmg.sh
echo The given value $1 bytes = $(kmg $1)
exit 0

I'm out of space here, but I hope you can see how this approach can be
applied to a wide variety of different shell tasks, making your shell
scripts far more efficient and faster to write too!

Dave Taylor is a 26-year veteran of UNIX, creator of The Elm Mail System,
and most recently author of both the best-selling Wicked Cool
Shell
Scripts and Teach Yourself Unix in 24
Hours, among his 16 technical
books. His main Web site is at www.intuitive.com.

Dave Taylor has been hacking shell scripts for over thirty years. Really.
He's the author of the popular "Wicked Cool Shell Scripts" and
can be found on Twitter as @DaveTaylor and more generally at
www.DaveTaylorOnline.com.