Throw out your Perl: One-line aggregation in awk

I ran into a student from a class I taught last summer. He’s a really sharp guy, and when I first met him, I was impressed with just how much Perl he could stuff into his brain’s cache. He would write what he called ‘one-liners’ in Perl that, in reality, took up 5-10 lines in his terminal. Still, he’d type furiously, without skipping a beat. But he told me when we met that he no longer does this, because I covered awk in my class.

His one-liners were mostly for data munging. The data he needed to munge was mostly data that was pretty predictable. It had a fixed number of fields, a consistent delimiter — in short, it was perfect for munging in awk without using any kind of esoteric awk-ness.

One thing I cover in the learning module I’ve developed on Awk is aggregation of data using pretty simple awk one-liners. For example, here’s a pretty typical /etc/passwd file (we need some data to munge):

The important thing to remember is that awk will initialize your variables to 0 for you, which cuts down on some clutter.

Let’s abuse awk a bit further. What if we want to know how many users use each shell in /etc/passwd, whatever those shells may be? Here’s a one-liner that’ll take care of this for you:

awk -F: '{x[$7]+=1} END {for(z in x) {print z, x[z]} }' /etc/passwd

While awk doesn’t technically support multi-dimensional arrays, it also doesn’t have to be numerically indexed. So here, we tell awk to increment x[$7]. $7 is the field that holds the shell for each user, so if $7 on the current line is /bin/bash, then we’ve told awk to increment the value in the array indexed at x[/bin/bash]. So, if there’s only one line containing /bin/bash up to the current record, then x[/bin/bash]=1

There’s a lot of great things you can move onto from here. You can do things that others use Excel for right in awk. If you have checkbook information in a flat file, you can add up purchases only in a given category, or, using the technique above, in every category. If you store your stock purchase price on a line with the current price, you can use simple math to get the spread on each line and tell you whether your portfolio is up or down. Let’s have a look at something like that. Here’s some completely made up, hypothetical data representing a fictitious user’s stock portfolio:

Save that in a file called “stocks.txt”. The columns are stock symbol, number of shares, purchase price, and current price, in that order. This awk one-liner indexes the ‘x’ array using the stock symbol, and the value at that index is set to the amount gained or lost:

Thanks, but I didn’t forget – I decided not to care, because I’m hoping that people will be able to pick up on what’s going on when they see a number with no label attached. In the courses I teach, as in the posts I write, I assume some prior knowledge and experience. This isn’t snobbery, it’s really just to save more experienced users from having to read an explanation that they can very easily grasp for themselves. The solution is an exercise for the reader, and is pretty trivial besides.

http://olifante.blogs.com olifante

The two last awk one-liners are identical, apart from a couple of spaces.

Great – ‘cos that’s so much cleaner, shorter, and more intuitive than the awk code, and I can really see why I would never, ever, ever choose anything over perl for anything now. Thanks for that.

Seriously, those are nice one-liners. I should’ve noted that I also do know, use, and teach perl (only upon request, by gunpoint — but it happens). The article makes it pretty clear, though, that I’m talking about awk in a particular context, for a given purpose. You Perl hackers should know all about context. It’s not a blanket “awk beats perl” thing – I just think awk is cleaner and easier for these types of tasks. Those people I’ve been able to show awk solutions to, who were using perl, now don’t use perl for those tasks. Not because you can’t do it in perl (duh), but because you don’t have to, and awk is a bit more intuitive for certain tasks.

I have plenty of other posts I have yet to write about why I hate perl, and why I think it will eventually become the next cobol — save your flames for those future posts.

So far, fields() does not support arbitrary separator (whitespace only) but than can be fixed 😉

Agate

I don’t think the Perl post is really a flame. But it’s a good example of how you can do the same thing in Perl.

The title is the intentionally-inflammatory “Throw out your Perl,” so is it really surprising that someone felt the need to respond?

To someone familiar with both languages, they are both about equivalently readable. Perl’s “$” are ugly, but then again, awk uses “$” for the column contents, so overall the one-liners seem to be about equally obfuscated to the casual viewer.

http://morlockhq.blogspot.com Jim

Brian-

That’s a good point about context. I look at it as having one more tool in the belt. Linux/Unix have so many specialized tools at the user’s disposal (as well as more generalized tools like perl, python, ruby, c, et..) that it is a shame to specialize in only one.

TrinityGroen2

My children were requiring IRS 706 recently and came across an online service with lots of fillable forms . If people are interested in IRS 706 too , here’s http://goo.gl/yux2cz