Displaying httpd log file entries per minute

$ sed -n 's/.*20[0-9][0-9]:\(.*\):.. +0.00.*/\1/p' "$@" | uniq -c

Let’s say you suspect one of your web servers is misbehaving. Users are complaining that some pages are not loading. You do a tail -f on the access log file and entries are whizzing up the screen, so it certainly seems to be doing something right.

However, your eye can’t tell the difference between 500 log entries per minute and 5000 log entries per minute.

Wouldn’t it be nice if you could see how many log file entries are being generated per minute?

With this short script – no problem!

To create a script that contains the above command and place it in your bin directory, do the following:

The grep command is looking for the pattern /wp-admin/admin.php in the file apache_access.log. Only lines containing that pattern will be in the output of grep. Lines recording the GETting of jpegs, css, js, etc will not be counted.

The sed command is looking for the pattern

.*20[0-9][0-9]:\(.*\):.. +0.00.*

That means it’s looking for the following sequence of patterns:

1) any bunch of characters “.*“

2) 20 followed by two digits “20[0-9][0-9]“

3) a colon “:”

4a) any bunch of characters followed by “\(.*\)“

4b) With the “\(.*\)“, the backslash-escaped parentheses mark a part of the pattern for later use in the second part of the sed expression. The substitution pattern “\1” is then used to output only the part of the string matching the pattern within the backlash-escaped parentheses

5) a colon followed by two arbitrary characters followed by a space “:.. “

This is piped through uniq. But not just any old uniq. This uniq has the option -c, which means “display each duplicated line only once with a count of duplicate lines before the line”

So uniq -c outputs:

317 20:57
221 20:58
4 20:59
244 21:00

This is telling us that /wp-admin/admin.php was fetched 317 times in the minute beginning at 20:57, 221 times at 20:58, only 4 times at 20:59 and 244 times at 21:00.

If you run this command over a few days’ log files you get a feel for how much traffic you normally get per minute. Any periods of unusually low (or high) traffic are easily spotted. Gaps, i.e. minutes with no traffic, can also be spotted easily. If you normally see counts in the hundreds per minute, and suddenly you’re seeing tens per minute, you know something is amiss. If some times are being output in the wrong order, you know that some entries are taking too long to process.

You can easily adjust sed‘s regular expression to display just hours, or hours, minutes and seconds, or even hours, minutes and tens of seconds; the ideal pattern depends on the busy-ness of your server.

Because you are armed with grep, you can issue commands to get any patterns that are of interest to you. You have massive flexibility to search for different things in a very short space of time. You can see what’s happening in real time if you use tail -f to get the output of the log file as it appears and pipe it through the command:

$ tail -f access_log | apache.byminute

As you can see, on the Unix command line you can create extremely powerful ad-hoc tools with just a line or two of code. This kind of code is applicable to many different situations.

For analysing timestamped log entries, get used to the pattern of

- looking for interesting entries with grep

- using sed with an appropriate regular expression to cut out the pattern that is of immediate interest

Real-World Context for your Apps

The idea of small, extremely portable, digital tags, equipped with an ARM processor and numerous sensors, attachable to real world objects and discoverable by any suitably enabled smartphone app had captured my imagination. So I duly shelled out US$99 for the pre-production not yet FCC-approved Estimote(TM) Stickers Nearables Developer Kit containing ten Bluetooth Low Energy Beacons.

I was excited to get my first packet of Estimote Stickers (“real world context for your apps”). This was my first encounter with beacon technology.

I downloaded the Swift tutorial and the SDK from the Estimote web site and started learning. I was finding out about two new technologies at once, the Internet of Things (IoT) in the form of the the Estimote Stickers, and Swift, Apple’s new computer language. It was a lot of fun.

Having programmed the Flipping Blocks game and scratched the surface of Unity3d’s capabilities I started experimenting with Unity’s physics functionality. Unity does so much of the heavy lifting that I was able using about 150 lines of C# code to program a nice 3d swarming simulation based on a few simple rules:

try to follow the leader

don’t get too close to your neighbours

slow down if you go too fast

don’t get too far from the camera (leader only)

It makes your hardware work quite hard. I can watch it for ages, it’s quite fascinating.

Do your eyes glaze over when you think of all the time you’re going to spend creating all those assets? Mine do. Particularly with icons you need practically the same symbol in many different size variations for iPhone, iPad and Android.

The good news is, instead of manually creating them all in Photoshop, Gimp or whatever, you can generate them all automatically from one graphic. This service is provided by the site MakeAppIcon. Just upload one graphical template in PNG, JPEG or PSD format (preferred dimensions 1024×1024), let the site do its thing and within a short time an email with a ZIP attachment containing all the icons in every size that you wanted, and some you didn’t know you wanted.

This saves a lot of time, money and frustration and gives you more time to do something fun, profitable, or both. Many thanks to the FileSquare people for making this available free to everybody.