Tuesday, May 30. 2006

If anything, this is a note to self. I tend to keep a session of multitail open on any server I like keeping an eye on. Typically it will include the security logs, http logs, error logs, and audit logs. In a few instances when 'logrotate' has kicked in overnight while a 'multitail' session was open, watching the files that were getting moved, all hell breaks loose.

Since the file handle/files that multitail watches suddenly disappear/become invalid, the spawned 'tail' children go into a psuedo-fork-bomb/zombie loop.

For instance, this morning I awoke to about 3000 rapidly spawning and dying 'tail' processes that made it difficult to near impossible to pull up a process list. When that failed, I ended up browsing /proc to see if something bad was going on. I also peeked at /proc/loadavg which was actually pretty normal. Since I have some sane ulimits set on the machine in question, the overall problem didn't become a system-wide out-of-memory problem/crash.

When I get a moment I'll see if I can hack a way to make multitail a little more friendly to files getting purged from under it.