Wednesday, August 27, 2008

Creating a Pseudo Daemon Using Perl

One of the trickier software jobs I've worked on is a Perl script that runs almost continuously.

The script was needed to check a folder for new content (in the form of news stories) and process them.

The stories are dropped into a folder as a group via ftp, and the name of each story is written into a separate file. The order in this file is the order the stories need to appear on the website.

The old version of the script ran by cron every minute, but there were three problems.

The first is that you might have to wait a whole minute for content to be processed, which is not really ideal for a news service.

The second is that should the script be delayed for some reason, it is possible to end up with a race condition with two (or more scripts) trying to process the same content.

The third is that the script could start reading the order file before all the files were uploaded.

In practice 2 and 3 were very rare, but very disruptive when they did occur. The code needed to avoid both.

The new script is still run once per minute via cron, but contains a loop which allows it to check for content every three seconds.

It works this way:

1. When the script starts it grabs the current time and tries to obtain an exclusive write lock on a lock file.2. If it gets the lock it starts the processing loop.3. When the order file is found the script waits for 10 seconds. This is to allow any current upload process to complete.4. It then reads the file, deletes it, and starts processing each of the story files.5. These are all written out to XML (which is imported into the CMS), and the original files are deleted.6. When this is done, the script continues to look for files until 55 seconds have elapsed since it started.7. When no time is left it exits.

This is what the loop code looks like:

my $stop_time = time + 55;

my $loop = 1;my $locking_loop = 1;my $has_run = 0;

while( $locking_loop ){

# first see if there is a lock file and# wait till the other process is done if there isif( open my $LOCK, '>>', 'inews.lock' ){