Chapter 8. Tokenizing Walkthrough

Contents:

So far, I've been showing examples of data in a
particular format, then presenting code for extracting the data out
of that format, as an illustration of newly introduced
HTML::TokeParser methods. But in real life, you do not proceed tidily
from the problem to an immediate and fully formed solution. And
ideally, the task of data extraction is simple: identify patterns
surrounding the data you're after and write a
program that matches those patterns and extracts the embedded data.

In practice, however, you write programs bit by bit and in fits and
starts, and with data extraction specifically; this involves a good
amount of trying one pattern, finding that its matching is too narrow
or too broad, trying to amend it, possibly having to backtrack and
try another pattern, and so on. Moreover, even equally effective
patterns are not equal; some patterns are easier to capture in code
than others, and some patterns are more temporary than others.

In this section, I'll try to make these points by
walking
though the implementation of a data extraction task, with all
alternatives considered, and even a misstep or two.

8.1. The Problem

As a starting point, consider the task of harvesting a month's
worth of listings and corresponding RealAudio URLs from the web site
of the National Public Radio program Fresh Air,
at http://freshair.npr.org. Fresh
Air is on NPR stations each weekday, and on every show,
different guests are interviewed. The show's web
site lists which guests appear on the show each day and has links to
the RealAudio files for each segment of each show. If your particular
weekday schedule doesn't have you listening to
Fresh Air every night or afternoon, you would
find it useful to have a program tell you who had been on in the past
month, so you could make a point of listening to the RealAudio files
for the guests you find interesting. Such a data-extraction program
could be scheduled with crontab to run on the first or second day of
every month, to harvest the past month's program
data.