Source filtering is an immensely powerful feature of recent versions of Perl. It allows one to extend the language itself (e.g. the Switch module), to simplify the language (e.g. Language::Pythonesque), or to completely recast the language (e.g. Lingua::Romana::Perligata). Effectively, it allows one to use the full power of Perl as its own, recursively applied, macro language.

The excellent Filter::Util::Call module (by Paul Marquess) provides a usable Perl interface to source filtering, but it is often too powerful and not nearly as simple as it could be.

To use the module it is necessary to do the following:

Download, build, and install the Filter::Util::Call module. (If you have Perl 5.7.1 or later, this is already done for you.)

Set up a module that does a use Filter::Util::Call.

Within that module, create an import subroutine.

Within the import subroutine do a call to filter_add, passing it either a subroutine reference.

Within the subroutine reference, call filter_read or filter_read_exact to "prime" $_ with source code data from the source file that will use your module. Check the status value returned to see if any source code was actually read in.

Process the contents of $_ to change the source code in the desired manner.

Return the status value.

If the act of unimporting your module (via a no) should cause source code filtering to cease, create an unimport subroutine, and have it call filter_del. Make sure that the call to filter_read or filter_read_exact in step 5 will not accidentally read past the no. Effectively this limits source code filters to line-by-line operation, unless the import subroutine does some fancy pre-pre-parsing of the source code it's filtering.

For example, here is a minimal source code filter in a module named BANG.pm. It simply converts every occurrence of the sequence BANG\s+BANG to the sequence die 'BANG' if $BANG in any piece of code following a use BANG; statement (until the next no BANG; statement, if any):

By default, the installed filter only filters up to a line consisting of one of the three standard source "terminators":

no ModuleName; # optional comment

or:

__END__

or:

__DATA__

but this can be altered by passing a second argument to use Filter::Simple or FILTER (just remember: there's no comma after the initial block when you use FILTER).

That second argument may be either a qr'd regular expression (which is then used to match the terminator line), or a defined false value (which indicates that no terminator line should be looked for), or a reference to a hash (in which case the terminator is the value associated with the key 'terminator'.

For example, to cause the previous filter to filter only up to a line of the form:

Except for FILTER_ONLY code => sub {...}, each of the component filters is called repeatedly, once for each component found in the source code.

Note that you can also apply two or more of the same type of filter in a single FILTER_ONLY. For example, here's a simple macro-preprocessor that is only applied within regexes, with a final debugging pass that prints the resulting source code:

Most source code ceases to be grammatically correct when it is broken up into the pieces between string literals and regexes. So the 'code' and 'code_no_comments' component filter behave slightly differently from the other partial filters described in the previous section.

Rather than calling the specified processor on each individual piece of code (i.e. on the bits between quotelikes), the 'code...' partial filters operate on the entire source code, but with the quotelike bits (and, in the case of 'code_no_comments', the comments) "blanked out".

That is, a 'code...' filter replaces each quoted string, quotelike, regex, POD, and __DATA__ section with a placeholder. The delimiters of this placeholder are the contents of the $; variable at the time the filter is applied (normally "\034"). The remaining four bytes are a unique identifier for the component being replaced.

This approach makes it comparatively easy to write code preprocessors without worrying about the form or contents of strings, regexes, etc.

For convenience, during a 'code...' filtering operation, Filter::Simple provides a package variable ($Filter::Simple::placeholder) that contains a pre-compiled regex that matches any placeholder...and captures the identifier within the placeholder. Placeholders can be moved and re-ordered within the source code as needed.

In addition, a second package variable (@Filter::Simple::components) contains a list of the various pieces of $_, as they were originally split up to allow placeholders to be inserted.

Once the filtering has been applied, the original strings, regexes, POD, etc. are re-inserted into the code, by replacing each placeholder with the corresponding original component (from @components). Note that this means that the @components variable must be treated with extreme care within the filter. The @components array stores the "back- translations" of each placeholder inserted into $_, as well as the interstitial source code between placeholders. If the placeholder backtranslations are altered in @components, they will be similarly changed when the placeholders are removed from $_ after the filter is complete.

For example, the following filter detects concatentated pairs of strings/quotelikes and reverses the order in which they are concatenated:

Filter::Simple generates a special import subroutine for your module (see "How it works") which would normally replace any import subroutine you might have explicitly declared.

However, Filter::Simple is smart enough to notice your existing import and Do The Right Thing with it. That is, if you explicitly define an import subroutine in a package that's using Filter::Simple, that import subroutine will still be invoked immediately after any filter you install.

The only thing you have to remember is that the import subroutine must be declared before the filter is installed. If you use FILTER to install the filter:

Immediately after the filter has been applied to the source, Filter::Simple will pass control to Exporter, so it can do its magic too.

Of course, here too, Filter::Simple has to know you're using Exporter before it applies the filter. That's almost never a problem, but if you're nervous about it, you can guarantee that things will work correctly by ensuring that your use base Exporter always precedes your use Filter::Simple.

The Filter::Simple module exports into the package that calls FILTER (or uses it directly) -- such as package "BANG" in the above example -- two automagically constructed subroutines -- import and unimport -- which take care of all the nasty details.

In addition, the generated import subroutine passes its own argument list to the filtering subroutine, so the BANG.pm filter could easily be made parametric:

The specified filtering subroutine is called every time a use BANG is encountered, and passed all the source code following that call, up to either the next no BANG; (or whatever terminator you've set) or the end of the source file, whichever occurs first. By default, any no BANG; call must appear by itself on a separate line, or it is ignored.