All right, Steve, I'll try to give this one some thought. Since you seem
to be a local, if I've answered your question sufficiently, feel free to
treat me to a pint of Colorado Kind Ale at the Mountain Sun brewery on
Pearl Street in Boulder. :-)

WARNING: This article contains more punctuation that many
non-programmers like. That's because regexps and finite automata are
concise ways of expressing powerful concepts. They are not for the
faint of heart. Don't blame Perl if you don't like regular
expressions. And don't blame Thompson or anyone else. If you don't
like them, don't use them. But don't despise those of us who do.

Here goes...

Why is Perl so useful for sysadmin and WWW and text hacking? It has
a lot of nice little features that make it easy to do nearly anything
you want to text. A lot of perl programs look like a weird synergy of
C and shell and sed and awk. For example:

#!/usr/bin/perl
# manpath -- tchrist@perl.com
foreach $bindir (split(/:/, $ENV{PATH})) {
($mandir = $bindir) =~ s/[^\/]+$/man/;
next if $mandir =~ /^\./ || $mandir eq '';
if (-d $mandir && ! $seen{$mandir}++ ) {
($dev,$ino) = stat($mandir);
if (! $seen{$dev,$ino}++) {
push(@manpath,$mandir);
}
}
}
print join(":", @manpath), "\n";
Can anyone see what that does? I'd like to think it's not too hard, even
devoid of commentary. It does have some naughty bits, like using side
effect operators of assignment operators as expressions and double-plus
postfix autoincrement. C programmers don't have a problem with it, but a
lot of others do. That's why Guido banned such things in Python (a rather
nice language in many ways), and why I don't advocate using them to non-C
programmers, whom it generally confuses whether it be done in C or in Perl
or C++ or any such language.

By far the most bizarre thing is that dread punctuation lying within funny
slashes. Often folks call Perl unreadable because they don't grok
regexps, which all true perl wizards -- and acolytes -- adore. The
slashes and their patterns govern matching and splitting and substituting,
and here is where a lot of the Perl magic resides: its unmatched :-)
regular expressions. Certainly the above code could be rewritten in tcl
or python or nearly anything else. It could even be rewritten in more
legible perl. :-)

So what's so special about perl's regexps? Quite a bit, actually,
although the real magic isn't demonstrated very well in the manpath
program. Once you've read the perlre(1) and the perlop(1) man pages,
there's still a lot to talk about. So permit me, if you would, to now
explain Far More Than Everything You Ever Wanted to Know about Perl
Regular Expressions... :-)

Perl starts with POSIX regexps of the "modern" variety, that is,
egrep
style not grep style. Here's the simple case of matching a number

/Th?om(as)? (Ch|K)rist(ia|e)ns{1,2}s[eo]n/
This avoids a lot of backslashes. I believe many languages also
support such regular rexpressions.

Now, Perl's regexps "aren't" -- that is, they aren't "regular" because
backreferences per sed and grep are also supported, which renders the
language no longer strictly regular and so forbids "pure" DFA
implementations.

But this is exceedingly useful. Backreferences let you refer
back to match part of what you just had. Consider lines like
these:

1. This is a fine kettle of fish.
2. The best card is Island Fish Jasconius.
3. Is isolation unpleasant?
4. That's his isn't it?
5. Is is outlawed?
If you'd like to pick up duplicate "is" strings there, you
could use the pattern
/(is) \1/ # matches 1,4
As written, that will match sentences 1 and 4. The others
fail due to mixed case. You can't fix it just by saying
/([Ii]s) \1/ # still matches 1,4
because the \1 refers back to the real match, not the potential
match. So what do we do? Well, POSIX specifies a REG_ICASE
flag you can pass into your matcher to help support "grep -i"
etc. To get perl to do this, affix an i flag after the match:
/(is) \1/i # matches 1,2,3,4,5
And now all 5 of those sentences match. If you only wanted
them to match legit words, you might use the \b notation for
word boundaries, making it
/\b(is) \1/i # matches 2,3,5
/(is) \1\b/i # matches 1,5
/\b(is) \1\b/i # matches 5
This means you will see Perl code like
if ( $variable =~ /\b(is) \1\b/i ) {
print "gotta match";
}
One might argue that is "should" be written more like
if ( rematch(variable, '\b(is) \1\b', 'i') ) {
print "gotta match";
}
but that's not how Perl works. I suspect that other languages
could make it work that way.

If you'd like to know where you matched, you might want to use
these:

$MATCH full match
$PREMATCH before the match
$POSTMATCH after the match
$LAST_PAREN_MATCH useful for alternatives
Although the most normal case is just to use $1,
$2, etc, which
match the first, second, etc parenthesized subexpressions.

Another nice thing that Perl supports are the notions from
C's ctype.h include file:

C function Perl regexp
isalnum \w
isspace \s
isdigit \d
That means that you don't have to hard-code [A-Z]
and have it break when
someone has some interesting locale settings. For example, under
charset=ISO-8859-1, something like "façade" properly
matches /^\w+$/,
because the c-cedille is considered an alphanum. In theory,
LC_NUMERIC
settings should also take, but I've never tried.

This quickly leads to a pattern that detects duplicate words in
sentences:

/\b(\w+)(\s+\1)+\b/i
In fact, that one matches multiple duplicates as well. If
if if you read in your input data a paragraph at a time, it will
catch dups crossing line boundaries as as well. For example, using
some convenient command line flags, here's a
perl -00 -ne 'if ( /\b(\w+)(\s+\1)+\b/i ) { print "dup $1 at $.\n" }'
which when used on this article says:
dup Is at 10
dup If at 33
the $. variable ($NR in English mode)
is the record number. I set
it to read paragraph records, so paragraphs 10 and 33 of this posting
contain duplicate words.

Actually, we can do something a bit nicer: we can find multiple duplicates
in the same paragraph. The /g flag causes a match to store a bit
of state and start up where it last left off. This gives us:

#!/usr/bin/perl -00 -n
while ( /\b(\w+)(\s+\1)+\b/gi ) {
print "dup $1 at paragraph $.\n";
}
This now yields:
dup Is at paragraph 10
dup if at paragraph 33
dup as at paragraph 33
Of course, we're getting a bit hard to read here. So let's use
the /x flag to permit embedded white space and comments in our
pattern -- you'll want 5.002 for this (the white space worked
in 5.000, but the comments were added later :-). For legibility,
instead of slashes for the match, I'll embrace the real m()
function,
Since /foo/ and m(foo) and m{foo}
are all equivalent.
#!/usr/bin/perl -n
require 5.002;
use English;
$RS = '';
while (
m{ # m{foo} is like /foo/, but helps vi's % key
\b # first find a word boundary
(\w+) # followed by the biggest word we can find
# which we'll save in the \1 buffer
(
\s+ # now have some white space following it
\1 # and the word itself
)+ # repeat the space+word combo ad libitum
\b # make sure there's a boundary at the end too
}xgi # /x for space/comment-expanded patterns
# /g for global matching
# /i for case-insensitive matching
)
{
print "dup $1 at paragraph $NR\n";
}
While it's true that someone who doesn't know regular expressions
won't be able to read this at first glance, this is not a problem.
So even though we can build up rather complex patterns, we can
format and comment them nicely, preserving understandability.
I wonder why no one else has done this in their regexp libraries?

I actually wrote a sublegible version of this many years ago. It runs
even on ancient versions of Perl. I'd probably to that a bit differently
these days -- my coding style has certainly matured. It violates several
of my own current style guidelines.

#!/usr/bin/perl
undef $/; $* = 1;
while ( $ARGV = shift ) {
if (!open ARGV) { warn "$ARGV: $!\n"; next; }
$_ = ;
s/\b(\s?)(([A-Za-z]\w*)(\s+\3)+\b)/$1\200$2\200/gi || next;
split(/\n/);
$NR = 0;
@hits = ();
for (@_) {
$NR++;
push(@hits, sprintf("%5d %s", $NR, $_)) if /\200/;
}
$_ = join("\n",@hits);
s/\200([^\200]+)\200/[* $1 *]/g;
print "$ARGV:\n$_\n";
}
here's that will output when run on this
article up to this current point:
51 5. [* Is is *] outlawed?
124 In fact, that one matches multiple duplicates as well. [* If
125 if if *] you read in your input data a paragraph at a time, it will
126 catch dups crossing line boundaries [* as as *] well. For example, using
Which is pretty neat.

Speaking of ctype.h macros, Perl borrows the vi
notation of case translation
via \u, \l, \U, and \L.
So you could say

Speaking of substitutes, we can use a /e flag on the substitute
to get the RHS to evaluate to code instead of just a string.
Consider:

s/(\d+)/8 * $1/ge; # multiple all numbers by 8
s/(\d+)/sprintf("%x", $1)/ge; # convert them to hex
This is nice when renumbering paragraphs. I often write
s/^(\d+)/1 + $1/
or from within vi, just
%!perl -pe 's/^(\d+)/1 + $1/'
Here's a more elaborate example of this.
If you wanted to expand %d or %s or
whatnot, you might just do
s/%(.)/$percent{$1}/g;
given a %percent definition like this:
%percent = (
'd' => 'digit',
's' => 'string',
);
But in fact, that's got quite enough. You might well want
to call a function, like
s/%(.)/unpercent($1)/ge;
(assuming you have an unpercent() function defined.)

You can even use /ee for a double-eval, but that seems
going overboard in
most cases. It is, however, nice for converting embedded variables like
$foo or whatever in text into their values.
This way a sentence with
$HOME and $TERM in it, assuming there were
valid variables, might become a
sentence with /home/tchrist and xterm in it. Just do this:

s/(\$\w+)/$1/eeg;
Ok, what more can we do with perl patterns? split
takes a pattern.
Imagine that you have a record stored in plain text as blank line
separated paragraphs with FIELD: VALUE pairs on each line.
field: value here
somefield: some value here
morefield: other value here
field: second record's value here
somefield: some value here
morefield: other value here
newfield: other funny stuff
You could process that this way. We'll put it into key value
pairs in a hash, just as though it had been initialized as
%hash = (
'field' => 'value here',
'somefield' => 'some value here',
'morefield' => 'other value here',
);
I'll use a few command line switches for short cuts:
#!/usr/bin/perl -00n
%hash = split( /^([^:]+):\s*/m );
if ( $hash{"somefield"} =~ /here/) {
print "record $. has here in somefield\n";
}
The /m flag governs whether ^ can
match internally. I
believe this is the POSIX value REG_NEWLINE. Normally
perl does not have ^ match anywhere but the beginning
of the string. (

$line = "The food is under the bar in the barn.";
if ( $line =~ /foo(.*)bar/ ) {
print "got \n";
}
That prints out
Which is often not what you want. Instead, we can add an extra ?
after a repetition operator to render it stingy instead of greedy.
if ( $line =~ /foo(.*?)bar/ ) {
print "got \n";
}
That prints out
got
which is often more what folks want. It turns out that having both
stringy and greedy repetition operators in no way compromises a
regexp engines regularity, nor is it particularly hard to implement.
This comes up in matching quoted things. You can do tricks like
using [^:] or [^"] or [^"']
for the simple cases, but negating
multicharacter strings is hard. You can just use stingy matching
instead.

Or you could just use lookaheads.

This is other important aspect of perl matching I wanted to mention.
These are 0-width assertions that state that what follows must match or
must not match a particular thing. These are phrased as either
(?=pattern) for the assertion or (?!pattern)
for the negation.

/\bfoo(?!bar)\w+/
That will match "foostuff" but not "foo" or "foobar",
because I said
there must be some alphanums after the word foo, but these may
not begin with bar.

Why would you need this? Oh, there are lots of times. Imagine splitting
on newlines that are not followed by a space or a tab:

@list_of_results = split ( /\n(?![\t ])/, $data );
Let's put this all together and look at a couple of examples. Both have
to do with HTML munging, the current rage. First, let's solve the problem
of detecting URLs in plaintext and highlighting them properly. A problem
is if the URL has trailling punctuation, like ftp://host/path.file. Is
that last dot supposed to be in the URL? We can probably just assume that
a trailing dot doesn't count, but even so, most scanners seem to get this
wrong. Here's a different approach:
#!/usr/bin/perl
# urlify -- tchrist@perl.com
require 5.002; # well, or 5.000 if you see below
$urls = '(' . join ('|', qw{
http
telnet
gopher
file
wais
ftp
} )
. ')';
$ltrs = '\w';
$gunk = '/#~:.?+=&%@!\-';
$punc = '.:?\-';
$any = "${ltrs}${gunk}${punc}";
while (<>) {
## use this if early-ish perl5 (pre 5.002)
## s{\b(${urls}:[$any]+?)(?=[$punc]*[^$any]|\Z)}{$1}goi;
## otherwise use this -- it just has 5.002ish comments
s{
\b # start at word boundary
( # begin $1 {
$urls : # need resource and a colon
[$any] +? # followed by on or more
# of any valid character, but
# be conservative and take only
# what you need to....
) # end $1 }
(?= # look-ahead non-consumptive assertion
[$punc]* # either 0 or more puntuation
[^$any] # followed by a non-url char
| # or else
$ # then end of the string
)
}{$1}igox;
print;
}
Pretty nifty, eh? :-)

Here's another HTML thing: we have an html document, and we
want to remove all of its embedded markup text. This requires
three steps:

Wow! I really can't believe that anyone has read this far
in this very long news posting about irregular expressions. :-)
Is anyone really still with me? If so, make my day and
drop me a piece of email.

What's missing from Perl's regular expressions? Anything? Well, yes.
The first is that they should be first-class objects. There are some
really embarassing optimization hacks to get around not having compiled
regepxs directly-usable accessible. The /o flag I used above
is just one
of them. (I'm *not* talking about the study() function,
which is a neat
thing to turbo-ize your matching.) A much more egregious hack
involving closures is demonstrated here using the match_any
funtion, which itself returns a function to do the work:
$f = match_any('^begin', 'end$', 'middle');
while (<>) {
print if &$f();
}
sub match_any {
die "usage: match_any pats" unless @_;
my $code = < 5;
study;
EOCODE
for $pat (@_) {
$code .= <
That's the kind of thing I just despise writing: the only thing
worse would be not being able to do it at all. :-( 1st-class
compiled regexps would surely help a great deal here.

Sometimes people expect backreferences to be forward references, as in the
pattern /\1\s(\w+)/, which just isn't the way it works.
A related issue
is that while lookaheads work, these are not lookbehinds, which can
confuse people. This means /\n(?=\s)/ is ok, but you
cannot use this for
lookbehind: /(?!foo)bar/ will not find an occurrence of
"bar" that is
preceded by something which is not "foo". That's because the
(?!foo) is
just saying that the next thing cannot be "foo"--and it's not, it's a
"bar", so "foobar" will match.

There isn't really much support for user-defined character classes.
You see a bit of that in the urlify program above. On the other
hand, this might be the most clear way of writing it.

Another thing that would be nice to have is the ability to someone
specify a recursive match with nesting. That ways you could pull out
matching parens or braces or begin/end blocks etc. I don't know what a
good syntax for this might be. Maybe (?{...) for the opening
one and (?}...) for the closing one, as in:

/\b(?{begin)\b.*\b(?}end)\b/i
Finally, while it's cool that perl's patterns are 8-bit clean,
will match strings even with null bytes in them, and have
support for alternate 8-bit character sets, it would certainly
make the world happy if there were full Unicode support.