Mixup - "Little" language for creating annotations for text.

At the simplest level, Mixup is a pattern language for Spans (i.e. token
sequences). Each expression is defined relative to labeling called a {@link
edu.cmu.minorthird.text.TextLabels}. From the ground up:

A "simple pattern component" (SPC) matches a single token. The SPC's
include:

any matches any token.

eq('foo') matches the token foo. This can also be abbreviated
as 'foo' (in single quotes).

a(foo) matches any token whose string value in the dictionary
named foo. Dictionaries are defined in a TextLabels. For example,
a(weekday) might match one of 'sun', 'mon', 'tues', ..., 'sat'.

foo:bar matches any token that has been tagged as having the value
bar for the property foo. For example, pos:det
might match a determiner.

SPC's can be negated by prefixing them with a bang (!). A conjunction
of (optionally, negated) SPC's can be formed with angle brackets and commas,
for instance: <a(month),!may> might match any of 'jan', 'feb',
..., 'april', 'june', ..., or 'december'.

A "repeated pattern component" (RPC) matches a sequence of adjacent
tokens. An RPC is formed by appending one of the regex-like postfix
operators *, +, ?, or {i,j} (where
i and j are numbers) to a SPC. The RPC any*
can be abbreviated as .... An RPC matches any sequence of between
i and j tokens such that every token in the sequence
matches the underlying SPC. For example:

a(name){1,3} matches any sequences of 1-3 tokens in the 'name'
dictionary.

<!a(punct),!'and'>* matches any sequence of tokens that are
not in the 'punct' dictionary and are not the token 'and'.

pos:noun? matches the a one-token sequence with the 'pos' property
set to 'noun', or an empty sequence.

A "repeated pattern component" (RPC) can also be preceded by the token
'L' or followed by the token 'R'. An RPC modified by a 'L' matches unless
the sequence it corresponds to can be extended one token to the left, and
still match. An RPC modified by a 'R' is analogous, but can't be extended
to the right. For instance:

pos:adj+

matches any sequence of adjectives (if that's what 'pos:adj' means).
However, L pos:adj+ only matches a sequence of adjectives that
does NOT have an adjective immediately to the left of it.

any{3,5}

matches any sequence of 3-5 tokens. However, any{3,5} R only
matches a sequence of 3-5 tokens that can't be extended to the right---in
other words, a sequence that is either exactly 5 tokens long, or which
ends with the final token of a document.

A "repeated pattern component" (RPC) can also be either @foo or
@foo?, where foo is a type. The RPC @foo matches
a span of type 'foo'. The RPC @foo?matches either a span of type
foo or an empty sequence.

A "mixup pattern" is a bunch of RPC's concatenated together. A mixup pattern
matches a token sequence if all tokens in the sequence match up with some
RPC. For instance:

... ',' 'Ph' '.' 'D' matches any token sequence ending in ", Ph.D".

... '(' !eq(')'){,10} ')' ... matches any sequence containing
a parenthesized expression with less than 10 tokens in it.

Returning for a moment to the 'L' and 'R' operators, which say that a matched
sequence can't be extended to the left of right...note that "can't be extended"
can be interpreted two ways: either (a) any extension causes that RPC to
fail to match or (b) any extension causes that RPC to fail to match, or
else causes some other RPC pattern elsewhere in mixup pattern to fail.
The implemenentation current adopts the first interpretation, (a).

Extraction with Mixup

Mixup is normally used for extraction, not matching. For extraction, every
Mixup expression should contain matching left and right square brackets.
For each Span that the expression is matched against, and for every
possible way the expression can be matched, a subspan of the tokens
matching the RPCS's inside the square brackets will be extracted.

For example:

... a(endOfSent) [ re('^[A-Z]') !a(endOfSent){3,} a(endOfSent)] ...
will extract "sentences" (roughly - really, every sequence of at least
three words between things in the endOfSent dictionary.)

... [any any] ... will extract all token bi-grams.

Mixup Programs

The MixupProgram class allows a series of statements to be executed, one
after another, in order to modify a text labeling. Most of these statements
are based on evaluating Mixup patterns, and then modifying the labels in
response to those patterns.
The types of Mixup statements are:

defDict D = W1,W2,...,Wk: adds words W1...Wk to dictionary D.
If Wi is in double quotes, then Wi is interpreted as a filename, and all
lines from that file are loaded in the dictionary.

provide ANNOTATION_TYPE: puts a marker in the labels that annotations
of the given type are present.

require ANNOTATION_TYPE,FILE: sees if annotations of the given
type are present in the current labels. If not, the mixup program in 'file'
is executed. (File might be in single quotes.)

defSpanType TYPE SPAN_GENERATOR: adds all spans generated by the
SPAN_GENERATOR to the given TYPE. There are several types of SPAN_GENERATOR's.

=T: EXPR runs the Mixup expression EXPR on every span of type
T, and returns all spans extracted by it.

=T- EXPR runs the Mixup expression EXPR on every span of type
T, and returns all spans S in T such that nothing was successfully extracted
by EXPR.

=T~ re REGEX,N runs the Java 1.4 regular expression REGEX on the
string associated with each span S in T, and returns the span associated
with the N-th group in that REGEX. If the N-th group of the regex matches
something that doesn't align with token boundaries, the closest legal token
span will be used instead.

defSpanProp PROP:VAL SPAN_GENERATOR: same, but asserts that property
PROP has value VAL for all generated spans.

defoTokenProp PROP:VAL SPAN_GENERATOR: same, but asserts that
property PROP has value VAL for all tokens contained in a generated span.