2.1 How Lexical Tokens Match

A lexical rule must be used to define how to match a lexical token.

For instance:

%keyword FOO "foo"

Means that FOO is a reserved language keyword, matched as such
by looking up into a keyword table, See (grammar-fw)keyword Decl. This is because "foo" will be converted to
FOO in the lexical analysis stage. Thus the symbol FOO
won’t be available any other way.

If we specify our token in this way:

%token <symbol> FOO "foo"

then FOO will match the string "foo" explicitly, but it
won’t do so at the lexical level, allowing use of the text
"foo" in other forms of regular expressions.

In that case, FOO is a symbol-type token. To match, a
symbol must first be encountered, and then it must
string-match "foo".

Caution:

Be especially careful to remember that "foo", and more
generally the %token’s match-value string, is a regular expression!

Non symbol tokens are also allowed. For example:

%token <punctuation> PERIOD "[.]"
filename : symbol PERIOD symbol
;

PERIOD is a punctuation-type token that will explicitly
match one period when used in the above rule.

Please Note:

symbol, punctuation, etc., are predefined lexical token
types, based on the syntax class-character associations
currently in effect.