Well, the short answer is that you lose. No version of lex checks for
token buffer overflow, although it is less destructive in flex than it is
in lex. This can be called a lex design error, though it's hard to
check for overflow without making the lexer a lot slower. The solution
is not to use tokens that can be arbitrarily long.

If you switch to flex (highly recommended) it can handle up to 8K per
token with no trouble. In AT&T lex, you can redefine YYLMAX, the size of
the token buffer, yourself at the front of the lex file.

The other possibility is to use start states to build the string
yourself: