The Parser class relies upon this method to retrieve the next token. The $wordseperators array is a collection of strings that delineate tokens for the current parser state. $wordseperators is set by the parser with a call to Parser::configWordParser() every time a new parser state is reached.

For example, while parsing the source code for a class, the word

var

is a token, and

global

is not, but inside a function, the reverse is true. The parser state PARSER_STATE_CLASS has a token list that includes whitespace, code delimiters like ; and {}, and comment/DocBlock indicators

If the whitespace option has been turned off using setWhitespace(), then no whitespace is returned with tokens

{@ is of course the string containing the PHP code to be parsed, and $pos is the cursor, or current location within the parsed data. }}

return: the next token, an empty string if there are no token separators in the $wordseperators array, or false if the end of input has been reached