Introduction

The token type in Wave is the main carrier of information. It is returned
by dereferencing the iterators exposed by the lexing component and the iterator exposed by the preprocessor component. The tokens are originally generated by the
lexer ("An entity that lexically transforms the subject of parsing to a
sequence of objects (called tokens) more suitable for subsequent parsing.").
The Wave library contains two different, interchangable C++ lexers, which may
be used as a starting point during developing your own application. The lexer
generated tokens are transformed by the preprocessing engine (macro expansion,
include file processing etc.) and after this returned to the user of the Wave
library.

You can use arbitrary token types in conjunction with your own lexer as long
as these implement some required interface. The required token type interface
is described below and is implemented by the wave::cpplexer::lex_token
template, the required lexer interface is described here.

In the following is described the token type predefined inside the Wave
library, which is used in conjunction with the two predefined C++ lexers contained
in the Wave library too. If you need to use your own token type, it
is required to implement the interafce described below.

Template parameters

The predefined token type uses a template parameter Position, which
allows to specify the type to be used to carry the file position information
contained inside the generated tokens. Your own token type do not need to take this Position template parameter, but please note, that the token type in any case needs to have an embedded type definition position_type (see below) .

Public Typedefs

The token type needs to define two embedded types: string_type and position_type. The string_type needs to be a type compatible to the std::basic_string<> class.

This type should contain at least the filename, the line number and the column number of the position, where the token was recognized. For the predefined token type it defaults to a simple file_position template class described here. Note, that your own position_type should follow the interface described for the file_position template as well.

Member functions

The first (default) constructor is for generating an end of stream token,
which is used for indicating the end of the underlying input stream.

The second constructor initializes the newly created token object with its
token id (for a list of valid token id's please look here),
the string representation of its value and the file position, describing the
position inside the input stream , where this token was recognized.

Accessor functions

operator token_id

operator token_id() const;

Allows the access to the token id of the token. This accessor allows the usage of Spirit
parsers directly on top
top of a token stream generated by Wave. The possible token id's are described
here.

This function does not throw in any case.

get_value

string_type const &get_value() const;

Returns the value of the token, as it was recognized in the input stream.
Even for constant tokens (as keywords or operators etc.) the returned value
reflects the character sequence as found in the input stream.

This function does not throw in any case.

get_position

Position const &get_position() const;

Returns the position of the token in the input stream, where it was recognized.
The position contains information about the filename, the line number and
the column number of the token. By default the Wave library uses
a file_position template for this purpose, which is described in more detail
here.

This function does not throw in any case.

set_token_id

void set_token_id(token_id id);

Changes the token id of the token to the new value. The possible token id's
are described here. Probably this function is
of little value for the library user, but it is required under certain circumstances
for correct operation of the preprocessing engine.

This function does not throw in any case.

set_value

void set_value(string_type const &newval);

Changes the value stored inside the token to the new value. Propably this
function is of little value for the library user, but it is required under
certain circumstances for correct operation of the preprocessing engine.

set_position

void set_position(Position const &newpos);

Changes the position stored inside the token to the new value. This is used
for instance for implementing the functionality required for to implement
the #line directive.