When you download the svn source code of code::blocks,(see here [[Installing_Code::Blocks_from_source_on_Windows#Code::Blocks_sources]] the source code of this plugin was already included. See a screenshot of these code opened in code::blocks under windows.

For someone havn't heard what does "Token" and "Tokenize" means, you should read the wikibooks article [http://en.wikibooks.org/wiki/C%2B%2B_Programming/Compiler#Compilation A brief explain of what does a parser do] and [http://en.wikipedia.org/wiki/Lexical_analysis Tokenize on wikipedia]. Shortly, a parser treats your C++ or C code as a large array of characters, then this big string was divided to small atom strings, meanwhile "spaces" and "comments" were ignored.

−

===Tokenizer class===

−

There are several steps to running the Tokenizer class

−

*A thread must be created to parse a source file.

−

*Open the source file and convert the file buff to Unicode mode.(since we are all using Unicode build of code::blocks, and ANSI mode is outdated).

−

*The class contains a Pointer to the current position of the character, so, you can Get or Peek the current character.

−

*Nested Value was kept to indicate your are in the correct brace pair.

−

==High level parser==

−

Each identifier will be recorded and saved in the memory for later usage.