A token is a group of characters (like a word) which is separated by a delimiter. A delimiter is a symbol which separates tokens from each other. The process of tokenizing is extracting tokens from a string or a stream.

"a b cde"

If you define your delimiter to be a single space, ' ', then you have three tokens: "a", "b", and "cde". Using whitespace as a delimiter makes tokenizing easy. Stringstreams and input streams have the ability to tokenize based on this.

Sometimes you want to ignore some areas of your string. What if your delimiter was a dash?

numbers_str = "forty-five-forty-six-forty-seven-forty-eight";

(Pretend that these numbers need the dashes, and that you need to use the '-' as a delimiter)

You could take two tokens and combine them yourself. That often is enough. But sometimes there are situations where you need more flexibility.

equation = "3+f(x+y)+4"

Simply parsing by plus signs won't do because it would tear apart your function f(x+y). In situtations like these it's often easiest to grab the area in between parentheses before your tokenizing function can get into it.