Attributes

Description

A token identifies a unit of information.
Usually, “tokens” are the result of some processing pass
that has performed lexical analysis and divided a data set into the
smallest units of information used for subsequent processing.