Tokenization

Tokenization is the process of breaking text into smaller pieces called tokens — words, parts of words, or characters — so a language model can process it.