Tokenization Strategies

By the end of this lesson you'll be able to split text into tokens by hand, run one Byte-Pair Encoding merge, read real token IDs from a live tokenizer, and explain why token count drives both cost and context length.

Learn Tokenization Strategies in our free AI & Machine Learning course — a beginner-friendly interactive lesson with worked examples, a practice exercise and…

Part of the free AI & Machine Learning course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

Imagine you have to build any sentence out of Lego. You have three choices. Keep a giant bin of pre-built words (fast, but you have no brick for a word you've never seen). Keep only single studs — one per character (you can build anything, but even "the" takes three pieces and sequences get painfully long). Or keep a smart middle bin of common chunks : whole bricks for frequent words like "the" and "ing", and smaller pieces you snap together for rare words.

That middle bin is subword tokenization , and it is what every modern LLM uses. A model first learns which chunks are worth keeping by scanning huge amounts of text, then reuses those chunks to encode any new sentence — even words it has never seen, which it simply spells out from smaller pieces.

A token is the smallest chunk of text a model reads. Tokenization is the process of cutting text into those chunks. There are three families of approach, and each trades vocabulary size against sequence length.

One token per word. Short sequences, but a huge vocabulary and no token at all for unseen words (the "out-of-vocabulary" problem).

One token per character. Tiny vocabulary and never stuck on a new word, but sequences get very long and meaning is spread thin.

Common words stay whole; rare words split into pieces. Moderate vocabulary, moderate length, and no out-of-vocabulary words.

The example below builds a tiny tokenizer by hand so you can see exactly what "splitting text" means, then runs the single most important step of subword learning.

BPE is the tokenizer behind GPT and LLaMA. The idea is simple enough to do on paper. You just repeat one rule until your vocabulary is the size you want:

Each merge adds one entry to the vocabulary. After thousands of merges the frequent words and fragments ("the", "ing", "tion") have become single tokens, while anything rare still decomposes into smaller learned pieces — so the model is never stuck on a new word. You already ran step 1 through 3 once in the example above; the only thing a real trainer adds is a loop.

BPE is one subword algorithm; two close cousins power most other models. They differ in how they pick merges and how they mark where words begin.

Instead of merging the most frequent pair, it merges the pair that most increases the likelihood of the training data. Continuation pieces get a ## prefix, so "tokenization" becomes token + ##ization .

Works directly on the raw text stream with no whitespace pre-splitting, so it is truly language-agnostic (great for Chinese or Japanese, which have no spaces). It marks the start of a word with an underscore ▁ instead of ## .

Below, a real Hugging Face tokenizer shows the ## continuation prefix and the special tokens BERT wraps around your text. Read the # Expected output comments to see exactly what each call returns.

This needs pip install transformers , so it is read-only here — but every line is annotated with what it returns. Notice the [CLS] and [SEP] markers that encode() adds automatically.

The output of training is a vocabulary : a fixed list of every token the model knows, each paired with a unique integer token ID . The model never works with text — it works with these IDs. Encoding turns text into a list of IDs; decoding turns IDs back into text.

A handful of IDs are reserved for special tokens that carry structure rather than meaning. They never appear in ordinary text:

The example below uses OpenAI's tiktoken — the actual tokenizer GPT-4 uses — so you can see real token IDs and where a leading space lives.

Token count is not an academic detail — it is what you pay for and what limits how much the model can read at once. Two hard constraints flow directly from it:

A useful rule of thumb for English: 1 token ≈ 4 characters ≈ 0.75 words . Code, long numbers, and non-English scripts use far more tokens per word, so always measure rather than guess. The exercise below turns a token count into a dollar cost and a remaining-context budget.

Counting words to estimate tokens undercounts badly.

✅ Fix: 1 token ≈ 0.75 words in English; measure code and non-English text with the real tokenizer, where "fibonacci" can be 3+ tokens.

❌ "The tokenizer will choke on a word it's never seen" (OOV)

Word-level vocabularies hit out-of-vocabulary (OOV) words and fail. People assume subword tokenizers do too.

✅ Fix: BPE / WordPiece / SentencePiece have no OOV — an unknown word just decomposes into smaller subwords, falling back to single characters or bytes if needed.

Tokenizers are trained mostly on English, so other scripts fragment heavily.

✅ Fix: expect multilingual text to use 2–3× more tokens per character; a single Chinese, Japanese, or emoji character can become several tokens, raising cost and shrinking effective context.

Byte-level BPE attaches the space to the following word, so "hello" and "hello " (or " hello" ) can tokenize differently.

✅ Fix: be deliberate about leading/trailing spaces in prompts and stop sequences — an unexpected trailing-space token can break exact string matching.

Rule of thumb (English): 1 token ≈ 4 characters ≈ 0.75 words. Common special tokens: [CLS] , [SEP] (BERT) and <|endoftext|> / <eos> (GPT).

Put it all together. Tokenize a sentence, assign each unique token an integer ID, then re-encode the sentence as a list of those IDs — exactly what a real tokenizer does, in miniature. The outline is below; the logic is up to you.

Lesson 26 complete — you can read how an LLM reads!

You tokenized text by hand, ran a Byte-Pair Encoding merge, compared BPE with WordPiece and SentencePiece, saw real token IDs and special tokens, and turned token counts into cost and context budgets. This is the layer every prompt passes through before the model ever sees it.

🚀 Up next: Fine-Tuning LLMs — adapt a pre-trained model to your own data with techniques like LoRA.

Practice quiz

• Browse all free courses
• Cheat sheets & quizzes
• Coding dictionary