TL;DRTokenization splits text into tokens — subword pieces the model actually processes.
Models don't see characters or words; they see tokens. A tokenizer (BPE, WordPiece, Unigram) splits text into common subword chunks — frequent words are one token, rare words split into pieces. This balances vocabulary size against sequence length. Token counts drive cost and context limits, and quirks of tokenization explain odd model behaviors (e.g. counting letters, arithmetic on digits).
Key points
Models process tokens, not characters/words
BPE/WordPiece/Unigram split into subwords
Token count drives cost + context use
Tokenization quirks explain odd failures
Common mistakes
Assuming one word = one token
Ignoring token counts in cost estimates
Blaming "reasoning" for tokenization artifacts
Try it: Explain why a model may struggle to count the letters in a word.
Example code
<!doctype html><html><head><meta charset="utf-8"></head>
<body style="background:#06040d;color:#e6e0ff;font-family:monospace;padding:20px"><pre>"tokenization" → ["token","ization"] (2 tokens)
model sees tokens, not letters
→ letter-counting is hard</pre></body></html>