Tensor TownISSUE #52 of 120

context windows · how much fits at once

NeuraVSThe Overfit Ogre

Neura saysThe context window is how many tokens a model can consider at once — its working memory.

A model can only attend to a fixed number of tokens — its context window (from a few thousand to millions in modern models). Everything the model "knows" in a conversation must fit: system prompt, history, documents, and the reply. Exceed it and earlier content is truncated or must be summarized. Bigger windows cost more compute (attention scales with length), so there's a price for using them.

Power-ups you unlock

Max tokens the model attends to at once
Holds prompt + history + docs + reply
Overflow → truncation or summarization
Bigger window = more compute/cost

The Overfit Ogre attacks — common mistakes

Assuming unlimited memory across a chat
Stuffing the window and paying for waste
Forgetting attention cost grows with length

Boss battleList what must collectively fit inside the context window.

Example code

<!doctype html><html><head><meta charset="utf-8"></head>
<body style="background:#06040d;color:#e6e0ff;font-family:monospace;padding:20px"><pre>context window = system + history + docs + answer
exceed it → oldest content drops
bigger window → more compute/cost</pre></body></html>

▶ Open the interactive comic issue

‹ Tokenization · Bpe · Wordpiece · Unigram Scaling Laws · Params · Data · Compute ›