Tensor TownISSUE #49 of 120

decoder-only · gpt family

NeuraVSThe Overfit Ogre

Neura saysDecoder-only models (GPT) predict the next token left-to-right — the basis of chat LLMs.

GPT-style models use only the decoder with causal (masked) attention: each token sees only what came before, so they generate text one token at a time. Pretrained on next-token prediction over huge corpora, then instruction-tuned, they became the dominant LLM design — GPT, Claude, Llama, Gemini are all decoder-only. Generation, chat, code, reasoning all flow from this single objective.

Power-ups you unlock

Decoder-only, causal (left-to-right) attention
Generate one token at a time
Pretrained on next-token prediction
The dominant LLM architecture

The Overfit Ogre attacks — common mistakes

Expecting bidirectional context (it’s causal)
Thinking next-token prediction is "simple"/limited
Confusing pretraining with instruction-tuning

Boss battleExplain why causal masking is required for generation.

Example code

<!doctype html><html><head><meta charset="utf-8"></head>
<body style="background:#06040d;color:#e6e0ff;font-family:monospace;padding:20px"><pre>token t sees only tokens &lt; t (causal mask)
predict next → append → repeat
GPT/Claude/Llama all decoder-only</pre></body></html>

▶ Open the interactive comic issue

‹ Encoder-Only · Bert Family Encoder-Decoder · T5 · Bart ›