Tensor TownISSUE #84 of 120

inference · greedy · sampling · temperature

NeuraVSThe Overfit Ogre

Neura saysInference is running a trained model to get outputs — sampling controls how it picks tokens.

At inference time the model outputs a probability distribution over the next token; how you pick from it shapes the result. Greedy always takes the top token (deterministic, can be dull/repetitive). Sampling draws randomly by probability. Temperature scales the randomness — low = focused/deterministic, high = creative/risky. Choosing decoding is how you trade reliability against creativity.

Power-ups you unlock

Model emits a next-token distribution
Greedy: always top token (deterministic)
Sampling: draw by probability
Temperature scales randomness

The Overfit Ogre attacks — common mistakes

High temperature for factual/structured tasks
Greedy where diversity is wanted
Ignoring decoding settings entirely

Boss battleChoose a temperature for code generation vs brainstorming.

Example code

<!doctype html><html><head><meta charset="utf-8"></head>
<body style="background:#06040d;color:#e6e0ff;font-family:monospace;padding:20px"><pre>temp 0   → focused, deterministic (code, facts)
temp 0.9 → creative, varied (brainstorm)
greedy = always top token</pre></body></html>

▶ Open the interactive comic issue

‹ Alignment · The Meta-Objective Top-K · Top-P · Repetition Penalty ›