Pretraining Tutorial: Next-Token Prediction

TL;DRPretraining teaches a base model language and knowledge via next-token prediction at scale.

The expensive first phase: train on trillions of tokens of text/code, predicting the next token, until the model internalizes grammar, facts, reasoning patterns, and style. The result is a base model — knowledgeable but not yet helpful or safe; it just continues text. Pretraining costs millions of dollars and is done once; everything after (tuning) is comparatively cheap refinement.

Key points

Common mistakes

Try it: Explain what a base model can and cannot do before tuning.

Example code

<!doctype html><html><head><meta charset="utf-8"></head>
<body style="background:#06040d;color:#e6e0ff;font-family:monospace;padding:20px"><pre>pretrain: trillions of tokens, predict next
→ base model: knows a lot, just continues text
not yet helpful/safe → needs tuning</pre></body></html>
Open the interactive lesson →
Emergent Abilities · The Surprise Phase Fine-Tuning · Adapting a Base Model