Neura saysPretraining teaches a base model language and knowledge via next-token prediction at scale.
The expensive first phase: train on trillions of tokens of text/code, predicting the next token, until the model internalizes grammar, facts, reasoning patterns, and style. The result is a base model — knowledgeable but not yet helpful or safe; it just continues text. Pretraining costs millions of dollars and is done once; everything after (tuning) is comparatively cheap refinement.
Power-ups you unlock
Train on trillions of tokens, predict next token
Learns grammar, facts, reasoning, style
Produces a base model (knowledgeable, not aligned)
Expensive and done once
The Overfit Ogre attacks — common mistakes
Expecting a base model to follow instructions
Confusing pretraining with fine-tuning
Underestimating the compute cost
Boss battleExplain what a base model can and cannot do before tuning.
Example code
<!doctype html><html><head><meta charset="utf-8"></head>
<body style="background:#06040d;color:#e6e0ff;font-family:monospace;padding:20px"><pre>pretrain: trillions of tokens, predict next
→ base model: knows a lot, just continues text
not yet helpful/safe → needs tuning</pre></body></html>