AI Scaling Laws Tutorial: Params, Data, Compute

TL;DRScaling laws: model quality improves predictably with more parameters, data, and compute.

Empirical scaling laws show loss falls as a smooth power law as you increase model size, dataset size, and compute together. This predictability — bigger + more data + more compute = better — drove the race to ever-larger models. The Chinchilla result refined it: for a compute budget, many models were under-trained on data; balance parameters and tokens. Scaling, not just clever ideas, drove recent leaps.

Key points

Loss falls as a power law with scale
Parameters + data + compute together
Predictability fueled the scale-up race
Chinchilla: balance params and tokens

Common mistakes

Adding parameters without enough data (Chinchilla lesson)
Assuming scaling improves everything equally
Expecting scaling laws to hold forever

Try it: State the Chinchilla insight about balancing model size and data.

Example code

<!doctype html><html><head><meta charset="utf-8"></head>
<body style="background:#06040d;color:#e6e0ff;font-family:monospace;padding:20px"><pre>more params + data + compute → lower loss (power law)
Chinchilla: don’t over-size params, feed enough tokens</pre></body></html>

Open the interactive lesson →

← Context Windows · How Much Fits at Once Emergent Abilities · The Surprise Phase →