Tensor TownISSUE #54 of 120

emergent abilities · the surprise phase

NeuraVSThe Overfit Ogre

Neura saysEmergent abilities appear suddenly at scale — capabilities small models simply don’t have.

As models grow, some abilities (multi-step arithmetic, in-context learning, instruction following) seem to switch on past a size threshold rather than improving gradually. These emergent behaviors surprised researchers and are part of why scaling was pursued so aggressively. There's debate about how "sharp" emergence really is (it can depend on the metric), but the practical point stands: scale unlocked qualitatively new capabilities.

Power-ups you unlock

Some abilities appear only past a size threshold
In-context learning, multi-step reasoning
Surprised researchers; motivated scaling
Sharpness debated, but scale unlocked new skills

The Overfit Ogre attacks — common mistakes

Assuming all skills scale smoothly
Treating emergence as magic vs measurable
Over-claiming sharp thresholds without metric care

Boss battleName an ability that emerged at scale rather than improving gradually.

Example code

<!doctype html><html><head><meta charset="utf-8"></head>
<body style="background:#06040d;color:#e6e0ff;font-family:monospace;padding:20px"><pre>small model: can’t do multi-step arithmetic
past a size threshold → it suddenly can
= emergent ability</pre></body></html>

▶ Open the interactive comic issue

‹ Scaling Laws · Params · Data · Compute Pretraining · Next-Token Prediction ›