Tensor TownISSUE #56 of 120

fine-tuning · adapting a base model

NeuraVSThe Overfit Ogre

Neura saysFine-tuning adapts a pretrained model to a specific task or domain with more training.

Rather than train from scratch, you take a capable base model and train it further on a smaller, targeted dataset — legal documents, your product's tone, a classification task. It transfers the base's broad knowledge to your narrow need cheaply. Risks: catastrophic forgetting (losing general ability) and overfitting the small set. For many use cases, prompting or RAG beats fine-tuning — try those first.

Power-ups you unlock

Continue training a base model on targeted data
Transfers broad knowledge to a narrow task
Risks: forgetting + overfitting
Often prompting/RAG is enough — try first

The Overfit Ogre attacks — common mistakes

Fine-tuning when a prompt/RAG would do
Tiny datasets causing overfitting
Catastrophic forgetting of general skills

Boss battleDecide fine-tune vs prompt vs RAG for "answer questions about our docs".

Example code

<!doctype html><html><head><meta charset="utf-8"></head>
<body style="background:#06040d;color:#e6e0ff;font-family:monospace;padding:20px"><pre>need: answer from your docs
→ usually RAG (not fine-tune)
fine-tune: when you need a specific behavior/format/tone</pre></body></html>

▶ Open the interactive comic issue

‹ Pretraining · Next-Token Prediction Lora · Parameter-Efficient Tuning ›