Tensor TownISSUE #58 of 120

rlhf · reinforcement learning from human feedback

NeuraVSThe Overfit Ogre

Neura saysRLHF aligns a model to human preferences using reinforcement learning from human feedback.

A base model is knowledgeable but not aligned. RLHF: humans rank model outputs, those rankings train a reward model, and the LLM is then optimized (via RL, e.g. PPO) to produce responses the reward model scores highly. This is what turned raw text-predictors into helpful, harmless assistants. It's powerful but complex, and reward-model flaws can be gamed (reward hacking, sycophancy).

Power-ups you unlock

Humans rank outputs → train a reward model
RL optimizes the LLM toward high reward
Turned predictors into helpful assistants
Reward flaws → hacking, sycophancy

The Overfit Ogre attacks — common mistakes

Assuming RLHF makes a model "correct"
Ignoring reward-model bias
Over-optimizing → sycophantic/gamed behavior

Boss battleTrace the RLHF pipeline from human rankings to an aligned model.

Example code

<!doctype html><html><head><meta charset="utf-8"></head>
<body style="background:#06040d;color:#e6e0ff;font-family:monospace;padding:20px"><pre>humans rank answers → reward model
RL (PPO) optimizes LLM toward reward
→ helpful, harmless assistant</pre></body></html>

▶ Open the interactive comic issue

‹ Lora · Parameter-Efficient Tuning Dpo · Direct Preference Optimization ›