AI Optimizers Tutorial: Sgd, Adam, Adamw

TL;DROptimizers like SGD, Adam, and AdamW improve on plain gradient descent.

Beyond vanilla SGD, momentum accelerates consistent directions and damps oscillation; Adam adapts a per-parameter learning rate using gradient statistics (fast, robust, the common default); AdamW fixes Adam's weight-decay handling and is standard for training transformers. The optimizer shapes how fast and how well you converge — Adam/AdamW are safe starting points.

Key points

SGD + momentum: accelerate, damp oscillation
Adam: per-parameter adaptive rate (robust default)
AdamW: correct weight decay (transformer standard)
Optimizer affects speed and final quality

Common mistakes

Hand-tuning SGD when Adam would just work
Using Adam where AdamW’s decay matters
Ignoring optimizer choice entirely

Try it: Explain what Adam adapts that plain SGD does not.

Example code

<!doctype html><html><head><meta charset="utf-8"></head>
<body style="background:#06040d;color:#e6e0ff;font-family:monospace;padding:20px"><pre>SGD: one global learning rate
Adam: per-parameter adaptive rate (uses grad stats)
AdamW: Adam + proper weight decay → transformers</pre></body></html>

Open the interactive lesson →

← Learning Rate · The Most Important Knob Regularization · L1 · L2 · Dropout →