TL;DRAttention lets the decoder look back at all input positions, weighted by relevance.
Instead of one fixed context vector, attention lets the model, at each output step, compute a weighted blend of all input positions — focusing on the relevant words. It dissolved the seq2seq bottleneck and produced interpretable alignment (which input word maps to which output). Attention was the key idea that, scaled up and used alone, became the transformer.
Key points
Weight all input positions by relevance
Dissolves the fixed-context bottleneck
Gives interpretable alignment
The seed of the transformer
Common mistakes
Thinking attention is only for translation
Confusing attention weights with certainty
Missing that it enabled long-range focus
Try it: Explain how attention fixes seq2seq’s bottleneck.
Example code
<!doctype html><html><head><meta charset="utf-8"></head>
<body style="background:#06040d;color:#e6e0ff;font-family:monospace;padding:20px"><pre>each output word → weighted look at ALL input words
focus where relevant → no single bottleneck vector</pre></body></html>