Tensor TownISSUE #46 of 120

multi-head attention · parallel views

NeuraVSThe Overfit Ogre

Neura saysMulti-head attention runs several attention operations in parallel to capture different relationships.

One attention "head" can only focus one way. Multi-head attention splits the representation into several heads, each learning its own Q/K/V projections — one head might track syntax, another long-range references, another local context. Their outputs are concatenated and combined. Multiple parallel perspectives are why transformers capture rich, varied relationships in a single layer.

Power-ups you unlock

Several attention heads run in parallel
Each head learns a different relationship
Outputs concatenated and combined
Captures varied patterns per layer

The Overfit Ogre attacks — common mistakes

Assuming one head suffices
Thinking heads are redundant copies
Ignoring head dimensionality trade-offs

Boss battleExplain what different attention heads might specialize in.

Example code

<!doctype html><html><head><meta charset="utf-8"></head>
<body style="background:#06040d;color:#e6e0ff;font-family:monospace;padding:20px"><pre>head1: syntax · head2: long-range refs · head3: local
run in parallel → concat → richer representation</pre></body></html>

▶ Open the interactive comic issue

‹ Self-Attention · Queries · Keys · Values Positional Encoding · Giving Order Back ›