Tensor TownISSUE #40 of 120

lstms · gates · long-term memory

NeuraVSThe Overfit Ogre

Neura saysLSTMs add gates and a cell state to remember information over long sequences.

Long Short-Term Memory networks fix the vanishing-gradient problem with a cell state (a memory conveyor) and three gates (forget, input, output) that learn what to keep, add, and emit. This lets them retain context across hundreds of steps. LSTMs powered the pre-transformer era of translation and speech, and still appear where sequences are long but data/compute is limited.

Power-ups you unlock

Cell state = long-term memory conveyor
Forget/input/output gates control it
Retain context over long sequences
Pre-transformer workhorse for sequences

The Overfit Ogre attacks — common mistakes

Reaching for LSTMs where transformers now dominate
Confusing the cell state with the hidden state
Underestimating their training cost

Boss battleName the three LSTM gates and what each decides.

Example code

<!doctype html><html><head><meta charset="utf-8"></head>
<body style="background:#06040d;color:#e6e0ff;font-family:monospace;padding:20px"><pre>cell state = memory line
forget gate → drop · input gate → add · output gate → emit
→ remembers across long sequences</pre></body></html>

▶ Open the interactive comic issue

‹ Rnns · Recurrence · Vanishing Gradients Grus · Simpler Gated Recurrence ›