Tensor TownISSUE #39 of 120

rnns · recurrence · vanishing gradients

NeuraVSThe Overfit Ogre

Neura saysRNNs process sequences step by step, carrying a hidden state — but suffer vanishing gradients.

A Recurrent Neural Network reads a sequence one element at a time, updating a hidden state that carries context forward — natural for text, audio, time series. The problem: over long sequences, gradients shrink (vanish) during backprop, so early context is forgotten and training stalls. This limitation motivated gated variants (LSTM/GRU) and, ultimately, attention.

Power-ups you unlock

Process sequences step by step
Hidden state carries context forward
Vanishing gradients → forget long context
Motivated LSTMs and attention

The Overfit Ogre attacks — common mistakes

Using a vanilla RNN for long dependencies
Ignoring sequence length effects
Expecting parallel training (RNNs are sequential)

Boss battleExplain why long sequences cause vanilla RNNs to "forget".

Example code

<!doctype html><html><head><meta charset="utf-8"></head>
<body style="background:#06040d;color:#e6e0ff;font-family:monospace;padding:20px"><pre>x1 → h1 → x2 → h2 → ... (carry state)
long sequence → gradients vanish → early context lost</pre></body></html>

▶ Open the interactive comic issue

‹ Cnns · Filters · Pooling · Stride Lstms · Gates · Long-Term Memory ›