AI Rnns Tutorial: Recurrence, Vanishing Gradients

TL;DRRNNs process sequences step by step, carrying a hidden state — but suffer vanishing gradients.

A Recurrent Neural Network reads a sequence one element at a time, updating a hidden state that carries context forward — natural for text, audio, time series. The problem: over long sequences, gradients shrink (vanish) during backprop, so early context is forgotten and training stalls. This limitation motivated gated variants (LSTM/GRU) and, ultimately, attention.

Key points

Process sequences step by step
Hidden state carries context forward
Vanishing gradients → forget long context
Motivated LSTMs and attention

Common mistakes

Using a vanilla RNN for long dependencies
Ignoring sequence length effects
Expecting parallel training (RNNs are sequential)

Try it: Explain why long sequences cause vanilla RNNs to "forget".

Example code

<!doctype html><html><head><meta charset="utf-8"></head>
<body style="background:#06040d;color:#e6e0ff;font-family:monospace;padding:20px"><pre>x1 → h1 → x2 → h2 → ... (carry state)
long sequence → gradients vanish → early context lost</pre></body></html>

Open the interactive lesson →

← Cnns · Filters · Pooling · Stride Lstms · Gates · Long-Term Memory →