Without a nonlinear activation, stacked linear layers collapse into one linear layer. ReLU (max(0,x)) is the modern default — simple, fast, avoids vanishing gradients. Sigmoid (0-1) and tanh (-1 to 1) squash outputs but saturate and can vanish gradients in deep nets. Choosing the right activation per layer (ReLU hidden, sigmoid/softmax output) matters for training.
Power-ups you unlock
Nonlinearity prevents layer collapse
ReLU = modern default (fast, avoids vanishing)
Sigmoid/tanh squash but can saturate
Output activation depends on the task
The Overfit Ogre attacks — common mistakes
Stacking linear layers with no activation
Sigmoid/tanh in deep hidden layers (vanishing)
Wrong output activation for the task
Boss battleExplain why stacked linear layers need a nonlinearity between them.
Example code
<!doctype html><html><head><meta charset="utf-8"></head>
<body style="background:#06040d;color:#e6e0ff;font-family:monospace;padding:20px"><pre>ReLU(x) = max(0, x) ← default hidden
sigmoid → 0..1, tanh → -1..1 (can vanish)
no activation → many layers = one linear layer</pre></body></html>