freecoding.school100% FREE · NO SIGNUP
Tensor TownISSUE #48 of 120

encoder-only · bert family

NeuraVSThe Overfit Ogre
Neura saysEncoder-only models (BERT) read bidirectionally — great for understanding, not generation.

BERT-style models use only the transformer encoder and attend in both directions, so each token sees full left and right context. Pretrained by masking words and predicting them, they excel at understanding tasks: classification, sentiment, search, named-entity recognition, embeddings. They don't generate fluent long text — that's the decoder's job. Use encoders to comprehend, not to write.

Power-ups you unlock

The Overfit Ogre attacks — common mistakes

Boss battlePick encoder-only or decoder-only for sentiment classification vs writing an essay.

Example code

<!doctype html><html><head><meta charset="utf-8"></head>
<body style="background:#06040d;color:#e6e0ff;font-family:monospace;padding:20px"><pre>BERT (encoder): sees full context → classify/search
generate an essay → need a decoder (GPT)</pre></body></html>
▶ Open the interactive comic issue
‹ Positional Encoding · Giving Order BackDecoder-Only · Gpt Family ›