TL;DREncoder-only models (BERT) read bidirectionally — great for understanding, not generation.
BERT-style models use only the transformer encoder and attend in both directions, so each token sees full left and right context. Pretrained by masking words and predicting them, they excel at understanding tasks: classification, sentiment, search, named-entity recognition, embeddings. They don't generate fluent long text — that's the decoder's job. Use encoders to comprehend, not to write.
Key points
Encoder-only, bidirectional context
Pretrained via masked-word prediction
Best for classification/search/embeddings
Not built for open-ended generation
Common mistakes
Using BERT to generate long text
Confusing masked LM with next-token LM
Assuming bidirectional = better for everything
Try it: Pick encoder-only or decoder-only for sentiment classification vs writing an essay.
Example code
<!doctype html><html><head><meta charset="utf-8"></head>
<body style="background:#06040d;color:#e6e0ff;font-family:monospace;padding:20px"><pre>BERT (encoder): sees full context → classify/search
generate an essay → need a decoder (GPT)</pre></body></html>