Neura saysGloVe and subword embeddings refined the idea — global stats and handling unknown words.
GloVe built embeddings from global word co-occurrence counts rather than local windows. Subword approaches (fastText, BPE) embed pieces of words, so a model can represent rare or unseen words by their parts ("unhappiness" = un + happi + ness) — fixing word2vec's out-of-vocabulary blind spot. Subword tokenization is now standard in LLMs, which is why they handle novel words gracefully.
Power-ups you unlock
GloVe: embeddings from global co-occurrence counts
Subword: embed word pieces (fastText, BPE)
Handles rare/unseen words via parts
Subword tokenization is standard in LLMs
The Overfit Ogre attacks — common mistakes
Assuming word-level embeddings handle new words
Confusing GloVe (counts) with word2vec (prediction)
Ignoring tokenization’s role downstream
Boss battleExplain how subword embeddings represent a word the model never saw.
Example code
<!doctype html><html><head><meta charset="utf-8"></head>
<body style="background:#06040d;color:#e6e0ff;font-family:monospace;padding:20px"><pre>unseen word "unhappiness"
→ subwords: un + happi + ness (each embedded)
→ no out-of-vocabulary blind spot</pre></body></html>