Tensor TownISSUE #76 of 120

benchmarks · mmlu · gsm8k · humaneval

NeuraVSThe Overfit Ogre

Neura saysBenchmarks like MMLU, GSM8K, and HumanEval measure knowledge, math, and coding.

Common yardsticks: MMLU (broad multiple-choice knowledge across 57 subjects), GSM8K (grade-school math word problems), HumanEval (does generated code pass tests). They make models comparable on specific skills. Caveats: test data can leak into training (contamination), and models can be tuned to the benchmark. Treat them as rough signals, not ground truth.

Power-ups you unlock

MMLU: broad knowledge · GSM8K: math · HumanEval: code
Make models comparable per skill
Risk: training-data contamination
Rough signals, not ground truth

The Overfit Ogre attacks — common mistakes

Assuming benchmarks are contamination-free
Comparing models on mismatched benchmark versions
Treating one benchmark as overall "intelligence"

Boss battleMatch MMLU, GSM8K, and HumanEval to what each measures.

Example code

<!doctype html><html><head><meta charset="utf-8"></head>
<body style="background:#06040d;color:#e6e0ff;font-family:monospace;padding:20px"><pre>MMLU      → broad knowledge (57 subjects)
GSM8K     → grade-school math
HumanEval → code passes tests</pre></body></html>

▶ Open the interactive comic issue

‹ Evaluation · Benchmarks Vs Vibes Eval Harnesses · Lm-Eval · Helm · Big-Bench ›