MMLU Explained: What the Benchmark Measures and Where It Falls Short

Contents

What the most-quoted number in LLM announcements actually measures

"Scores 90% on MMLU" sounds like a model that passed a graduate exam. What was it actually tested on, and how much of that number can you trust?

MMLU is the most quoted benchmark in LLM announcements. This article explains what it measures, then walks through six known problems: wrong answer keys, a guessing floor, noisy subject scores, averaging choices, prompt sensitivity, and contamination. The simulations use synthetic questions — they explain the mechanics rather than measuring any real model. It continues the benchmark-mislead series from the lm-evaluation-harness guide.

What MMLU Is

MMLU (Massive Multitask Language Understanding) comes from a 2020 paper. Its abstract describes 57 tasks spanning subjects such as "elementary mathematics, US history, computer science, law, and more." The questions are multiple choice, with four options each (clear from the MMLU-Pro paper, which describes expanding "the choice set from four to ten options"). An open-source contamination report lists MMLU's size as 13,987 questions — that count is from the report, not the original abstract.

The paper's original findings: most recent models performed at "near random-chance accuracy." The largest GPT-3 model beat chance by "almost 20 percentage points on average." The best models fell short on all 57 tasks and "still need substantial improvements before they can reach expert-level accuracy," with "lopsided performance" and models that "frequently do not know when they are wrong." Those were 2020 results — scores have risen enormously since, which is part of why the benchmark now struggles to separate top models.

What It Actually Measures

Be precise. MMLU measures whether a model picks the keyed option on a fixed set of four-choice questions about academic and professional subjects. That mostly reflects knowledge recall and test-taking skill under a specific prompt format.

It doesn't measure:

• Whether answers hold up in open-ended conversation.
• Reasoning on problems that need many steps (MMLU-Pro's authors added "more challenging, reasoning-focused questions" because MMLU is mostly knowledge-driven).
• Honesty, safety, tool use, or long-document work.
• Anything about your task.

A high score says "good at this exam." It doesn't say "good at your job."

Shortfall 1: The Guessing Floor Depends on Option Count

With four options, random guessing gets 25%. And partial knowledge pays off quickly: a model that can rule out two wrong options per question and then guesses scores 50% on four options — the same skill scores only 12.5% on a ten-option test.

That's why option count matters when comparing benchmarks. MMLU-Pro expanded to ten options, and its authors report "a significant drop in accuracy by 16% to 33% compared to MMLU." Part of that drop is the lower guessing floor, part is harder questions — the abstract doesn't split them.

Shortfall 2: Some Answer Keys Are Wrong

The most damaging problem. The "Are We Done with MMLU?" paper estimates that "6.49% of MMLU questions contain errors," and that "57% of the analysed questions in the Virology subset contain errors." The authors released MMLU-Redux, "a subset of 5,700 manually re-annotated questions across all 57 MMLU subjects," reporting "significant discrepancies with the model performance metrics that were originally reported."

What do errors do to scores? A simulated fraction of questions whose key points at a wrong option, four options each:

Error rate	True accuracy	Measured
0%	90%	0.900
6.49%	70%	0.661
6.49%	90%	0.844
6.49%	100%	0.935
15%	100%	0.850

def measured_accuracy(true_acc, error_rate, n_options=4):
    """Errored keys: a knowing model answers the true (non-key) option and is marked wrong."""
    random_hit = (1 - true_acc) * (1 / (n_options - 1))
    return (1 - error_rate) * true_acc + error_rate * random_hit

print(measured_accuracy(0.90, 0.0649))   # 0.844 — a 90% model reads as 84%
print(measured_accuracy(1.00, 0.0649))   # 0.935 — the ceiling at 6.49% errors

Two consequences:

• There's a ceiling. At the estimated error rate, even a perfect model scores about 93.5%. A model in the high 80s or low 90s may be near the ceiling already.
• Gaps shrink. A real 2-point difference between strong models shows up as ~1.8 points, since 91% of the true gap survives — small, but it adds noise to an already small gap.

One caveat: the paper's "errors" cover more than wrong keys (ambiguous questions included), and the simulation modeled only the wrong-key case. Treat these numbers as an illustration.

Shortfall 3: Averaging Rules Change the Winner

MMLU reports a single score across 57 subjects of different sizes. Pool all questions (micro average) or average the 57 subject scores (macro average) — they don't always agree.

A simulation with 57 synthetic subjects sized 111 to 1,260 questions (made up, not MMLU's real distribution), where model A is strong on the 10 biggest subjects and model B on the 20 smallest:

Model	Micro	Macro
A (strong on big subjects)	0.744	0.721
B (strong on small subjects)	0.719	0.735

Micro ranks A first. Macro ranks B first. When you read a leaderboard, check which average it reports — it should always say.

Shortfall 4: Subject Scores Are Noisy

People love to say "model X is weak at law" from a subject score. But a subject might have only a couple hundred questions.

In simulation, every subject was truly 70% for a model identical across subjects. At a typical size of 182 questions, the standard error was 3.4 points, measured scores spread 3.3 points, and about 3.8 of 57 subjects scored below 65% by chance alone.

import math

def subject_se(true_acc, n_questions):
    return math.sqrt(true_acc * (1 - true_acc) / n_questions)

print(f"{subject_se(0.70, 182) * 100:.1f} points")   # 3.4 — one subject, 182 questions

A model with no real weaknesses still shows some. Only large, consistent gaps are worth believing, and only after you check the size of the subject.

Figure 1: Fifty-seven subjects, four options, one number — what the exam frame hides

Shortfall 5: Prompt and Format Sensitivity

MMLU-Pro's abstract says score sensitivity to prompt wording "decreased from 4-5% in MMLU to just 2% in MMLU-Pro." By the authors' account, MMLU scores can move 4 to 5 points depending on the prompt — larger than many leaderboard gaps.

That's consistent with the toy simulations of multiple-choice scoring in the harness guide: few-shot count, prompt template, answer extraction, and option position can all move the number. Always check the settings behind a claimed score.

Shortfall 6: Contamination and Saturation

MMLU has been public for years. The open-source contamination report estimated MMLU at 29.1% contaminated via web search, though it saw "little consistent gain" from it — the mechanics are covered in the contamination article.

Top models now cluster near the ceiling set by label errors. When many models land within a couple of points, noise and averaging choices matter more than ability.

What to Do Instead

• Treat MMLU as a rough filter, not a decision tool. It separates a weak model from a strong one. It can't reliably rank two strong ones.
• Report error bars. At ~14,000 questions the overall standard error is ~a third of a point — but prompt and format effects are far larger.
• Use the cleaned subset (MMLU-Redux) or the harder variant (MMLU-Pro) for less noise, with the caveats above.
• Match settings: same prompt, few-shot count, answer extraction, and averaging.
• Test on your own task. A private evaluation set built from your data beats any public number — the custom dataset guide shows how.

A one-point MMLU lead tells you almost nothing about which model is better for you.

Common Pitfalls

• Reading a single MMLU number as capability — it's one four-choice exam format.
• Ranking strong models on sub-2-point gaps — prompt effects alone move 4-5 points.
• Quoting subject scores as strengths and weaknesses — 182 questions carry a ±3.4-point standard error.
• Comparing scores across averaging rules — micro and macro can flip the winner.
• Ignoring the guessing floor — 25% free on MMLU, 10% on MMLU-Pro.
• Forgetting the label-error ceiling — ~6.49% wrong keys cap perfect models near 93.5%.
• Assuming "little consistent contamination gain" means clean — detection limits still apply.
• Never checking the prompt and few-shot settings behind a published score.

Unlock AI That Actually Works

Get lifetime access to GPT-6 Astra, Claude Fable 5.1, Gemini 3.5, Grok 4.5, and more — all in one platform. Build websites, apps, videos, content, and digital products from a single command. No monthly fees. No tool-hopping.

Click here to get GPTAstra Max now — one-time payment, lifetime access.

Frequently Asked Questions

What does the MMLU benchmark measure?

MMLU (Massive Multitask Language Understanding), from a 2020 paper, is 57 multiple-choice tasks with four options each, spanning subjects such as elementary mathematics, US history, computer science, and law, totaling 13,987 questions per an open-source contamination report. Precisely: whether a model picks the keyed option on this fixed exam — mostly knowledge recall and test-taking skill under a specific prompt format. It doesn't measure open-ended conversation quality, multi-step reasoning (MMLU-Pro's authors added more challenging reasoning-focused questions because MMLU is mostly knowledge-driven), honesty, safety, tool use, long-document work, or anything about your task. A high score says "good at this exam," not "good at your job." The 2020 results set the baseline: most models were at near random-chance accuracy, GPT-3's largest beat chance by almost 20 percentage points on average, and the authors noted lopsided performance and that models frequently do not know when they are wrong.

How much does contamination and wrong answer keys affect MMLU scores?

Wrong keys are the most damaging issue: the "Are We Done with MMLU?" paper estimates 6.49% of MMLU questions contain errors (57% of the Virology subset), and released MMLU-Redux, 5,700 manually re-annotated questions across all 57 subjects, reporting significant discrepancies with originally reported model performance. Simulated effect with four options: at a 6.49% wrong-key rate, even a perfect model scores about 0.935 — the ceiling — and a real 2-point gap between strong models survives as only ~1.8 points. A model in the high 80s or low 90s may already be near the ceiling. (Caveat: the paper's errors include ambiguous questions; the simulation modeled only wrong keys.) On contamination, the open-source report estimated MMLU at 29.1% contaminated via web search but saw little consistent gain from it — and top models now cluster near the label-error ceiling, so noise and averaging choices matter more than small ability differences.

Why do MMLU scores move with prompt, format, and averaging choices?

Three mechanics. Averaging: MMLU pools 57 subjects of different sizes; a micro average (all questions) and a macro average (mean of 57 subject scores) can disagree — in simulation, a model strong on the 10 biggest subjects won micro (0.744 vs 0.719) while a model strong on the 20 smallest won macro (0.735 vs 0.721); always check which average a leaderboard reports. Prompt sensitivity: MMLU-Pro's authors report score sensitivity to prompt wording decreased from 4-5% in MMLU to just 2% in MMLU-Pro — larger than many leaderboard gaps, consistent with few-shot count, prompt template, answer extraction, and option position all moving the number. Subject noise: with every subject truly 70%, a 182-question subject has a standard error of ~3.4 points, measured scores spread ~3.3 points, and about 3.8 of 57 subjects fall below 65% by chance — so "weak at law" claims need large, consistent gaps and a check of subject size.

How should you evaluate models instead of trusting MMLU?

Treat MMLU as a rough filter: it separates weak from strong models but can't reliably rank two strong ones. Report error bars — at ~14,000 questions the overall standard error is about a third of a point, but prompt and format effects (4-5 points) dwarf it. Prefer the cleaned MMLU-Redux subset or the harder MMLU-Pro (10 options, 16-33% accuracy drop vs MMLU — part lower guessing floor, part harder questions). Match settings across compared models: same prompt, few-shot count, answer extraction, and averaging. Then test on your own task: a private evaluation set built from real data beats any public number — the guessing floor alone means a four-option score carries 25% free accuracy, and partial knowledge (ruling out two options then guessing) scores 50%. A one-point MMLU lead tells you almost nothing about which model is better for you.

Wrapping This Up

MMLU measures multiple-choice knowledge across 57 subjects. It was a useful yardstick for 2020's models. Today it suffers from wrong answer keys (an estimated 6.49% of questions), a four-option guessing floor, sensitivity to prompts, noisy subject scores, and averaging choices that can flip rankings.

Use it to roughly place a model, not to pick a winner. Then run a small private test on your real task before you decide anything.

Related Articles

• Why Public LLM Benchmarks Mislead: Contamination and Overfitting Explained
• Running LM Evaluation Harness: Benchmark Open-Source Models Yourself
• How to Build a Custom Evaluation Dataset for Your LLM App
• LLM Evaluation: Complete Guide to Testing and Measuring LLM Outputs