Why Public LLM Benchmarks Mislead: Contamination and Overfitting Explained

Contents

Three ways a leaderboard number lies — and how to check yours

A model tops the leaderboard by two points. Is it better, or did it just see the test? Sometimes the honest answer is "both, a bit, and the two points are noise anyway."

This article explains the three ways public benchmark scores mislead — contamination, selection luck, and noise — what the research shows, and what to do about it. Everything numeric below comes from simulations with synthetic models and text, so you can see the mechanics: none of it measures a real model or benchmark. It's the benchmarking companion to the lm-evaluation-harness guide and the evaluation guide.

Problem 1: The Model May Have Seen the Test

Benchmark data contamination is defined in a 2024 survey as when "language models inadvertently incorporate evaluation benchmark information from their training data," which leads to "inaccurate or unreliable performance during the evaluation phase."

Benchmarks are public, and models train on huge web scrapes. Overlap is easy to create by accident.

How Big Can the Effect Be?

Take a model whose true skill on unseen questions is 60%. Suppose it answers memorized questions correctly 98% of the time. The measured score rises with the contaminated share:

Contaminated share	Measured score	Score on clean items only
0%	0.600	0.600
5%	0.618	0.599
10%	0.638	0.600
25%	0.695	0.600
50%	0.791	0.600

def measured_score(share, clean_skill=0.60, memorized_acc=0.98):
    return share * memorized_acc + (1 - share) * clean_skill

for share in (0.00, 0.05, 0.10, 0.25, 0.50):
    print(f"{share:>5.0%} contaminated -> {measured_score(share):.3f}")

The formula matched the simulation. The clean-items column recovers the true 0.60 — that's the key diagnostic, but it needs you to know which items leaked.

Contamination Can Flip a Ranking

In the simulation, model A has true skill 0.70 and no contamination. Model B has true skill 0.62 and 25% contamination. Measured scores: A 0.699, B 0.710. The weaker model ranks higher.

What Researchers Have Found in Real Models

Real evidence is mixed, so it deserves careful quoting:

• GSM1k. Researchers built a new benchmark designed to mirror GSM8k's style and difficulty. They report "accuracy drops of up to 8%" for some models, with "several families of models showing evidence of systematic overfitting across almost all model sizes," and a positive relationship (Spearman's r-squared = 0.36) between a model's likelihood of generating a GSM8k example and its performance gap — suggesting some models "may have partially memorized GSM8k." The same paper says many frontier models show minimal signs of overfitting, and all models broadly generalize to the new problems.
• An open-source contamination report. Researchers analyzed 15+ LLMs on six multiple-choice benchmarks using search and web-crawl matching, estimating contaminated fractions (their numbers, not mine) from 1.1% on Winogrande up to 45.8% on C-Eval, with MMLU at 29.1%. They report accuracy gains of up to 14% on C-Eval and 7% on HellaSwag for contaminated items, while MMLU "shows little consistent gain." Contamination doesn't uniformly inflate scores — and those gains compare subsets that differ in other ways, so don't read them as pure cause and effect.

The honest summary: contamination is real, its size varies by benchmark and model, and some evidence suggests top models are less affected than feared.

Problem 2: Detection Is Harder Than It Sounds

If you know the training data, you can search it. The common approach is n-gram overlap: flag a benchmark item if any run of n tokens appears in the training corpus.

The simulation used 1,000 items of 40 tokens, with 200 copied into a 400,000-token corpus, some lightly reworded by swapping a fraction of tokens:

Words swapped	n = 3	n = 8	n = 13
0% (exact copy)	100%	100%	100%
5%	100%	100%	96%
15%	100%	95%	56%
30%	100%	50%	6%

The n = 3 column looks great until you see its false positives: 97.5% of clean items were flagged too, because short n-grams occur everywhere. At n = 8 and n = 13, false positives were 0%, but recall fell as rewording grew. Whole-item matching at a 2% rewording threshold found 45.5% against a theoretical 44.6%.

Exact copies are easy, light paraphrase slips past, and a threshold that avoids false alarms misses rewording. The contamination-detection survey says the same: "rephrased samples," which keep the meaning with a different surface form, make checks challenging. It also says n-gram overlap "fails in the coding domain," and cites work showing that paraphrasing benchmarks and fine-tuning on them "can bypass even oracle-access contamination detection methods."

Another complication: the open-source report above deliberately used search plus a fuzzier similarity score (METEOR), because exact string matching produces false negatives.

And for closed models, you usually can't see the training data at all. The survey notes that makes detection harder, and it lists indirect methods — performance gaps, membership inference, likelihood-based checks — each with stated limitations.

Problem 3: Leaderboards Reward Luck

Even with zero contamination, picking the top of a leaderboard selects for noise. A benchmark score is a sample, so it has a standard error. At 500 questions and 70% accuracy, that's about 2.0 points.

The simulation had 100 models scored on the same 500 questions:

Spread of real skill (sd)	Top model's public score minus its true skill	Rank correlation, public vs fresh draw
0.00 (all identical)	+0.050	0.01
0.01	+0.045	0.17
0.03	+0.026	0.66
0.06	+0.012	0.88

import random

rng = random.Random(7)

def public_score(true_skill, n=500):
    """One model's measured score on the shared benchmark."""
    return sum(rng.random() < true_skill for _ in range(n)) / n

# 100 models, all truly 70% accurate, one shared leaderboard
scores = [public_score(0.70) for _ in range(100)]
winner = max(scores)
print(f"true skill 0.700 -> winner's public score {winner:.3f} "
      f"(+{(winner - 0.70) * 100:.1f} pts of pure luck)")

When all models were truly identical, the "winner" scored 5 points above its real skill, and the ranking was uncorrelated with a fresh benchmark. When models truly differ a lot, rankings hold up better, but the top score is still inflated. The takeaway: gaps of one or two points between close models are mostly noise.

Figure 1: The winner's curse — when every model is truly 70%, someone still tops the board by five points

Does Repeated Testing Really Cause Overfitting?

Less than you'd think, at least in some settings. A meta-analysis of over a hundred Kaggle competitions compared public and final rankings and found "little evidence of substantial overfitting," arguing it supports "the robustness of the holdout method." A separate paper (Feldman, Frostig and Hardt) says there is "little concrete evidence of significant overfitting due to holdout reuse in popular multiclass benchmarks today."

Those results are about classic machine-learning competitions and classification benchmarks, not LLM leaderboards — don't carry them over unchanged. Training-data contamination is a different mechanism from reusing a holdout. Treat them as a reminder not to assume the worst without checking.

What You Can Do

1. Build a private test set. It's the one thing contamination can't touch. The custom evaluation dataset guide shows how to build one.
2. Make a fresh variant. The GSM1k approach: new questions with the same style and difficulty. Compare scores on the public and fresh sets. A big drop is a red flag.
3. Check subsets. If you know or can estimate which items might have leaked, score them separately.
4. Report uncertainty. Always include the standard error, and don't rank within it.
5. Prefer many benchmarks over one, since leaking all of them is less likely — common sense, not a tested result.
6. Run it on your task. A leaderboard score is a rough prior. Your own data is the verdict.

One caution: be careful with "no contamination found." Failing to detect leakage isn't proof there's none.

The cheapest, most valuable step is the first one. Even 200 well-chosen private questions will teach you more than another leaderboard.

A Quick Skeptic's Checklist

• Is the gap bigger than about two standard errors?
• Is the benchmark old and widely posted?
• Do the authors report a decontamination check, and how did it work?
• Does the model do worse on a fresh, similar test?
• Does it do well on your own questions?

None of these prove anything alone. Together they tell you how much to trust a number.

Common Pitfalls

• Quoting a leaderboard gap smaller than two standard errors — at 500 questions, that's about four points of wiggle room.
• Assuming contamination explains every high score — real evidence is mixed, and top models may generalize fine.
• Reporting "no contamination found" as proof — paraphrase slips past every practical detector.
• Trusting an n=3 overlap check — 97.5% false positives make it noise.
• Reading contaminated-vs-clean subset gaps as cause and effect — the subsets differ in other ways too.
• Carrying "holdout reuse is fine" over from Kaggle studies — different mechanism, different era, different models.
• Relying on one benchmark — leaking one is easy; leaking ten is harder.
• Never asking how old and widely posted a benchmark is — age and exposure predict leakage.
• Skipping the fresh-variant test — GSM1k-style checks are cheap and decisive.
• Treating "minimal overfitting in frontier models" as "no contamination" — an average hides the tail.

Unlock AI That Actually Works

Get lifetime access to GPT-6 Astra, Claude Fable 5.1, Gemini 3.5, Grok 4.5, and more — all in one platform. Build websites, apps, videos, content, and digital products from a single command. No monthly fees. No tool-hopping.

Click here to get GPTAstra Max now — one-time payment, lifetime access.

Frequently Asked Questions

How much can contamination inflate a benchmark score?

Mechanically, as much as the memorized share of the test. If a model's true skill on unseen questions is 60% and it answers memorized questions 98% of the time, the measured score is share × 0.98 + (1 − share) × 0.60: 0.638 at 10% contamination, 0.695 at 25%, 0.791 at 50% — while its score on clean items stays 0.600 throughout. That's enough to flip rankings: in simulation, a clean model with true skill 0.70 measured 0.699 while a weaker model (0.62) with 25% contamination measured 0.710. Real-world evidence is mixed: GSM1k, a fresh benchmark styled after GSM8k, showed accuracy drops of up to 8% for some models with systematic overfitting across almost all sizes and a positive link (Spearman's r² = 0.36) between generating GSM8k examples and the performance gap — yet many frontier models showed minimal signs of overfitting and all generalized to the new problems. An open-source report on 15+ LLMs across six multiple-choice benchmarks estimated contaminated fractions from 1.1% (Winogrande) to 45.8% (C-Eval), with MMLU at 29.1%, and gains up to 14% on C-Eval and 7% on HellaSwag for contaminated items, while MMLU showed little consistent gain. Honest summary: contamination is real, uneven, and top models may be less affected than feared.

How do you detect benchmark contamination?

The common approach is n-gram overlap: flag a benchmark item if any run of n tokens appears in the training corpus. In simulation (1,000 items of 40 tokens, 200 copied into a 400,000-token corpus, some reworded), exact copies were caught 100% of the time at every n, but at 30% of words swapped recall fell to 50% for n=8 and 6% for n=13 — while n=3 flagged 97.5% of clean items too, because short n-grams occur everywhere. Whole-item matching at a 2% threshold found 45.5% against a theoretical 44.6%. The contamination-detection survey says the same: rephrased samples that keep meaning with a different surface form make checks challenging, n-gram overlap fails in the coding domain, and paraphrasing a benchmark plus fine-tuning on it can bypass even oracle-access detection methods. Search plus fuzzy similarity (METEOR) is the alternative when exact matching produces false negatives. For closed models you usually can't see training data at all; indirect methods (performance gaps, membership inference, likelihood checks) each come with stated limitations — so treat "no contamination found" as "not detected."

Why do leaderboards reward luck even with zero contamination?

A benchmark score is a sample, so it has a standard error — at 500 questions and 70% accuracy, about 2.0 points. When 100 simulated models with identical true skill were scored on the same 500 questions, the "winner" finished +5.0 points above its real skill and the public ranking had a rank correlation of 0.01 against a fresh draw — pure noise. As the spread of real skill grows, rankings hold up better (correlation 0.66 at sd 0.03, 0.88 at sd 0.06) but the top score is still inflated (+2.6 and +1.2 points). The takeaway: gaps of one or two points between close models are mostly noise; report uncertainty and don't rank within it. Repeated-testing overfitting is milder than assumed in classic settings: a meta-analysis of 100+ Kaggle competitions found little evidence of substantial overfitting, and Feldman, Frostig and Hardt found little concrete evidence of significant overfitting from holdout reuse in popular multiclass benchmarks — but those results concern classic competitions, not LLM leaderboards, and holdout reuse is a different mechanism from training-data contamination.

What should you do instead of trusting leaderboards?

Six moves, in order of value: build a private test set — it's the one thing contamination can't touch, and even 200 well-chosen questions beat another leaderboard; make a fresh variant (the GSM1k approach: new questions, same style and difficulty) and compare — a big drop on the fresh set is a red flag; check subsets, scoring items you know or suspect leaked separately; report uncertainty with the standard error and never rank two models within it; prefer many benchmarks over one, since leaking all of them is less likely (common sense, not a tested result); and run the benchmark on your task — a leaderboard score is a rough prior, your own data is the verdict. The skeptic's checklist for any published number: is the gap bigger than about two standard errors, is the benchmark old and widely posted, do the authors report a decontamination check and how it worked, does the model do worse on a fresh similar test, does it do well on your own questions. No single check proves anything; together they tell you how much to trust the number.

Wrapping This Up

Public benchmarks mislead in three ways: models may have seen the questions, leaderboards select for lucky scores, and small gaps are noise. The research shows contamination is real but uneven, and detection is hard once text is reworded.

Keep a private test set, compare against fresh variants, and report uncertainty.

Pick one benchmark you rely on today, and ask how you'd know if it were contaminated. If you can't answer, start building a private set this week.

Related Articles

• How to Build a Custom Evaluation Dataset for Your LLM App
• Running LM Evaluation Harness: Benchmark Open-Source Models Yourself
• LLM Evaluation: Complete Guide to Testing and Measuring LLM Outputs
• LLM Evaluation Metrics Explained: Perplexity, BLEU, ROUGE, and Beyond