HumanEval and Code Benchmarks: How to Evaluate Code-Generating LLMs

Contents

The 95% vs 94% headline can't answer the only question that matters

A vendor announces a new coding model with a 95% HumanEval score. Another announces a competitor with 94%. Which one writes better code for your codebase?

The score can't tell you. HumanEval once separated good coding models from weak ones, and today it mostly confirms that a model clears a low bar. This guide explains what HumanEval measures, why it stopped helping, which benchmarks replaced it, and how to build an evaluation that reflects your own work — the code companion to the benchmark contamination article and the custom dataset guide. Leaderboard numbers move constantly, so this focuses on how to read benchmarks rather than who currently leads.

What HumanEval Actually Measures

OpenAI released HumanEval in 2021 alongside its Codex research. It contains 164 hand-written Python problems. Each problem gives the model a function signature and a docstring, and the model writes the body.

The benchmark then runs the generated code against unit tests — so HumanEval measures functional correctness, not whether the code looks nice or reads well.

This design gave the field a real advance. Earlier metrics compared text similarity, which rewarded code that resembled a reference without running correctly. Execution-based scoring fixed that.

Why HumanEval No Longer Separates Models

Three problems accumulated over time:

• Saturation: frontier models score in the low-to-mid 90s, and some sources cite even higher. A gap of a few points near the ceiling says almost nothing.
• Small size: 164 problems means each one swings the score by about 0.6 points.
• Contamination: the problems and their solutions have circulated online for years, so training data may include them.

Treat HumanEval as a floor check. A model that fails it has a serious problem. A model that passes it tells you very little about how it handles a real repository.

Understand pass@k Before You Compare Scores

Code benchmarks usually report pass@k. The metric asks: if the model generates k attempts for a problem, how often does at least one attempt pass all tests? pass@1 measures first-try success; pass@10 measures whether any of ten attempts works. Higher k always produces higher scores, so never compare a pass@10 number against a pass@1 number.

The standard approach generates n samples per problem, counts c correct ones, and applies an unbiased estimator:

import numpy as np

def pass_at_k(n, c, k):
    """Unbiased estimator of pass@k from n samples with c correct."""
    if n - c < k:
        return 1.0
    return 1.0 - np.prod(1.0 - k / np.arange(n - c + 1, n + 1))

print(pass_at_k(10, 1, 10))   # 1.000 — the single correct sample is always "found"
print(pass_at_k(20, 1, 10))   # 0.500 — same one-in-twenty skill, honest estimate

This estimator beats the naive approach because it avoids the bias you get from simply taking the best of k samples. The same model and the same 20 samples read very differently by k:

for k in (1, 5, 10):
    print(f"pass@{k}: {pass_at_k(20, 4, k):.3f}")
# pass@1: 0.200 — pass@5: 0.718 — pass@10: 0.957 — same skill, three headline numbers

Always report n, c, and k alongside the score — a pass@1 from one sample and a pass@1 from 200 samples are different claims.

The Benchmarks That Matter Now

No single benchmark covers everything. Pick a set that matches what you build.

Benchmark	What it tests	Best for	Main limit
HumanEval / HumanEval+ / MBPP	Entry-level Python functions, unit-tested	Floor checks	Saturated; HumanEval+ adds hidden tests to catch lucky passes
LiveCodeBench	Fresh competitive problems, date-tagged	Contamination-measured algorithmic skill	Single-file puzzles, not repositories
SWE-bench Verified	500 human-validated real GitHub issues	The standard vendors quote	Contamination concerns; vendors still over-quote it
SWE-bench Pro	1,865 problems, held-out splits, contamination controls	Harder repository-scale comparison	Scores run far lower than Verified
SWE-bench Live	Fresh tasks from recent GitHub activity	Leakage-resistant repository work	Newer, smaller evidence base
Multi-SWE-bench	Repository issues beyond Python	Multi-language teams	Same family caveats as SWE-bench
Terminal-Bench / Aider polyglot	Command-line tasks; cross-language editing	Agent evaluation	Measures the scaffold as much as the model

LiveCodeBench: Fresh Problems Against Contamination

LiveCodeBench draws new competitive programming problems on a rolling basis and tags each with a release date. You can score a model only on problems published after its training cutoff — contamination becomes a measurable property instead of a guess. Always report the benchmark version and the date window you used.

SWE-bench Family: Real Repository Work

SWE-bench asks a model or agent to resolve real GitHub issues in real Python repositories, then checks the fix against the project's test suite. The family has several members:

• SWE-bench Verified: a 500-task subset that human reviewers validated — the standard vendors quote for agentic coding.
• SWE-bench Pro: 1,865 problems across public, commercial, and held-out splits, with contamination controls. Scores on Pro run far lower than scores on Verified for the same models.
• SWE-bench Live: fresh tasks from recent GitHub activity, which makes training-data leakage much less likely.
• Multi-SWE-bench: the idea extended beyond Python.

One warning deserves emphasis: OpenAI has said it no longer reports SWE-bench Verified scores, citing contamination and tests that reward shortcuts. Many vendors still publish Verified numbers — read them with care and look for Pro results alongside them.

Terminal-Bench and Agentic Tasks

Terminal-Bench tests agents on tasks they complete through a command line — a different skill from patching one GitHub issue. Aider's polyglot benchmark offers another popular view of editing code across languages. Use these when you evaluate coding agents, not only base models.

Contamination: The Silent Score Inflator

Contamination happens when benchmark problems leak into training data. The model then remembers answers instead of solving problems, and the score overstates its real ability. How to spot it:

• Compare across benchmarks. One roundup suggests a gap of 30 or more points between HumanEval and LiveCodeBench signals contamination worth investigating.
• Use date-windowed problems. Score only on tasks published after the model's cutoff.
• Check held-out splits. Benchmarks like SWE-bench Pro keep private problems out of public circulation.
• Run your own private tests. Nothing beats problems that never touched the internet.

Figure 1: The eval that matters runs against your repository, with your tests, in your sandbox

The Scaffold Matters as Much as the Model

Agentic benchmarks score a system, not just a model. The scaffold decides how the agent explores files, runs tests, and retries. One analysis reports that a good agent scaffold lifts the same model by roughly 5 to 15 points on SWE-bench Pro.

So when you compare two scores, ask what harness produced each one. A strong model in a weak harness can lose to a weaker model in a strong one. Compare results only when the scaffolding matches — the same discipline the lm-evaluation-harness guide applies to backends and prompts.

Build Your Own Code Eval

Public benchmarks tell you about general ability. Your own eval tells you whether a model works on your code. Six steps:

1. Collect real tasks from your repositories: closed bug tickets, small features, and refactors.
2. Write or reuse tests as the pass/fail oracle. Strong tests matter, because weak tests let wrong code pass.
3. Freeze the starting commit for each task, so every run begins from identical code.
4. Run in a sandbox. Never execute model-generated code on a machine that holds secrets or production access.
5. Sample multiple attempts and report pass@1 and pass@k.
6. Review failures by hand. Patterns in failures teach you more than the aggregate number.

Start with 30 to 50 tasks. That's enough to see big differences between models and prompts.

Measure More Than Pass Rate

A passing test suite doesn't equal good code. Add these measurements:

Measurement	What it catches
Cost per solved task	Total spend divided by successes — the economic verdict
Latency and turns	How long the agent takes and how many steps it needs
Security issues	Static analysis and secret scanning on generated code
Code quality	Readability, style, maintainability — with human review on a sample
Regression risk	Whether the change breaks existing tests
Test gaming	Fixes that special-case the test instead of solving the problem

Cost per solved task often decides real-world model choice — a model that solves 5% more tasks at three times the price may lose.

Read Leaderboards Skeptically

Every published score comes with context. Ask these questions before you trust one:

• Who reported it? Vendor-reported numbers deserve extra scrutiny.
• Which version? "SWE-bench" alone means little. Look for Verified, Pro, or Live.
• What harness? Agent scaffolding can swing results by double digits.
• What date window? For LiveCodeBench, the version and time range matter.
• How many attempts? Check pass@1 versus pass@k.
• Does a second benchmark agree? Cross-check at least two.

Aggregator sites also disagree with each other on exact numbers, so go to primary sources when a decision depends on a figure.

Common Pitfalls

• Choosing a model from one headline number — benchmarks measure narrow slices of ability.
• Trusting saturated benchmarks — a 1-point HumanEval gap carries no meaning.
• Ignoring contamination — scores that look too good often are, as the contamination mechanics show.
• Comparing different harnesses — you end up measuring the scaffold.
• Mixing pass@k values — a pass@10 is not a pass@1, ever.
• Skipping private evals — public benchmarks never match your stack.
• Running generated code unsandboxed — that's a security incident waiting to happen.
• Reporting pass rate alone — cost, latency, and security decide real deployments.
• Quoting Verified scores without checking the date and vendor caveat — even OpenAI stopped reporting them.

Unlock AI That Actually Works

Get lifetime access to GPT-6 Astra, Claude Fable 5.1, Gemini 3.5, Grok 4.5, and more — all in one platform. Build websites, apps, videos, content, and digital products from a single command. No monthly fees. No tool-hopping.

Click here to get GPTAstra Max now — one-time payment, lifetime access.

Frequently Asked Questions

What does HumanEval measure, and why did it stop separating models?

HumanEval, released by OpenAI in 2021 alongside its Codex research, contains 164 hand-written Python problems. Each gives the model a function signature and a docstring; the model writes the body, and the benchmark runs the generated code against unit tests — so it measures functional correctness, not style. That was a real advance: earlier metrics compared text similarity and rewarded code that resembled a reference without running correctly. Three problems accumulated: saturation (frontier models score in the low-to-mid 90s, so a few points near the ceiling say almost nothing), small size (164 problems means each one swings the score by about 0.6 points), and contamination (the problems and solutions have circulated online for years). Treat it as a floor check: a model that fails it has a serious problem; a model that passes tells you very little about how it handles a real repository.

What is pass@k and how is it computed?

pass@k asks: if the model generates k attempts for a problem, how often does at least one attempt pass all tests? pass@1 is first-try success; pass@10 asks whether any of ten attempts works. Higher k always produces higher scores, so never compare a pass@10 number against a pass@1 number. The standard computation generates n samples per problem, counts c correct ones, and applies an unbiased estimator: 1 − prod(1 − k/i) for i from n−c+1 to n, returning 1.0 when n − c < k. This beats the naive "best of k samples" approach, which carries bias whenever samples are few. Always report n, c, and k alongside the score — a pass@1 from 1 sample and a pass@1 from 200 samples are different claims.

Which code benchmarks matter now?

No single benchmark covers everything. Floor checks: HumanEval, HumanEval+ (many more hidden tests per problem, catching solutions that pass the original weak tests by luck), and MBPP (larger set of entry-level Python) — sanity checks, not ranking tools. LiveCodeBench draws new competitive programming problems on a rolling basis, tagged with release dates, so you score a model only on problems published after its training cutoff — contamination becomes measurable; limit: single-file algorithmic puzzles, not repository-scale work; always report the version and date window. The SWE-bench family asks agents to resolve real GitHub issues checked against the project's test suite: SWE-bench Verified (500 human-validated tasks, the standard vendors quote), SWE-bench Pro (1,865 problems across public, commercial, and held-out splits with contamination controls — same models score far lower than on Verified), SWE-bench Live (fresh GitHub tasks, minimal leakage), and Multi-SWE-bench (beyond Python). OpenAI has said it no longer reports Verified scores, citing contamination and tests that reward shortcuts — read vendor Verified numbers with care. Terminal-Bench (command-line tasks) and Aider's polyglot benchmark evaluate agents rather than base models.

How do you build a private code evaluation?

Six steps: collect real tasks from your repositories (closed bug tickets, small features, refactors); write or reuse tests as the pass/fail oracle — strong tests matter, because weak tests let wrong code pass; freeze the starting commit for each task so every run begins from identical code; run in a sandbox, never executing model-generated code on a machine with secrets or production access; sample multiple attempts and report pass@1 and pass@k; review failures by hand, because patterns teach more than aggregates. Start with 30 to 50 tasks — enough to see big differences between models and prompts. Then measure beyond pass rate: cost per solved task (total spend divided by successes), latency and turns, security issues via static analysis and secret scanning, code quality with human review on a sample, regression risk against existing tests, and test gaming — fixes that special-case the test instead of solving the problem. Cost per solved task often decides real-world model choice: a model that solves 5% more tasks at three times the price may lose. Re-run the eval whenever you switch models or prompts.

Wrapping This Up

HumanEval did its job and now mostly confirms the basics. To evaluate a code-generating LLM today, combine fresh, contamination-resistant benchmarks with repository-scale tests, and then add a private eval built from your own code — the same discipline any eval program needs.

Start with thirty real tasks and a sandbox. Within a day, you'll know more about which model fits your team than any leaderboard could tell you.

Related Articles

• Why Public LLM Benchmarks Mislead: Contamination and Overfitting Explained
• How to Build a Custom Evaluation Dataset for Your LLM App
• Running LM Evaluation Harness: Benchmark Open-Source Models Yourself
• LLM Evaluation: Complete Guide to Testing and Measuring LLM Outputs