Running LM Evaluation Harness: Benchmark Open-Source Models Yourself

Contents

Reproduce the leaderboard number yourself — then read the failures

Every model release comes with a leaderboard screenshot. Can you reproduce the number yourself? With EleutherAI's lm-evaluation-harness, mostly yes — if you control the settings that move the score.

This guide covers installing it, running your first benchmark, and reading the results without fooling yourself. One honest caveat first: the commands below come from the project's README and were not executed in this sandbox, because the package index is unreachable. A toy simulation of multiple-choice scoring was run, so you can see exactly why settings matter — the benchmarking half of the evaluation guide.

What the Harness Does

It's a framework for running language models through standard benchmark tasks and getting comparable numbers. You point it at a model and a list of tasks, and it handles prompting, scoring, and reporting.

It supports many backends per the README: Hugging Face transformers (hf), vLLM, sglang, OpenAI-style and OpenAI-compatible endpoints, hosted APIs, litellm, GGUF models through llama.cpp, and more. Models without logprobs work only with generate_until tasks.

Install It

From the README:

pip install lm-eval

The base package doesn't include model backends, so add the extras you need:

pip install "lm_eval[hf]"      # Hugging Face transformers
pip install "lm_eval[vllm]"    # vLLM backend
pip install "lm_eval[api]"     # OpenAI-style and hosted APIs

You can also install from source: git clone --depth 1 https://github.com/EleutherAI/lm-evaluation-harness, then pip install -e .

Run Your First Benchmark

The README's basic example evaluates a Hugging Face model on one task:

lm_eval --model hf \
    --model_args pretrained=EleutherAI/gpt-j-6B \
    --tasks hellaswag \
    --device cuda:0 \
    --batch_size 8

Breaking that down:

• --model picks the backend.
• --model_args passes settings to it, like the checkpoint name.
• --tasks takes one or more task names, comma-separated.
• --device picks the GPU.
• --batch_size controls memory use. The README says --batch_size auto detects the largest size that fits.

To pin a revision or precision, put them in --model_args, as in the README: pretrained=EleutherAI/pythia-160m,revision=step100000,dtype="float". List available tasks with lm-eval ls tasks.

Worth pinning: the revision matters. "Latest" can change under you, and then your comparison isn't repeatable.

Save What You Need

Three flags from the README worth using every time:

• --output_path results saves the numbers.
• --log_samples records model responses, so you can read the failures later.
• --use_cache <DIR> skips samples already evaluated when you resume. The README warns caching is rank-dependent, so restart with the same GPU count.

A score tells you how often the model failed. The samples tell you why.

How a Task Is Defined

The task guide describes YAML configs with fields like task, dataset_path, output_type, doc_to_text, doc_to_target, doc_to_choice, and metric_list. The output types are generate_until, loglikelihood, loglikelihood_rolling, and multiple_choice.

Two metric names you'll see constantly are acc (accuracy) and acc_norm (length-normalized accuracy). The exact computation isn't defined in the docs, and the source file couldn't be read here — so everything below about normalization comes from a toy model, not the harness's code.

For multiple-choice tasks, the model doesn't write an answer. It scores each choice's text by log-likelihood, and the highest score wins. That's why prompt wording and choice length matter so much.

Why acc and acc_norm Can Disagree

Longer answer text means more tokens, so the total log-likelihood is usually lower. A raw score can favor short choices. Normalizing by length tries to fix that. The toy model (each character costs a fixed amount of log-likelihood, four choices per question, acc_norm dividing by length in bytes — an assumption, not a verified detail of the harness) shows both sides.

A model that knows nothing (chance is 25%):

Dataset	acc	acc_norm
Answer lengths independent of truth	0.250	0.250
Correct answers ~25 characters longer	0.065	0.182
Correct answers ~25 characters shorter	0.665	0.367

A model with no knowledge scored 66.5% on one dataset, purely because correct answers happened to be shorter. That's the failure shape to watch for.

A model with real knowledge:

Dataset	acc	acc_norm
Independent lengths	0.338	0.631
Correct answers longer	0.043	0.466
Correct answers shorter	0.765	0.813

Neither metric wins everywhere. In the toy the length penalty is pure noise, so normalizing helps even when lengths are independent — real models and tasks may behave differently. When you compare two models, use the same metric the task recommends, and don't mix acc for one with acc_norm for another.

Position Bias

A second toy: a model with no knowledge that leans toward the first option. When 10% of correct answers sat in position A, it scored 0.186. At 40%, it scored 0.306. At 25%, it scored 0.245. The bias was moderate in the toy, so the effect is partial, but it's real: a model with a habit looks better or worse depending on answer placement. Check that correct answers are spread across positions.

How Many Questions Do You Need?

The harness lets you cap examples. The README mentions --limit 10 in two places — as a small trial on closed chat APIs to check answer extraction, not as a way to get a real score.

Here's why that's wise. For a model that's truly 60% accurate:

Questions	Standard error	95% range for the measured score
10	15.5 pts	30% to 90%
100	4.9	50% to 70%
500	2.2	56% to 64%
2,000	1.1	58% to 62%

import math

def standard_error(p, n):
    return math.sqrt(p * (1 - p) / n)

for n in (10, 100, 500, 2000):
    se = standard_error(0.60, n)
    lo, hi = 0.60 - 1.96 * se, 0.60 + 1.96 * se
    print(f"n={n:>5}  SE={se * 100:4.1f} pts  95% range: {lo:.0%} to {hi:.0%}")

The analytic formula matched a simulation at every size, and a bootstrap check agreed. So --limit 100 is a smoke test. Two models two points apart can't be told apart with it. Run the full task for any comparison you plan to quote.

Figure 1: One small model, one task, both logging flags on — the habit that makes every later number defensible

Settings That Quietly Change Your Score

These aren't all from one source, so here's which are documented:

• Model backend. The README notes vLLM output "can differ from Hugging Face," which it treats as the reference. It provides scripts/model_comparator.py to check.
• Few-shot count. The README's only few-shot flag is --num_fewshot in its prompt-inspection example. Keep it fixed across models.
• Prompt format. Task YAML controls it through doc_to_text. Don't edit it between runs without noting it.
• Batch size and hardware. Numerical differences can nudge scores slightly — a caution, not a measurement.
• Apple MPS. The README calls it early-stage and suggests comparing against CPU if results look odd.
• Version pinning. Both the model revision and the harness version belong in your notes, or next month's run won't match this month's.

Checklist for a Fair Comparison

• Pin the model revision and the harness version.
• Use the same tasks, few-shot count, and metric for every model.
• Run full tasks, not --limit, for anything you'll publish.
• Save results and logged samples.
• Read at least 20 failures per task.
• Report the standard error alongside every score.
• Write down hardware, backend, and batch size.

That last item is what separates a reproducible benchmark from a screenshot.

What the Harness Can't Tell You

A benchmark score says how a model did on that benchmark. It doesn't say how it does on your documents, your users, or your prompts. Public tasks can also leak into training data, and that wasn't investigated here. For your own app, build a custom evaluation set — the dataset guide shows how — and use the harness as a baseline check.

Common Pitfalls

• Quoting --limit numbers as real scores — 100 questions carries a ±5-point standard error at 60% accuracy.
• Comparing acc for one model with acc_norm for another — pick the task's recommended metric and stay with it.
• Letting "latest" revisions move under you — pin the checkpoint and the harness version.
• Never reading logged samples — the score says how often, the samples say why.
• Answer-length artifacts fooling raw accuracy — a knowledge-free model scored 66.5% when correct answers happened to be shorter.
• Ignoring answer-position distribution — a position habit swung a toy model from 0.186 to 0.306.
• Mixing backends across compared models — vLLM can differ from the Hugging Face reference; check with model_comparator.py.
• Reporting no standard error — without it, readers can't tell a real gap from noise.
• Treating a benchmark score as app performance — leakage and task mismatch make it a baseline, not an answer.
• Skipping the smoke trial on chat APIs — answer extraction breaks silently without it.

Unlock AI That Actually Works

Get lifetime access to GPT-6 Astra, Claude Fable 5.1, Gemini 3.5, Grok 4.5, and more — all in one platform. Build websites, apps, videos, content, and digital products from a single command. No monthly fees. No tool-hopping.

Click here to get GPTAstra Max now — one-time payment, lifetime access.

Frequently Asked Questions

How do you run your first lm-evaluation-harness benchmark?

Install the base package with pip install lm-eval, then add only the backend extras you need: lm_eval[hf] for Hugging Face transformers, lm_eval[vllm], or lm_eval[api] for OpenAI-style and hosted endpoints (GGUF models run through llama.cpp, and litellm is also supported). The basic command evaluates a Hugging Face model on one task: lm_eval --model hf --model_args pretrained=EleutherAI/gpt-j-6B --tasks hellaswag --device cuda:0 --batch_size 8, where --model picks the backend, --model_args passes settings like the checkpoint name, --tasks takes comma-separated task names, --device picks the GPU, and --batch_size auto detects the largest size that fits. Pin a revision or precision inside --model_args (pretrained=EleutherAI/pythia-160m,revision=step100000,dtype='float') because "latest" can change under you and break repeatability. Always pass --output_path to save numbers, --log_samples to record model responses for failure reading, and --use_cache to resume — noting that caching is rank-dependent, so restart with the same GPU count. List available tasks with lm-eval ls tasks, and remember models without logprobs work only with generate_until tasks.

What is the difference between acc and acc_norm?

On multiple-choice tasks the model doesn't write an answer: it scores each choice's text by log-likelihood and the highest score wins, which is why prompt wording and choice length matter. Longer answer text means more tokens, so total log-likelihood is usually lower — raw accuracy (acc) can favor short choices, and acc_norm divides by length to compensate (in a toy simulation this was measured in bytes; the harness's exact normalization formula wasn't verifiable here, so treat that detail as an assumption). The failure shape is concrete: a simulated model that knows nothing scored 0.250 on both metrics when answer lengths were independent of truth, but 0.065 acc versus 0.182 acc_norm when correct answers were longer, and 0.665 acc when correct answers happened to be shorter — 66.5% for pure knowledge-free luck. With real knowledge injected, neither metric won everywhere either. So use the same metric the task recommends, and never compare acc for one model against acc_norm for another.

How many questions does a benchmark need to be trustworthy?

Enough that the standard error is smaller than the difference you care about. For a model that is truly 60% accurate: 10 questions give a standard error of 15.5 points (measured score anywhere from 30% to 90%), 100 give 4.9 points (50% to 70%), 500 give 2.2 points (56% to 64%), and 2,000 give 1.1 points (58% to 62%) — the analytic formula matched simulation at every size, and a bootstrap check agreed. That's why --limit 10 is a smoke-test flag: the README suggests small trials on closed chat APIs to check answer extraction, not to quote scores. --limit 100 cannot separate two models two points apart. Run the full task for anything you plan to publish, and report the standard error alongside every score.

What settings quietly change a benchmark score?

Seven, in descending order of surprise: model backend (the README notes vLLM output can differ from Hugging Face, which it treats as the reference — scripts/model_comparator.py exists to check); few-shot count (keep --num_fewshot fixed across every model you compare); prompt format (task YAML controls it through doc_to_text — don't edit it between runs without noting it); batch size and hardware (numerical differences can nudge scores — a caution, not a measurement here); Apple MPS support (called early-stage in the README — compare against CPU if results look odd); answer-position distribution (a position-biased model scored 0.186 when 10% of correct answers sat in position A and 0.306 when 40% did, with 0.245 at a balanced 25%); and unpinned versions on both sides — model revision and harness version. Pin everything, run the same tasks, few-shot count, and metric for all models, and write down hardware, backend, and batch size with your results.

Wrapping This Up

The harness gives you a standard way to benchmark open models yourself: install a backend extra, run lm_eval with a model and tasks, and save the samples. The hard part is control. Metric choice, answer position, few-shot count, backend, and sample size all move the number — and the toy model shows how a model that knows nothing can look good on length alone.

Run one small model on one task today, with --output_path and --log_samples on. Then read the failures before you trust the score.

Related Articles

• LLM Evaluation: Complete Guide to Testing and Measuring LLM Outputs
• LLM Evaluation Metrics Explained: Perplexity, BLEU, ROUGE, and Beyond
• How to Build a Custom Evaluation Dataset for Your LLM App
• Evaluating AI Agents: Metrics, Benchmarks, and Test Strategies