How to Build a Custom Evaluation Dataset for Your LLM App

Contents

Representative inputs, honest labels, and enough volume to see the truth

Public benchmarks tell you how a model does on someone else's questions. Your users ask different ones. So how do you build a test set that actually predicts whether your app works?

You don't need thousands of examples or a research team. You need examples that look like your real traffic, labels you can trust, and enough volume to tell a real improvement from noise. The statistics below were run on synthetic data so you can see exactly how many examples that takes — the dataset half of the full evaluation guide.

Start With What "Good" Means

Anthropic's evaluation docs say success criteria should be "specific" and "measurable," giving the example of replacing "good performance" with "accurate sentiment classification." They also note that most use cases need evaluation "along several success criteria."

Write down two or three criteria before collecting anything. For a support bot, that might be "answers from the knowledge base only," "doesn't invent policies," and "escalates when unsure." Each criterion becomes a column in your dataset.

Mine Real Traffic First

OpenAI's evaluation guidance says to "log as you develop so you can mine your logs for good eval cases." It also warns against datasets that "don't faithfully reproduce production traffic patterns."

So your first source is real inputs: production logs, support tickets, search queries, chat transcripts. Strip or mask personal data before it goes into a shared dataset.

No traffic yet? Use these, in order:

1. Examples you write yourself from people who know the domain.
2. Historical records, such as old tickets and their resolutions.
3. Synthetic cases — asking a model to expand a baseline set works, as do purchased datasets.

Treat synthetic data as a supplement. A model asked to invent user questions tends to write tidy ones. Real users write fragments, typos, and angry all-caps.

Find Your Failure Modes by Reading

This step beats any amount of clever automation. A write-up from Langfuse describes a five-step error-analysis process:

1. Pull a representative sample of traces.
2. Open coding: read each one and write a free-text note about "the first thing that went wrong."
3. Cluster the notes into named failure categories.
4. Label every trace and compute failure rates per category.
5. For each category, choose a fix: a prompt or code change, an evaluator, or monitoring.

A practitioner FAQ puts numbers on step one: review at least 100 diverse traces, annotate the first 30 yourself, and stop when new traces stop revealing new failure modes — "theoretical saturation." Both write-ups are third-party advice, not standards, but they give a concrete starting point.

Those categories become the skeleton of your dataset. Each failure mode needs its own examples.

Include the Awkward Cases on Purpose

Both vendor guides say the same thing: include typical cases, edge cases, and adversarial cases — Anthropic lists irrelevant or nonexistent input, overly long input, harmful input, and "ambiguous test cases where even humans would find it hard to reach an assessment consensus."

The checklist:

• Typical: the 60% of traffic that's routine.
• Edge: empty input, very long input, other languages, pasted tables.
• Adversarial: prompt injection, requests outside scope, attempts to extract the system prompt.
• Known past failures: every bug you've fixed becomes a permanent test.
• Ambiguous: keep a few, but mark them so they don't drag your pass rate.

Will Your Sample Even Contain the Rare Cases?

Random sampling misses rare categories more often than people expect. The math, checked by simulation:

Share of traffic	Chance a 100-example sample has none	Examples needed to see one with 95% confidence
10%	0.0%	29
5%	0.6%	59
1%	36.6%	299
0.5%	60.6%	598

import math

def miss_probability(share, n):
    return (1 - share) ** n              # P(no rare case in n random draws)

def n_for_95(share):
    return math.ceil(math.log(0.05) / math.log(1 - share))

print(f"1% category missing from 100 examples: {miss_probability(0.01, 100):.1%}")  # 36.6%
print(f"examples for 95% chance to see one: {n_for_95(0.01)}")                       # 299

A category that's 1% of traffic is absent from a 100-example sample more than a third of the time. If that 1% is "customer threatens legal action," you probably care a lot.

So add rare categories on purpose, then report them as their own slices. Don't average a rare, high-stakes category into one overall score, where it vanishes.

Label Carefully, and Check That Labelers Agree

Your labels are the ground truth. If they're noisy, your metric is noise.

Human-judgment evals provide "the highest quality but are slow and expensive" (OpenAI), so use "detailed, clear rubrics" (Anthropic), and labels for LLM-judge calibration should come from a trusted domain expert.

To check your labels, have two people label the same 50 to 100 items, then compute Cohen's kappa, which corrects raw agreement for chance (90% agreement on a balanced set gives kappa 0.80).

Here's the trap. In simulation with 90% of items "pass," two labelers who were each 90% accurate agreed on about 82% of items — which sounds good — but their kappa was only about 0.37 to 0.39. When one class dominates, raw agreement flatters you. Check kappa, and read the disagreements. They usually reveal a vague rubric.

How Many Examples Do You Need?

It depends on what you want to detect. First, how precise is a pass rate? These are 95% Wilson intervals around a measured 80%:

Examples	Interval	Half-width
30	62.7% to 90.5%	±13.9 points
100	71.1% to 86.7%	±7.8
500	76.3% to 83.3%	±3.5
2,000	78.2% to 81.7%	±1.8

import math

def wilson_interval(k, n, z=1.96):
    p = k / n
    denom = 1 + z**2 / n
    center = (p + z**2 / (2*n)) / denom
    half = z * math.sqrt(p*(1 - p)/n + z**2/(4*n**2)) / denom
    return center - half, center + half

lo, hi = wilson_interval(24, 30)          # measured 80% on 30 examples
print(f"{lo:.1%} to {hi:.1%}")             # 62.7% to 90.5% — ±13.9 points

Simulated coverage sat near 95% at every size. So 30 examples can tell you "roughly 65 to 90%" — fine for a smoke test, useless for deciding between two prompts.

Comparing Two Prompts

This is where small sets mislead. In simulation, prompt A is 80% accurate and prompt B is 85%, both scored on the same items, which vary in difficulty. How often does an evaluation actually confirm that B is better?

Examples	B scores higher	Paired test says B better
50	74%	6.4%
100	84%	14.5%
200	93%	29.6%
500	99%	64.0%
1,000	100%	92.0%

Look at the first two columns. At 100 examples, B scores higher 84% of the time, yet a proper significance test confirms it only 14.5% of the time. A five-point gain is real, and you can't prove it with a small set. A paired test (same items for both prompts) helped, but only modestly here — and the test is conservative: with identical prompts it fired 3.7% of the time, under the nominal 5%.

Use small sets to catch big regressions and obvious wins. For subtle comparisons, expect to need hundreds of examples, or accept that you're deciding without certainty — the regression testing guide covers how to wire even a small set into CI honestly.

The practitioner FAQ gives a more practical sizing guide: for LLM judges, label 100 to 200 examples per failure mode, and aim for 30 to 50 pass and 30 to 50 fail examples in each of the dev and test splits. Repeatable eval sets often grow to 100 or more examples, sized by coverage. Anthropic says to "prioritize volume over quality," arguing more questions with slightly lower-signal automated grading beat fewer hand-graded ones. That's a judgment call worth softening: volume helps only if your labels are still mostly right.

Figure 1: The error-analysis session — reading traces together finds failure modes no sampler invents

Don't Overfit to Your Own Test Set

Every time you tune a prompt against the eval set and keep the best version, you leak the set into your decisions. Simulated with every prompt variant truly 80% accurate and 100 eval examples:

Variants tried	Best measured score	Same variant on a fresh set
1	0.800	0.799
5	0.846	0.800
20	0.872	0.799

import random

rng = random.Random(42)

def best_of(n_variants, n_eval=100, true_rate=0.80):
    scores = []
    for _ in range(n_variants):
        hits = sum(rng.random() < true_rate for _ in range(n_eval))
        scores.append(hits / n_eval)       # every variant is truly identical
    return max(scores)                     # only noise differs

print(best_of(1))    # ~0.80 — one prompt, honest score
print(best_of(20))   # ~0.87 — nothing improved; noise won the contest

Nothing improved, yet the "winner" looked 7 points better. So split your data: keep a dev set for iterating and a test set you touch rarely. The FAQ suggests roughly 10 to 20% train, 40 to 45% dev, and 40 to 45% test for judge calibration — the exact ratio matters less than the habit.

Keep It Alive

Continuous evaluation grows the set over time:

• Add every production failure as a new case — the loop the feedback signals guide closes end to end.
• Version the dataset like code, so scores stay comparable.
• Refresh stale items. Policies and products change.
• Re-check labels when your rubric changes.
• Store it in your own format. OpenAI says it is deprecating its Evals platform (shutdown listed as November 30, 2026 in its docs — check current status), and a plain JSONL file survives any vendor change.

A Practical Starting Recipe

1. Write 2 or 3 success criteria.
2. Collect 100 real inputs, covering the range of traffic.
3. Read them, and write down failure modes.
4. Add 5 to 10 examples for each failure mode, including rare, high-stakes ones.
5. Label with a written rubric, and have a second person label a sample.
6. Compute kappa, and fix the rubric where you disagree.
7. Split into dev and test.
8. Run your app, and look at the failures by category.
9. Grow the set every time production surprises you — and calibrate any judge on it with the LLM-as-a-judge guide.

Common Pitfalls

• Synthetic-only datasets: models write tidy questions; real users write fragments, typos, and all-caps.
• Raw agreement without kappa: 82% agreement sounds great, but kappa was 0.37 when 90% of items pass.
• Random sampling only: a 1% failure mode is missing from 100 examples 36.6% of the time.
• Averaging rare, high-stakes categories into one overall score, where they vanish.
• Small sets for subtle comparisons: at 100 examples, B is "higher" 84% of the time but proven better 14.5% of the time.
• Tuning against the whole set: 20 variants produced a fake 7-point win.
• No dev/test split, so iteration contaminates every score you report.
• An unversioned dataset: last month's number can't be compared to this month's.
• Vague rubrics: labeler disagreements are telling you where the rubric is soft.
• Ambiguous cases counted in the pass rate instead of marked and reported separately.
• Vendor-locked storage: eval platforms get deprecated mid-project.
• Never refreshing: stale items score yesterday's app, not today's.

Unlock AI That Actually Works

Get lifetime access to GPT-6 Astra, Claude Fable 5.1, Gemini 3.5, Grok 4.5, and more — all in one platform. Build websites, apps, videos, content, and digital products from a single command. No monthly fees. No tool-hopping.

Click here to get GPTAstra Max now — one-time payment, lifetime access.

Frequently Asked Questions

How many examples does an LLM evaluation set need?

It depends on the decision you're making. For a pass rate, 95% Wilson intervals around a measured 80% look like this: 30 examples give 62.7% to 90.5% (±13.9 points), 100 give 71.1% to 86.7% (±7.8), 500 give 76.3% to 83.3% (±3.5), and 2,000 give 78.2% to 81.7% (±1.8) — so 30 examples is a smoke test and useless for deciding between two prompts. For comparisons, small sets mislead: in simulation where prompt B is genuinely 5 points better than A on the same items, B scored higher 84% of the time at 100 examples yet a paired significance test confirmed it only 14.5% of the time; you need 500 examples to confirm ~64% of the time and 1,000 for ~92%. Practical guidance from an evals FAQ: label 100 to 200 examples per failure mode for LLM judges, with 30 to 50 pass and 30 to 50 fail examples in each of the dev and test splits, growing to 100+ examples sized by coverage. Anthropic says to prioritize volume over quality — a fair instinct, but soften it: volume helps only if labels stay mostly right.

How do you find the failure modes worth testing?

By reading traces, not by automating. Pull a representative sample of production traces, open-code each one (write a free-text note about the first thing that went wrong), cluster the notes into named failure categories, label every trace and compute failure rates per category, then for each category pick a prompt or code fix, an evaluator, or monitoring. A practitioner FAQ puts numbers on the sample: review at least 100 diverse traces, annotate the first 30 yourself, and stop when new traces stop revealing new failure modes (theoretical saturation). Those named categories become the skeleton of your dataset — each failure mode needs its own examples, ideally 5 to 10 to start — and every awkward case class gets added on purpose: typical traffic, edge cases (empty input, very long input, other languages, pasted tables), adversarial inputs (prompt injection, out-of-scope requests, system-prompt extraction attempts), known past failures as permanent tests, and a few marked ambiguous cases that don't drag the pass rate.

How do you label an eval set reliably?

Write the rubric first, then check that labelers agree. Human-judgment labels are the highest quality but slow and expensive, so rubrics must be detailed and clear, and labels used to calibrate an LLM judge should come from a trusted domain expert. To verify, have two people label the same 50 to 100 items and compute Cohen's kappa, which corrects raw agreement for chance: 90% agreement on a balanced set gives kappa 0.80. The trap is class imbalance — in simulation with 90% pass items, two labelers each 90% accurate agreed on about 82% of items (sounds good) but their kappa was only 0.37 to 0.39. When one class dominates, raw agreement flatters you. Check kappa, and read the disagreements: they usually reveal a vague rubric, which you fix before relabeling.

How do you stop overfitting an eval set?

Split the data and touch the test set rarely. Every time you tune a prompt against the eval set and keep the best version, you leak the set into your decisions — simulated honestly: with every variant truly 80% accurate and 100 eval examples, trying 1 variant scores 0.800, 5 variants scores 0.846, and 20 variants scores 0.872, while the winner on a fresh set is still 0.800. Nothing improved, yet the winner looked 7 points better. Keep a dev set for iterating and a test set for final checks (an evals FAQ suggests roughly 10-20% train, 40-45% dev, 40-45% test for judge calibration — the exact ratio matters less than the habit). Then keep the set alive: add every production failure as a new case, version the dataset like code so scores stay comparable, refresh stale items when policies and products change, re-check labels when the rubric changes, and store it in your own format — OpenAI is deprecating its Evals platform (shutdown listed as November 30, 2026; check current status), and a plain JSONL file survives any vendor change.

Wrapping This Up

A good eval set is representative, covers failure modes on purpose, has labels you've checked, and is big enough for the decision you're making. Small sets are fine for smoke tests and bad at proving small improvements. Tuning against one set inflates your scores, so hold some data back.

Start small: pull 100 real inputs this week, read every one, and write down what goes wrong. That list is the beginning of your dataset.

Related Articles

• LLM Evaluation: Complete Guide to Testing and Measuring LLM Outputs
• LLM-as-a-Judge: How to Use One Model to Grade Another
• LLM Regression Testing: Catch Broken Outputs After Prompt Changes
• Feedback Loops for LLM Apps: Turn User Signals into Better Outputs