Human Evaluation for LLMs: Annotation Guidelines and Inter-Annotator Agreement

Contents

Your gold standard is only as good as your agreement score

Two reviewers read the same chatbot answer. One rates it a 5. The other rates it a 2. Both feel confident, and both can defend their score. Welcome to human evaluation.

Humans stay the ground truth for LLM quality, and they also disagree with each other constantly. If you don't manage that disagreement, your "gold standard" turns into noise with a spreadsheet attached. This guide shows you how to write annotation guidelines, train reviewers, and measure inter-annotator agreement so your human labels deserve trust — the human layer of the evaluation guide and the calibration source every LLM judge depends on. The methods come from established practice in NLP and the social sciences, so check the cited metric definitions and library docs for your exact setup.

Why Human Evaluation Still Matters

Automated metrics and LLM judges scale well, but they need calibration against something. That something is human judgment.

Humans catch what machines miss: subtle factual distortions, tone problems, unhelpful-but-fluent answers, and cultural context. They also give you the labels you need to test whether your automated judges work at all.

But human labels only help when they're reliable. A label that depends on which reviewer picked up the task tells you about the reviewer, not the model.

Choose the Right Task Design

Your first decision shapes everything else: how will reviewers express judgment? Three designs dominate.

Absolute Ratings — reviewers score each output on a scale, such as 1 to 5 for helpfulness.

• Strength: you get a score for every output, and you can track trends over time.
• Weakness: people interpret scales differently. One reviewer's 4 equals another's 3.

Pairwise Comparison — reviewers see two outputs and pick the better one, or declare a tie.

• Strength: people compare more consistently than they rate, so agreement usually runs higher.
• Weakness: it scales poorly across many systems, and it tells you which is better without saying how good either one is.

Binary or Categorical Labels — reviewers answer a yes/no question or pick a category: "Does the answer contain an unsupported claim?"

• Strength: simple questions produce the highest agreement, and the results map cleanly to pass/fail decisions.
• Weakness: they capture less nuance.

Binary questions beat 1-to-5 scales for most production work. Break big judgments into small, concrete questions, and agreement improves.

Write Guidelines That Remove Ambiguity

Good guidelines turn vague opinions into repeatable decisions. A strong guideline document includes:

• Purpose: what decision will these labels inform?
• Definitions: a plain-language definition of every dimension and label.
• Decision rules: explicit rules for borderline cases, like "if an answer is correct but omits a required warning, mark it incomplete."
• Examples: at least three per label, including tricky edge cases.
• Counterexamples: cases that look like a label but don't qualify.
• An escape hatch: an "unsure" or "can't judge" option, with rules for when to use it.
• Context rules: what reviewers should assume about the user and the use case.

Anchor Every Scale Point

If you use a scale, define each point, not only the endpoints. Compare:

• Weak: "Rate faithfulness from 1 (bad) to 5 (good)."
• Strong: "1 = contains a clear factual error. 3 = contains a minor inaccuracy that doesn't change the meaning. 5 = every claim matches the source."

The strong version leaves much less room for personal taste.

Rate One Dimension at a Time

Asking for a single "overall quality" score blends accuracy, tone, and style into one number nobody can interpret. Split them. Reviewers can judge faithfulness and fluency separately, and then you can see which one failed.

Select and Train Your Annotators

Who annotates matters as much as what they annotate:

• Match expertise to the task. Fluency and tone often work with general reviewers. Medical, legal, or code correctness needs domain experts.
• Run a training session. Walk through the guidelines, label five to ten examples together, and discuss every disagreement.
• Give a qualification test. Have candidates label items with known answers, and keep only those who pass.
• Pay fairly. Rushed, underpaid reviewers produce rushed, sloppy labels.
• Limit fatigue. Cap session length, because quality drops over long sittings.

Run a Pilot Before the Real Thing

Never launch a full annotation round cold. A pilot exposes guideline gaps while fixing them still costs little:

1. Sample 50 to 100 items that represent your real data, including hard ones.
2. Have two or three annotators label them independently.
3. Compute agreement (details below).
4. Hold a discussion about every disagreement.
5. Revise the guidelines to cover each gap you find.
6. Repeat until agreement reaches your target.

Most disagreement traces back to unclear guidelines, not careless people.

Measure Inter-Annotator Agreement

Agreement tells you whether your labels are reliable. Several metrics exist, each fitting a different situation.

Percent Agreement

The simplest metric counts how often raters give the same label. It's easy to understand and easy to fool: two raters who guess randomly on a two-label task will still agree about half the time, so percent agreement overstates reliability.

Cohen's Kappa (Two Raters)

Cohen's kappa corrects for chance agreement: kappa = (observed agreement − chance agreement) / (1 − chance agreement). A value of 1 means perfect agreement; 0 means no better than chance. For ordinal scales, use weighted kappa, which penalizes big disagreements (a 1 versus a 5) more than small ones (a 4 versus a 5).

Fleiss' Kappa (Many Raters)

Fleiss' kappa extends the idea to three or more raters on categorical labels. It assumes a fixed number of raters per item, which doesn't always hold in practice.

Krippendorff's Alpha (The Flexible Option)

Krippendorff's alpha handles any number of raters, missing labels, and different data types: nominal, ordinal, interval, or ratio. That flexibility makes it a favorite for real annotation projects, where not every rater labels every item:

from sklearn.metrics import cohen_kappa_score
import krippendorff

# Two raters, categorical labels
kappa = cohen_kappa_score(rater_a, rater_b)

# Two raters, ordinal scale, weighted
weighted = cohen_kappa_score(rater_a, rater_b, weights="quadratic")

# Many raters, missing values allowed (rows = raters, columns = items)
alpha = krippendorff.alpha(
    reliability_data=matrix,
    level_of_measurement="ordinal",
)

Interpret Agreement Scores With Care

People love thresholds, but every threshold is a rough convention. Common guidance:

• Krippendorff's suggestion: alpha of 0.80 or higher counts as reliable; values from about 0.67 to 0.80 allow only tentative conclusions.
• Landis and Koch's scale for kappa: 0.61 to 0.80 counts as substantial agreement, above 0.80 as almost perfect. Many researchers criticize these labels as arbitrary.

Treat any threshold as a starting point and weigh the stakes. A safety-critical label deserves a higher bar than a style preference.

The Kappa Paradox

Here's a trap. Suppose 95% of your items deserve the same label, and your raters agree on 95% of items. Kappa can still come out low, because chance agreement runs high when one label dominates:

import numpy as np
from sklearn.metrics import cohen_kappa_score

# 100 items: 95 "good", 5 "bad". Raters agree on 95 of 100 items...
rater_a = ["good"] * 95 + ["bad"] * 5
rater_b = ["bad"] + ["good"] * 94 + ["bad"] + ["good"] * 4

print((np.array(rater_a) == np.array(rater_b)).mean())          # 0.95 percent agreement
print(round(cohen_kappa_score(rater_a, rater_b), 2))            # 0.26 kappa — barely "fair"

When your labels skew heavily, report percent agreement alongside kappa, and consider breaking out agreement for the rare class. One number never tells the whole story.

Figure 1: Adjudication in progress — the discussion is where guidelines actually improve

Handle Disagreement Productively

Disagreement isn't only a problem to eliminate. It also carries information:

• Adjudicate contested items with a senior reviewer or a group discussion, and record the reasoning.
• Update the guidelines with every resolved edge case.
• Keep the raw labels, not only the resolved ones. They show where your task stays genuinely ambiguous.
• Treat persistent disagreement as a finding. Some tasks, like creative quality, involve real subjectivity, and a single "correct" label doesn't exist.

Control for Common Biases

Human judges carry biases you can reduce with good design:

• Position bias: in pairwise tasks, randomize which output appears first.
• Length bias: reviewers often prefer longer answers. Control for length or tell reviewers to ignore it.
• Anchoring: early items influence later ratings, so shuffle item order.
• Halo effect: one strong trait lifts every other rating. Rate dimensions separately.
• Source bias: hide which model produced each output.

Keep Quality High During the Real Run

Agreement doesn't stop mattering after the pilot. Monitor it throughout:

• Insert gold items with known answers, and track each annotator's accuracy on them.
• Overlap a sample. Have multiple annotators label 10 to 20% of items to keep measuring agreement.
• Review drift. Compare early and late labels, because standards slide over time.
• Give feedback. Share errors with annotators and refine the guidelines.

Use LLMs to Help, Not Replace

LLM-assisted annotation can speed things up — pre-labeling items or drafting rationales for humans to confirm. It also introduces risk: reviewers may anchor on the model's suggestion and agree too easily.

If you use LLM assistance, measure it. Compare agreement and accuracy with and without suggestions, and keep a fully human-labeled sample as a control.

Document and Version Everything

Treat your annotation setup like code:

• Version the guidelines, with a changelog.
• Record annotator IDs (anonymized) and labeling dates.
• Store raw and adjudicated labels separately.
• Report agreement scores with every dataset you share.

This habit makes your results reproducible and your future audits painless.

Common Pitfalls

• Skipping the pilot — guideline gaps surface after you've paid for the full round.
• Using vague scales — "good to bad" means something different to everyone.
• Reporting only percent agreement — it overstates reliability.
• Ignoring disagreement — resolved labels hide useful signals.
• Rating overall quality only — you can't fix a blended score.
• Underpaying annotators — quality follows effort, and effort follows pay.
• Forgetting the kappa paradox — skewed labels can tank kappa despite 95% agreement.
• Letting reviewers see model identity — source bias inflates brand-favored scores.

Unlock AI That Actually Works

Get lifetime access to GPT-6 Astra, Claude Fable 5.1, Gemini 3.5, Grok 4.5, and more — all in one platform. Build websites, apps, videos, content, and digital products from a single command. No monthly fees. No tool-hopping.

Click here to get GPTAstra Max now — one-time payment, lifetime access.

Frequently Asked Questions

Why does human evaluation still matter for LLMs?

Automated metrics and LLM judges scale well, but they need calibration against something — and that something is human judgment. Humans catch what machines miss: subtle factual distortions, tone problems, unhelpful-but-fluent answers, and cultural context. They also provide the labels you need to test whether automated judges work at all — the judge calibration routine is built on human labels. But human labels only help when they're reliable: a label that depends on which reviewer picked up the task tells you about the reviewer, not the model. Reliability comes from task design, sharp guidelines, trained annotators, a pilot round, and agreement metrics that correct for chance — percent agreement alone overstates it, since two random guessers agree half the time on a two-label task.

How do you write annotation guidelines that remove ambiguity?

A strong guideline document includes seven parts: purpose (what decision these labels inform), definitions (plain-language definition of every dimension and label), decision rules for borderline cases ("if an answer is correct but omits a required warning, mark it incomplete"), at least three examples per label including tricky edge cases, counterexamples (cases that look like a label but don't qualify), an escape hatch ("unsure" or "can't judge" with rules for when to use it), and context rules (what reviewers should assume about the user and use case). If you use a scale, anchor every point, not just the endpoints: "1 = contains a clear factual error, 3 = minor inaccuracy that doesn't change meaning, 5 = every claim matches the source" leaves far less room for personal taste than "rate 1 to 5". And rate one dimension at a time — a single overall-quality score blends accuracy, tone, and style into a number nobody can interpret.

How do you measure inter-annotator agreement correctly?

Pick the metric for your situation. Percent agreement counts same-label rates — easy to grasp but easy to fool, since random guessers agree about half the time on two labels. Cohen's kappa (two raters) corrects for chance: kappa = (observed − chance) / (1 − chance), where 1 is perfect and 0 is chance-level; use weighted kappa for ordinal scales so a 1-vs-5 disagreement hurts more than a 4-vs-5. Fleiss' kappa extends to three or more raters on categorical labels with a fixed raters-per-item assumption. Krippendorff's alpha handles any number of raters, missing labels, and nominal/ordinal/interval/ratio data — the flexible favorite for real projects. Interpret with care: Krippendorff suggests alpha ≥ 0.80 reliable, 0.67–0.80 tentative; Landis & Koch call kappa 0.61–0.80 substantial and >0.80 almost perfect (widely criticized as arbitrary). Every threshold is a rough convention — weigh the stakes. And beware the kappa paradox: when 95% of items share one label, raters agreeing on 95% of items can still yield kappa around 0.26, because chance agreement runs high; report percent agreement alongside kappa and break out the rare class.

How do you control quality and bias during a real annotation run?

Design out the known biases: randomize output order in pairwise tasks (position bias), control for length or tell reviewers to ignore it (length bias), shuffle item order (anchoring), rate dimensions separately (halo effect), and hide which model produced each output (source bias). Then monitor during the run: insert gold items with known answers and track each annotator's accuracy, overlap 10–20% of items across annotators to keep measuring agreement, compare early versus late labels for drift, and give feedback while refining the guidelines. If you use LLM-assisted pre-labeling, measure it — reviewers can anchor on the model's suggestion and agree too easily — by comparing agreement and accuracy with and without suggestions, keeping a fully human-labeled sample as control. Treat the whole setup like code: version the guidelines with a changelog, record anonymized annotator IDs and dates, store raw and adjudicated labels separately, and report agreement scores with every dataset you share.

Wrapping This Up

Human evaluation only works when humans agree for reasons you can explain. Write sharp guidelines, train your reviewers, test the setup with a pilot, and measure agreement with metrics that correct for chance.

Start with one dimension, one page of guidelines, and fifty items labeled by two people. Whatever agreement score you get will teach you more about your task than weeks of debating the guidelines.

Related Articles

• LLM-as-a-Judge: How to Use One Model to Grade Another
• How to Build a Custom Evaluation Dataset for Your LLM App
• Evaluating LLM Summarization: ROUGE, BERTScore, and Human Review
• LLM Hallucination Detection: Methods and Metrics That Work