Bias in LLM-as-a-Judge: Position, Verbosity, and Self-Preference

Contents

Swap the order, watch the verdict flip — that's your ruler talking

Here's an uncomfortable experiment. Show an LLM judge two answers, then show it the same two answers in swapped order. A surprising share of the time, the verdict flips.

That tells you the judge isn't only reading the answers. It also reacts to where they sit, how long they run, and who wrote them. Treat its scores as measurements without checking those effects, and you'll ship decisions built on a biased ruler. This guide covers the three best-documented biases — position, verbosity, and self-preference — plus a few others, and shows how to test for them and fix them. Reported effect sizes vary widely across studies, so the goal here is measuring the bias in your judge instead of quoting one magic number — the deeper calibration workflow lives in our LLM-as-a-judge guide. The methods come from published NLP research and standard instrument-calibration practice, so check the cited studies and library docs for your exact setup.

Why Teams Use LLM Judges (and Why That's Fine)

LLM-as-a-judge means you ask a strong model to grade another model's output. The idea took off after researchers showed that strong judges can agree with human raters at rates close to human-to-human agreement. Judges scale to thousands of outputs for a fraction of the cost of human review.

The technique works best for relative comparisons between two outputs, not for absolute quality scores. It works well as a screening tool, and it doesn't replace human judgment for high-stakes decisions.

None of this means you should avoid judges. It means you should calibrate them, the way you'd calibrate any instrument.

The Big Three Biases

The paper that popularized LLM judges (Zheng and colleagues, on MT-Bench and Chatbot Arena) named three core failure modes. Later research confirmed and extended them.

Position Bias

What it is: the judge favors an answer because of its slot, often the first one, regardless of content.

Why it matters: in pairwise evals, a position preference can decide the winner. If your new model always sits in slot B, your results could reflect the slot instead of the model.

What research says: the effect varies by judge model and by task. It stays smaller for stronger judges and when quality gaps between answers run large. It grows when the two answers sit close in quality — exactly the situation where you need the judge most.

How to test: run every comparison twice with the order swapped. Count how often the verdict flips.

How to fix it:

• Swap and require consistency. Count a win only when the answer wins in both orders. Treat inconsistent verdicts as ties.
• Randomize order across your dataset so any leftover bias averages out.

def judge_pair(judge, prompt, a, b):
    first = judge(prompt, a, b)    # returns "A" or "B"
    second = judge(prompt, b, a)
    if first == "A" and second == "B":
        return "a_wins"
    if first == "B" and second == "A":
        return "b_wins"
    return "tie"  # inconsistent verdicts count as a tie

The inconsistency rate doubles as a metric. If 30% of verdicts flip on a swap, you've learned how noisy your judge runs.

Verbosity Bias

What it is: the judge rewards longer answers, even when the extra words add nothing.

Why it matters: longer outputs cost more, read slower, and sometimes hide errors. If your judge loves length, your optimization loop will happily make everything wordier.

What research says: early studies showed that several judges preferred longer, repetitive answers over shorter, equivalent ones, and that some strong judges resisted this trick better than others. Reported effect sizes differ across studies, so measure your own.

How to test: take a good answer, pad it with filler that adds no information, and ask the judge to compare the padded and original versions. A fair judge calls it a tie or prefers the original.

How to fix it:

• Write length into the rubric. Tell the judge to penalize padding, repetition, and unnecessary detail.
• Control for length statistically. Compare answers of similar length, or adjust scores for length, as arena leaderboards do with style control.
• Set length budgets so both candidates operate under the same constraint.

Self-Preference Bias

What it is: the judge favors outputs from itself or from similar models.

Why it matters: if you grade your model family with a judge from the same family, you may inflate your own results without knowing it.

What research says: work by Wataoka and colleagues links self-preference to perplexity. Judges favor text that feels familiar and predictable to them, which includes their own writing. The evidence isn't universal, though — some judges show little self-favoritism, and effects depend on the models and tasks.

How to test: have the judge grade equivalent answers written by different model families, with identities hidden. A consistent gap in favor of its own family points to self-preference.

How to fix it:

• Use a judge from a different family than the models you test.
• Use a panel of judges from several families, and aggregate their votes.
• Hide answer identity. Strip signatures, model names, and obvious style tells where you can.

Other Biases Worth Knowing

The big three get the attention, but they're not the whole list. A framework called CALM (from the "Justice or Prejudice" study) catalogs a dozen bias types by perturbing inputs and watching whether judgments change. Watch for these too:

• Format and style bias: judges reward confident, polished, heavily formatted answers, even when the content is wrong. This quietly ruins A/B tests where one variant changed the format.
• Authority bias: a fake citation or an impressive-sounding source can sway the verdict.
• Sentiment bias: judges may prefer positive or negative emotional tone.
• Leniency: judges often hand out high scores too easily, which compresses your scale.
• Calibration drift: a silent judge-model update changes your scores, so last month's numbers stop matching this month's.
• Weak verification: judges can miss errors in math, code, or facts they can't check, and they may still sound sure.

Solving the big three doesn't certify a judge as unbiased. It removes the biases you can measure most cleanly.

Build a Bias Test Suite for Your Judge

Don't take any paper's numbers on faith. Test your own judge on your own task. A small suite catches most problems:

1. Swap test: run each pair in both orders, and track the flip rate.
2. Padding test: add filler to one answer, and see whether the judge changes its pick.
3. Identity test: compare outputs from different model families with hidden names.
4. Format test: rewrite one answer with bullets and bold headers, keeping the content identical.
5. Planted-error test: insert a factual error into a good answer, and check whether the judge catches it.
6. Human agreement test: compare judge verdicts against human labels on 100 or more items, and compute agreement with Cohen's kappa.

The swap test first pass looks like this:

def flip_rate(judge, pairs):
    """Run every pair in both orders; count verdict flips."""
    flips = 0
    for prompt, a, b in pairs:
        first = judge(prompt, a, b)    # slot A vs slot B
        second = judge(prompt, b, a)   # same pair, swapped
        if first == second:            # same SLOT won twice — a flip
            flips += 1
    return flips / len(pairs)          # 0.30 = 30% of verdicts are position-driven

The planted-error test is the most underrated. If a judge can't spot an error you inserted yourself, it can't police your model's errors either. Measuring judge-versus-human agreement is the same exercise we covered in human evaluation practice: label 100 items by hand first, then compute kappa.

Figure 1: A judge panel only helps if the panelists disagree with each other — and you measure how often

Stack Your Mitigations

The cheap fixes work better together. A practical recipe:

• Write a specific rubric with defined criteria and score anchors, and include explicit length and format guidance.
• Swap positions and keep only consistent verdicts.
• Use a different-family judge, or a small panel.
• Prefer pairwise comparison for relative quality, and add reference answers or checklists for correctness.
• Ask for evidence. Require the judge to quote the text that supports each claim, which reduces vague, vibes-based verdicts.
• Pin the judge version and rerun a fixed calibration set whenever you change it.
• Spot-check with humans on a regular sample.

Know When Not to Use a Judge

Some tasks fit judges poorly. Skip or limit them when:

• A deterministic check exists. Use unit tests for code and exact match for structured answers.
• The stakes run high. Medical, legal, and safety decisions need human review.
• You need absolute scores. Judges handle relative comparisons better.
• Ground truth is checkable. Verify facts against sources instead of asking a model to guess.

Run Judges Responsibly in Production

• Sample human review. Check 5 to 20% of judged items by hand, and track judge-versus-human agreement over time.
• Log judge prompts and versions so you can reproduce old results.
• Report uncertainty. Show confidence intervals, not single scores.
• Recalibrate on schedule. Judge updates and new data both shift behavior.

Common Pitfalls

• Trusting one judge run — one ordering hides position bias.
• Judging a model with its own family — self-preference inflates the score.
• Ignoring length — your optimization loop rewards padding.
• Skipping human calibration — a judge you haven't tested is just a confident opinion.
• Changing the judge silently — scores from different judges don't compare.
• Treating one bias fix as complete — other biases remain.
• Reading verdicts without a rubric — vibes-based grading drifts toward polish over substance.
• Never running the planted-error test — unverified judges pass fluent wrong answers.

Unlock AI That Actually Works

Get lifetime access to GPT-6 Astra, Claude Fable 5.1, Gemini 3.5, Grok 4.5, and more — all in one platform. Build websites, apps, videos, content, and digital products from a single command. No monthly fees. No tool-hopping.

Click here to get GPTAstra Max now — one-time payment, lifetime access.

Frequently Asked Questions

What are the three main biases in LLM-as-a-judge?

Position bias: the judge favors an answer because of its slot, often the first one, regardless of content — run every comparison twice with the order swapped and count how often the verdict flips; count a win only when the answer wins in both orders, and treat inconsistent verdicts as ties. Verbosity bias: the judge rewards longer answers even when the extra words add nothing — test it by padding a good answer with filler and checking whether the verdict changes; fix it by writing length into the rubric, comparing similar-length answers or adjusting scores statistically (as arena style control does), and setting equal length budgets. Self-preference bias: the judge favors outputs from itself or similar models — research by Wataoka and colleagues links this to perplexity, since judges favor text that feels familiar and predictable, including their own writing; test it with hidden identities across model families, and fix it by using a different-family judge, a panel of judges, or stripped signatures and style tells. All three grow when you need the judge most: when two answers sit close in quality.

How do you test an LLM judge for bias?

Run a six-test bias suite on your own judge, on your own task: (1) swap test — run each pair in both orders and track the flip rate; if 30% of verdicts flip on a swap, you know how noisy your judge runs. (2) padding test — add filler to one answer and see whether the judge changes its pick; a fair judge calls it a tie or prefers the original. (3) identity test — compare outputs from different model families with hidden names and look for a consistent gap favoring the judge's own family. (4) format test — rewrite one answer with bullets and bold headers, content identical, and check the verdict. (5) planted-error test — insert a factual error into a good answer and see whether the judge catches it; a judge that can't spot an error you planted can't police your model's errors. (6) human agreement test — compare judge verdicts against human labels on 100 or more items and compute Cohen's kappa. Don't take any paper's numbers on faith: reported effect sizes vary widely across studies, so measure the bias in your judge instead of quoting one magic number.

What mitigations reduce judge bias in practice?

The cheap fixes stack: write a specific rubric with defined criteria, score anchors, and explicit length and format guidance that penalizes padding and repetition; swap positions and keep only consistent verdicts; use a judge from a different model family than the models you test, or a small panel of families; prefer pairwise comparison for relative quality, adding reference answers or checklists for correctness; ask the judge to quote the text that supports each claim, which reduces vague, vibes-based verdicts; pin the judge version and rerun a fixed calibration set whenever you change it; and spot-check with humans on a regular sample — check 5 to 20% of judged items by hand and track judge-versus-human agreement over time. Also know when not to use a judge: when a deterministic check exists (unit tests, exact match), when stakes run high (medical, legal, safety need human review), when you need absolute scores (judges handle relative comparisons better), and when ground truth is checkable against sources.

Why do teams use LLM judges at all?

LLM-as-a-judge means asking a strong model to grade another model's output — the idea took off after researchers (Zheng and colleagues, on MT-Bench and Chatbot Arena) showed strong judges can agree with human raters at rates close to human-to-human agreement. Judges scale to thousands of outputs for a fraction of the cost of human review, which makes them ideal as screening tools in the broader evaluation pipeline. But the technique works best for relative comparisons between two outputs, not absolute quality scores, and it doesn't replace human judgment for high-stakes decisions. None of that means you should avoid judges — it means you should calibrate them the way you'd calibrate any instrument. Beyond the big three, watch a longer tail too: format and style bias (polished answers win even when wrong), authority bias (fake citations sway verdicts), sentiment bias, leniency that compresses your scale, calibration drift from silent judge-model updates, and weak verification of math, code, or facts the judge can't actually check. Solving the big three doesn't certify a judge as unbiased — it removes the biases you can measure most cleanly.

Wrapping This Up

LLM judges give you speed and scale, and they come with predictable biases: position, verbosity, and self-preference, plus a longer tail of subtler ones. You can measure each bias and reduce most of them with a few cheap habits.

Start with the swap test and the padding test this week. Those two checks take an hour, and they'll tell you whether your judge deserves the authority you give its scores.

Related Articles

• LLM-as-a-Judge: How to Use One Model to Grade Another
• How LLM Arenas Work: Pairwise Comparison and Elo Ratings Explained
• Human Evaluation for LLMs: Annotation Guidelines and Inter-Annotator Agreement
• LLM Evaluation: Testing and Measuring Outputs the Right Way