How LLM Arenas Work: Pairwise Comparison and Elo Ratings Explained

Contents

Millions of blind votes, one ranking — here's the math behind it

Imagine a blind taste test. You sip two sodas, you don't know the brands, and you pick the one you like better. Now repeat that millions of times with chatbots instead of soda, and you have an LLM arena.

Arena-style leaderboards shape how people talk about which model is "best." Vendors cite them in launch posts, and developers use them to shortlist models. Yet few people know how the numbers actually come together, or what they do and don't mean. This guide explains pairwise comparison, Elo ratings, and the Bradley-Terry model, then shows you where arena rankings mislead and how to build your own — it's the popularity-driven cousin of the formal evaluation methods covered on this site. The best-known arena has changed names (more below) and its numbers update constantly, so check its live site for current figures.

What an LLM Arena Is

An LLM arena ranks models by human preference in head-to-head battles. A user types a prompt, two anonymous models each answer, and the user votes for the better response. The system reveals the model names only after the vote.

The most famous example began as Chatbot Arena at LMSYS, then became LMArena as an independent company. According to one 2026 explainer, it rebranded again to simply "Arena" on January 28, 2026. As of April 2026, that explainer reported nearly 5.8 million votes across 339 models on the text leaderboard.

Scale matters here. Individual votes are noisy, but millions of them produce stable rankings.

Why Pairwise Comparison Works

Asking people to rate an answer from 1 to 5 sounds natural, and it works badly. Reviewers use scales differently, and they drift over time. Ask two people to rate the same answer, and you often get two different numbers.

Pairwise comparison sidesteps that problem. People find it much easier to say "A beats B" than to assign a calibrated score. That judgment is quick, intuitive, and more consistent across voters — the same reason human annotation guides push binary questions over vague scales.

The trade-off: a single battle tells you only who won, not by how much or why. You recover the rest by collecting many battles and fitting a statistical model.

Elo Ratings in Plain English

Elo comes from chess. Every player holds a rating, and the rating difference predicts the outcome of a match. A win against a stronger opponent earns more points than a win against a weaker one.

The Core Formula

The expected score for player A against player B is:

E(A) = 1 / (1 + 10^((R_B − R_A) / 400))

Some examples help:

• Equal ratings: each side has a 50% expected win rate.
• A 200-point gap: the higher-rated model should win about 76% of the time.
• A 400-point gap: the higher-rated model should win about 91% of the time.

After each match, the ratings move toward the actual result. If a model wins more than expected, its rating rises. If it loses to a weaker opponent, it drops:

def expected(r_a, r_b):
    return 1 / (1 + 10 ** ((r_b - r_a) / 400))

def update(r_a, r_b, score_a, k=32):
    """score_a: 1 for a win, 0.5 for a tie, 0 for a loss."""
    e_a = expected(r_a, r_b)
    new_a = r_a + k * (score_a - e_a)
    new_b = r_b + k * ((1 - score_a) - (1 - e_a))
    return new_a, new_b

The constant k controls how fast ratings move. A large k reacts quickly and bounces around. A small k moves slowly and steadily.

The Problem With Online Elo

Classic Elo updates after every match, so the order of matches affects the final ratings. That's fine for chess, where players improve over time. It's awkward for models, which stay fixed while you measure them. Two orderings of the same battles can yield different leaderboards.

Why Arenas Use Bradley-Terry Instead

The modern arena leaderboard uses the Bradley-Terry model, according to the arena's own description as summarized in recent explainers. The scores then pass through a transformation so they land in an Elo-like range. That's why the leaderboard looks like chess Elo, even though the math underneath differs.

How Bradley-Terry Works

The model gives each LLM a single strength parameter. The probability that model A beats model B depends only on the gap between those two strengths:

P(A beats B) = 1 / (1 + e^(−(β_A − β_B)))

If that looks familiar, it should. It's logistic regression. You collect all battles, then find the strengths that make the observed results most likely, using maximum likelihood.

The key advantage: the result doesn't depend on the order of the battles. You fit all the data at once and get one consistent answer:

import numpy as np
from sklearn.linear_model import LogisticRegression

def bradley_terry(battles, models):
    """battles: list of (model_a, model_b, a_wins) with ties removed."""
    idx = {m: i for i, m in enumerate(models)}
    X = np.zeros((len(battles), len(models)))
    y = np.zeros(len(battles))
    for row, (a, b, a_wins) in enumerate(battles):
        X[row, idx[a]] = 1
        X[row, idx[b]] = -1
        y[row] = a_wins
    lr = LogisticRegression(fit_intercept=False, C=1e6).fit(X, y)
    return 400 / np.log(10) * lr.coef_[0] + 1000  # Elo-like scale

This sketch drops ties for simplicity. Production systems handle ties explicitly.

Confidence Intervals

A rating without uncertainty misleads. Arena-style leaderboards show scores with margins, like "1504 ± 5." You can compute these by bootstrapping: resample the battles with replacement, refit the model many times, and measure the spread of each rating.

When two models' intervals overlap, you can't say which one is truly better — and this is the most ignored feature of every leaderboard. A 3-point gap with overlapping intervals means a statistical tie, whatever the rank column says.

What the Score Actually Means

The rating measures expected head-to-head human preference in that arena's battles. It does not measure overall intelligence, accuracy, or usefulness for your task.

Translate rating gaps into win probabilities with the Elo formula. A 50-point gap means roughly a 57% expected win rate. That's a real edge, but far from a blowout. Many users won't notice it in daily use.

Style Control: Fixing the Length Problem

Humans like longer, more polished, heavily formatted answers, even when those answers aren't better. That bias lets a model climb the leaderboard by writing more bullet points and bold text.

Arena addresses this with style control, a statistical adjustment that estimates and removes the effect of response length and formatting. The result: a raw leaderboard and a style-controlled leaderboard coexist, and model order can shift slightly between them.

When you compare models, look at the style-controlled ranking first. It tells you more about substance.

Figure 1: Head-to-head judgment — the same "which is better?" question, at arena scale

Where Arena Rankings Mislead

Arenas offer valuable signal, and they carry real limits. Keep these in mind:

• Voter population: voters skew toward online, tech-savvy users, and they may not resemble your customers.
• Prompt mix: arena prompts don't match your workload. Coding, medical, or legal tasks may behave differently.
• Vote quality: voters don't verify facts. They often prefer a confident, well-written wrong answer over a clumsy correct one — the failure mode hallucination detection exists to catch.
• Single-turn bias: many battles involve short interactions, which hide problems that appear in long conversations or agent workflows.
• Gaming risk: models can tune for arena preferences, and sampling choices can favor some vendors. Researchers have published critiques about uneven access and private testing practices, so read the arguments yourself.
• Narrow target: preference isn't safety, cost, latency, or reliability.

How to Use Arena Rankings Wisely

• Use them to shortlist, not to decide. Pick three to five candidates from the top and from your price range.
• Check the category. Arenas often publish subranks for coding, math, long prompts, or other categories.
• Respect the intervals. Treat overlapping scores as ties.
• Prefer style-controlled scores.
• Then test on your own data. Always.

Build Your Own Mini-Arena

Public arenas won't reflect your use case, so run a private one. The process takes a day or two:

1. Collect 100 real prompts from your product logs, with personal data removed.
2. Generate answers from each candidate model with identical settings.
3. Randomize pairs and positions, and hide model names.
4. Have reviewers pick the better answer for each pair, or declare a tie.
5. Collect multiple votes per pair from different reviewers.
6. Fit a Bradley-Terry model and bootstrap confidence intervals.
7. Inspect the losses. Read the answers that lost, and find out why.

Pair this with the human-evaluation practices that keep reviewers consistent — clear guidelines and agreement checks. You can also use an LLM judge for pairwise votes, but randomize answer order, since judges show position bias, and validate the judge against human votes first — the same caution our judge calibration guide details.

Common Pitfalls

• Treating Elo as absolute quality — it's relative preference inside one pool.
• Ignoring confidence intervals — a tiny gap means nothing.
• Reading rank, not rating — rank 3 and rank 7 may sit within noise of each other.
• Forgetting style bias — longer answers win unfairly.
• Choosing a model from one leaderboard — cross-check with task-specific evals; leaderboard scores alone mislead for familiar reasons.
• Running a private arena without randomization — position effects will skew your votes.
• Confusing preference with capability — liking an answer isn't verifying it.
• Never reading the method page — the math changed once already, and it will again.

Unlock AI That Actually Works

Get lifetime access to GPT-6 Astra, Claude Fable 5.1, Gemini 3.5, Grok 4.5, and more — all in one platform. Build websites, apps, videos, content, and digital products from a single command. No monthly fees. No tool-hopping.

Click here to get GPTAstra Max now — one-time payment, lifetime access.

Frequently Asked Questions

What is an LLM arena and how does voting work?

An LLM arena ranks models by human preference in head-to-head battles: a user types a prompt, two anonymous models each answer, and the user votes for the better response — names are revealed only after the vote. The most famous example began as Chatbot Arena at LMSYS, became LMArena as an independent company, and one 2026 explainer reports it rebranded again to simply "Arena" on January 28, 2026, with nearly 5.8 million votes across 339 models on the text leaderboard as of April 2026. Scale is the point: individual votes are noisy, but millions produce stable rankings. Pairwise comparison works because people find "A beats B" much easier than a calibrated 1-to-5 score — judgments are quick, intuitive, and far more consistent across voters. The trade-off: one battle tells you only who won, not by how much or why. You recover the rest by collecting many battles and fitting a statistical model.

How do Elo ratings and the Bradley-Terry model differ?

Classic Elo comes from chess: every player holds a rating, the difference predicts match outcomes via E(A) = 1 / (1 + 10^((R_B − R_A) / 400)), and ratings move after each game by k × (actual − expected). Equal ratings mean 50% expected win rate, a 200-point gap means about 76%, a 400-point gap about 91%. The problem for arenas: online Elo updates sequentially, so the order of matches changes the final leaderboard — fine for chess where players improve, awkward for fixed models you're just measuring. The modern arena leaderboard uses the Bradley-Terry model instead: each model gets one strength parameter, P(A beats B) = 1 / (1 + e^(−(β_A − β_B))), fitted by maximum likelihood over all battles at once — logistic regression under another name. The result doesn't depend on battle order, then passes through a transformation so scores land in an Elo-like range, which is why the leaderboard looks like chess Elo even though the math underneath differs.

What do arena scores actually measure, and where do rankings mislead?

The rating measures expected head-to-head human preference in that arena's battles — not overall intelligence, accuracy, or usefulness for your task. Translate gaps into win probabilities with the Elo formula: a 50-point gap means roughly a 57% expected win rate, a real edge but far from a blowout. Limits to keep in mind: voters skew toward online, tech-savvy users who may not resemble your customers; the prompt mix won't match your workload; voters don't verify facts and often prefer a confident, well-written wrong answer over a clumsy correct one; many battles are single-turn, hiding problems that appear in long conversations and agent workflows; models can tune for arena preferences, with critiques published about uneven access and private testing practices; and preference isn't safety, cost, latency, or reliability. Style control matters too — humans like longer, more formatted answers, so check the style-controlled ranking first, treat overlapping confidence intervals as ties, and use arenas to shortlist three to five candidates rather than to decide.

How do you build your own mini-arena?

Collect 100 real prompts from your product logs with personal data removed, generate answers from each candidate model with identical settings, randomize pairs and positions, and hide model names. Have reviewers pick the better answer per pair (or declare a tie), collect multiple votes per pair from different reviewers, fit a Bradley-Terry model, and bootstrap confidence intervals by resampling battles and refitting many times. Then inspect the losses — read the answers that lost and find out why. Pair this with human-evaluation practices that keep reviewers consistent: clear guidelines, anchored scales, and agreement checks. You can hand pairwise votes to an LLM judge, but randomize answer order since judges show position bias, and validate the judge against human votes first. During the run, compare confidence intervals rather than ranks, prefer style-controlled scores, and cross-check with task-specific evals on your own data before choosing.

Wrapping This Up

LLM arenas turn millions of blind head-to-head votes into a ranking. Pairwise comparison makes the judgments easier, and Bradley-Terry modeling turns them into stable ratings on an Elo-like scale. The result tells you about general human preference, with all the biases that implies.

Use arena rankings to build your shortlist, then run your own mini-arena on your own prompts. A 50-point lead on someone else's leaderboard matters far less than a clear win on your own data.

Related Articles

• Human Evaluation for LLMs: Annotation Guidelines and Inter-Annotator Agreement
• LLM-as-a-Judge: How to Use One Model to Grade Another
• Why Public LLM Benchmarks Mislead: Contamination and Overfitting
• LLM Evaluation: Testing and Measuring Outputs the Right Way