Evaluating LLM Summarization: ROUGE, BERTScore, and Human Review

Contents

Word overlap rewards summaries that lie beautifully

Take a good summary, swap one fact for a wrong one, and run it through ROUGE. The score barely moves. That's the core problem with evaluating summaries.

The easy metrics reward word overlap, and a summary can overlap beautifully with a reference while lying about what the document says. This guide explains what ROUGE and BERTScore measure, where each one breaks, and how to combine them with LLM judges and human review into an evaluation you can trust — the summarization chapter of the evaluation guide, and the faithfulness half pairs with the hallucination detection guide.

What Makes a Good Summary?

Before you pick a metric, define quality. Researchers commonly judge summaries on four dimensions, a framing popularized by the SummEval study:

• Relevance: the summary captures the important points and skips the trivia.
• Consistency (faithfulness): every claim matches the source. No invented facts.
• Coherence: the sentences flow into a logical whole.
• Fluency: the grammar and wording read cleanly.

Production apps usually add two practical dimensions: coverage (did it include everything required?) and concision (did it stay within the length budget?).

Consistency matters most for production apps. A fluent, relevant summary with one fabricated number does real damage.

ROUGE: The Old Workhorse

ROUGE compares a generated summary against one or more human-written reference summaries by counting overlapping text units. It dominated summarization research for years, so you'll see it everywhere.

The Common Variants

• ROUGE-1: overlap of single words (unigrams).
• ROUGE-2: overlap of two-word sequences (bigrams), which rewards matching phrases.
• ROUGE-L: overlap based on the longest common subsequence, which captures some word-order information.

Each variant reports precision, recall, and F1 — how much of the summary appears in the reference, how much of the reference appears in the summary, and the balance of both.

What ROUGE Does Well

• It's fast and free. You can score thousands of summaries in seconds.
• It's reproducible. Everyone computes it the same way, mostly.
• It catches gross failures. A summary with zero overlap probably misses the point.

Where ROUGE Fails

• It ignores meaning. Paraphrases score poorly even when they say the same thing.
• It ignores facts. A wrong number or a flipped negation barely changes the score.
• It depends on the reference. One reference summary represents only one valid way to summarize.
• It favors extractive text. Summaries that copy source sentences often beat better abstractive ones.
• It reacts to length. Longer summaries earn higher recall, so compare length-matched outputs.

Modern LLMs write fluent, abstractive summaries that paraphrase heavily — exactly the style ROUGE punishes — so it tends to correlate weakly with human judgment on LLM output. The metrics deep-dive covers this in the BLEU/ROUGE explainer.

BERTScore: Matching Meaning, Not Words

BERTScore fixes part of the paraphrase problem. Instead of counting exact word matches, it converts each token into a contextual embedding from a pretrained language model, then matches tokens between the summary and the reference by embedding similarity.

It reports precision, recall, and F1, just like ROUGE, but a synonym earns credit instead of a penalty.

What BERTScore Does Well

• It rewards paraphrases. "Car" and "automobile" count as close.
• It correlates better with humans than ROUGE in many studies, especially on fluency and relevance.
• It handles word-order variation more gracefully.

Where BERTScore Falls Short

• It still misses factual errors. A wrong date that resembles the right one in embedding space can slip through.
• It still needs a reference. One reference still limits you.
• Scores cluster tightly. Raw values often land in a narrow band, so use baseline rescaling and compare relative differences.
• Model choice changes results. Always report which embedding model and version you used.

BERTScore improves on ROUGE for semantic similarity but still can't verify that a summary tells the truth.

Computing Both in a Few Lines

Popular open-source libraries make this easy:

from rouge_score import rouge_scorer
from bert_score import score

reference = "The council approved the budget after a two-hour debate."
summary = "After debating for two hours, the council passed the budget."

scorer = rouge_scorer.RougeScorer(["rouge1", "rouge2", "rougeL"], use_stemmer=True)
rouge = scorer.score(reference, summary)

P, R, F1 = score([summary], [reference], lang="en")
print(rouge["rougeL"].fmeasure, F1.mean().item())

Run this on a full test set, and report averages with confidence intervals, not single numbers — the sizing math from the dataset guide applies to summaries too.

Check Faithfulness Directly

Overlap metrics can't catch hallucinated facts, so add a metric built for faithfulness. Three approaches dominate:

• NLI-based checks: a natural language inference model tests whether each summary sentence follows from the source document. SummaC takes this approach.
• QA-based checks: generate questions from the summary, answer them from both the summary and the source, and compare the answers. QAGS popularized this idea.
• Claim-level LLM verification: an LLM extracts individual claims from the summary and checks each one against the source.

def claim_level_faithfulness(summary, source, nli):
    """Verify each claim separately — a whole-score hides the one that lied."""
    claims = split_into_claims(summary)
    verdicts = [nli(source, c) for c in claims]   # entailment | contradiction | neutral
    supported = sum(v == "entailment" for v in verdicts)
    return supported / max(len(verdicts), 1), verdicts

score, verdicts = claim_level_faithfulness(summary, document, nli_model)

The claim-level approach gives the most useful diagnostics, because it tells you which sentence failed. It also costs more, so many teams run it on a sample — and validate the checker against human labels first, since any automated faithfulness score is a detector with its own precision and recall.

Figure 1: The source document is the only truth the summary gets measured against — every metric is a proxy for reading it

LLM-as-Judge for Summaries

LLM judges now score summaries on all four dimensions, without a reference summary. G-Eval helped popularize the approach: the judge model receives a rubric, reasons through the criteria, and then returns a score.

Practices that get reliable results:

• Write a specific rubric for each dimension, with defined score anchors.
• Score one dimension at a time. Combined scores blur the signal.
• Prefer pairwise comparison when you compare two systems, and swap the order to cancel position bias.
• Ask for evidence. Require the judge to quote the source text that supports or contradicts each claim.
• Validate against humans. Compare judge scores with human ratings on a labeled sample before you trust it.

Judges carry biases — they tend to favor longer outputs and text that resembles their own style (full bias catalog and calibration routine).

Human Review: The Ground Truth

Humans still anchor every serious summarization evaluation. Use them to calibrate your automated metrics and to judge the cases that matter most.

Design the Review Carefully

1. Define the dimensions you'll rate: faithfulness, relevance, coherence, fluency, and coverage.
2. Write a rubric with anchors, such as "1 = contains a clear factual error, 5 = every claim matches the source."
3. Train your reviewers with five or ten examples and a discussion.
4. Blind the review. Hide which system produced each summary.
5. Use at least two reviewers per item, and three for hard cases.
6. Measure agreement with Cohen's kappa for two raters or Krippendorff's alpha for more.
7. Resolve disagreements through discussion, and refine the rubric.

Low agreement means your rubric is vague, not that your reviewers are bad. Fix the rubric first.

Annotate Error Types, Not Only Scores

A 3 out of 5 doesn't tell you what to fix. Ask reviewers to tag the specific problems they find:

• Hallucination: the summary states something absent from the source.
• Distortion: the summary changes a fact, number, or relationship.
• Omission: the summary leaves out a required point.
• Redundancy: the summary repeats itself.
• Wrong emphasis: the summary highlights minor details over key ones.

Error tags turn evaluation into a to-do list for your prompts and models.

Pick the Right Reviewers

General crowd workers can judge fluency and coherence. Faithfulness in medical, legal, or technical text needs domain experts, because a non-expert can't spot subtle distortions. Budget accordingly.

Put It All Together

Combine the methods into layers, from cheap to expensive:

1. ROUGE and BERTScore for fast regression checks on every change.
2. Faithfulness metrics (NLI or claim-level) on every release candidate.
3. LLM judges for dimension-level scores on larger samples.
4. Human review on a stratified sample, plus every high-risk category.

What each layer catches:

• Overlap metrics catch gross quality drops and broken formatting.
• Embedding metrics catch semantic drift.
• Faithfulness metrics catch fabricated or distorted claims.
• LLM judges catch coherence and relevance problems at scale.
• Humans catch everything subtle, and they calibrate all the rest.

Report Results Honestly

Good reporting builds trust. Include these items in every evaluation:

• Dataset size and source, plus how you sampled it.
• Metric versions and settings — the embedding model, whether you rescaled.
• Summary lengths, so readers can spot length effects.
• Confidence intervals, since small samples produce noisy averages.
• Human agreement scores for any human-rated dimension.
• Failure examples, not only averages.

Common Pitfalls

• Treating ROUGE as quality — it measures overlap, nothing more.
• Using one reference summary — collect several when you can.
• Ignoring faithfulness — fluent fabrication scores well on overlap metrics.
• Trusting an uncalibrated judge — compare it with human labels first.
• Skipping length control — longer summaries win on recall, so match lengths.
• Reporting a single number — dimension-level scores reveal the real picture.
• Comparing BERTScore across embedding models — the model and version change the scale.
• Never tagging error types — scores tell you that something failed, tags tell you what to fix.

Unlock AI That Actually Works

Get lifetime access to GPT-6 Astra, Claude Fable 5.1, Gemini 3.5, Grok 4.5, and more — all in one platform. Build websites, apps, videos, content, and digital products from a single command. No monthly fees. No tool-hopping.

Click here to get GPTAstra Max now — one-time payment, lifetime access.

Frequently Asked Questions

What dimensions define a good summary?

Researchers commonly judge summaries on four dimensions, a framing popularized by the SummEval study: relevance (captures the important points, skips the trivia), consistency or faithfulness (every claim matches the source, no invented facts), coherence (sentences flow into a logical whole), and fluency (grammar and wording read cleanly). Production apps usually add two practical dimensions: coverage (did it include everything required?) and concision (did it stay within the length budget?). Consistency matters most for production: a fluent, relevant summary with one fabricated number does real damage. Define your dimensions and score anchors before choosing metrics, because each metric measures a different subset — overlap metrics track relevance-ish signals, embedding metrics track semantic similarity, faithfulness checks track consistency, and humans anchor all four.

What do ROUGE and BERTScore actually measure?

ROUGE compares a generated summary against human reference summaries by counting overlapping text units: ROUGE-1 overlaps single words, ROUGE-2 overlaps two-word phrases, ROUGE-L uses the longest common subsequence; each reports precision, recall, and F1. It's fast, reproducible, and catches gross failures — but it ignores meaning (paraphrases score poorly), ignores facts (a wrong number or flipped negation barely moves the score), depends on the single reference, favors extractive copying, and rewards length on recall. Modern LLMs write fluent abstractive summaries that paraphrase heavily — exactly the style ROUGE punishes — so it correlates weakly with human judgment on LLM output. BERTScore converts tokens into contextual embeddings and matches by similarity, so synonyms earn credit and it correlates better with humans on fluency and relevance; but it still misses factual errors (a wrong date resembling the right one slips through), still needs a reference, clusters tightly in a narrow band (use baseline rescaling and compare relative differences), and changes with the embedding model — always report which model and version. Neither metric verifies that a summary tells the truth.

How do you check summarization faithfulness directly?

Three approaches dominate. NLI-based checks run a natural language inference model on each summary sentence against the source document (SummaC takes this approach). QA-based checks generate questions from the summary, answer them from both the summary and the source, and compare (QAGS popularized this). Claim-level LLM verification extracts individual claims and checks each against the source. The claim-level approach gives the most useful diagnostics because it tells you which sentence failed, and it also composes with hallucination-detection practice: flag answers below a supported-claim threshold for review or fallback. It costs more, so many teams run it on a sample — and they validate the checker itself against human labels first, since any automated faithfulness score is a detector with its own precision and recall.

How do you run reliable human review for summaries?

Design it carefully: define the dimensions (faithfulness, relevance, coherence, fluency, coverage); write a rubric with anchors such as "1 = contains a clear factual error, 5 = every claim matches the source"; train reviewers with five or ten examples and a discussion; blind the review so no one knows which system produced each summary; use at least two reviewers per item and three for hard cases; measure agreement with Cohen's kappa for two raters or Krippendorff's alpha for more; and resolve disagreements through discussion, refining the rubric. Low agreement means the rubric is vague, not that reviewers are bad. Beyond scores, have reviewers tag error types — hallucination (absent from source), distortion (changed fact), omission (missing required point), redundancy, wrong emphasis — turning evaluation into a to-do list. Faithfulness in medical, legal, or technical text needs domain experts. Then layer it all: ROUGE and BERTScore for fast regression checks, faithfulness metrics on release candidates, LLM judges for dimension scores at scale, humans on a stratified sample plus every high-risk category.

Wrapping This Up

ROUGE and BERTScore give you fast, cheap signals, and neither one checks whether a summary tells the truth. Add faithfulness checks, calibrate an LLM judge, and keep humans in the loop for the cases where accuracy matters.

Start with 100 documents, one automated faithfulness check, and a short human review. You'll quickly see what your overlap scores hide.

Related Articles

• LLM Evaluation Metrics Explained: Perplexity, BLEU, ROUGE, and Beyond
• LLM Hallucination Detection: Methods and Metrics That Work
• LLM-as-a-Judge: How to Use One Model to Grade Another
• How to Build a Custom Evaluation Dataset for Your LLM App