LLM Hallucination Detection: Methods and Metrics That Work

Contents

Confident, polite, perfectly formatted — and completely made up

Your chatbot just told a customer that your company offers a "90-day price-match guarantee." You offer no such thing. The answer sounded confident, polite, and perfectly formatted, and it was completely fabricated.

That's the nasty part about hallucinations: they don't look like errors. They look like answers. This guide covers the detection methods that hold up in production, the metrics that tell you whether your detector works, and how to combine both into a system you can trust — the detection layer of the evaluation guide. Many published numbers come from vendors who sell detection tools, so this focuses on methods you can build and verify yourself.

Two Kinds of Hallucination (and Why It Matters)

Detection starts with a clear definition. Most practitioners split hallucinations into two types:

• Intrinsic (faithfulness) hallucinations: the answer contradicts or ignores the source material you gave the model, like a retrieved document.
• Extrinsic (factuality) hallucinations: the answer states something that conflicts with reality or that nobody can verify from any source.

The distinction changes your approach. In a RAG app, you can check answers against retrieved documents, which makes faithfulness checks practical. For open-ended questions, no document exists to check against, so you need other signals.

A model can score well on faithfulness and still produce extrinsic errors, so measure both when the stakes are high.

The Detection Toolbox

No single method catches everything. Most production systems layer several, ordered from cheap to expensive.

Layer 1: Deterministic Checks

Start with rules, because they cost almost nothing and never flake:

• Schema validation: does the output match the required structure?
• Citation checks: do cited document IDs, URLs, or quotes actually exist in your sources?
• Entity and number checks: do names, dates, prices, and product codes in the answer appear in the source material?
• Banned-claim patterns: flag phrases like "guarantee," "always," or specific legal language that your policy forbids.

The citation and number checks catch an impressive share of fabrications for almost no effort.

Layer 2: Grounding and Faithfulness Checks

For RAG apps, compare the answer against the retrieved context. Two common approaches work well:

• Natural language inference (NLI): a small model judges whether each claim follows from the source text.
• LLM-as-judge: a second model reads the context and the answer, then flags unsupported statements.

The most reliable pattern works at the claim level: break the answer into individual claims, then verify each one separately. A whole-answer score hides the one bad sentence buried in four good ones.

def faithfulness(answer, context, judge):
    claims = judge.extract_claims(answer)
    supported = sum(judge.is_supported(c, context) for c in claims)
    return supported / max(len(claims), 1)

score = faithfulness(answer, context, judge)
if score < 0.9:                       # one unsupported claim in ten is still one
    flag_for_review(answer, score)

One vendor roundup reports that LLM judges correlate about 81% with human raters on faithfulness. Treat that number as a rough hint — measure your own judge against your own labels (calibration routine here).

Layer 3: Consistency Checks

Ask the model the same question several times, then compare the answers. When a model knows a fact, its answers agree. When it invents one, the answers drift.

SelfCheckGPT popularized this: sample multiple responses, then check whether the claims in the main answer appear consistently across the samples. It needs only black-box access, so it works with any API.

• Strength: no ground truth or model internals required.
• Weakness: it multiplies cost and latency, and models can also repeat the same wrong answer consistently.

Layer 4: Semantic Entropy

Semantic entropy extends the consistency idea: instead of comparing exact wording, it groups sampled answers by meaning and measures how spread out those groups are. High entropy means the model keeps changing its story — a likely confabulation.

Research published in Nature backs this for catching answers that vary arbitrarily. You need multiple generations plus a way to cluster meanings, so it fits best as a targeted check on high-risk queries.

Layer 5: Confidence Signals

If you control the model or your provider exposes token probabilities, log probabilities are a cheap uncertainty signal. Low confidence on key tokens often marks shaky claims.

Caveats: models can sound confident and be wrong, and many hosted APIs limit access to probabilities. Use it as one signal among several.

Layer 6: Human Review

Humans stay the gold standard. Use them to label a sample, audit your automated detectors, and review flagged cases. You can't skip this layer — it supplies the ground truth that tells you whether everything else works.

Figure 1: Ground truth is a human reading the answer next to the source — everything else is an approximation of this

The Metrics That Matter

Two groups of metrics: ones that measure your app's quality, and ones that measure your detector's quality. Teams often confuse them.

Metrics for Your App

• Hallucination rate: the share of answers containing at least one unsupported claim.
• Claim-level precision: supported claims divided by total claims.
• Faithfulness or groundedness score: how well the answer sticks to the provided context.
• Citation accuracy: the share of citations that actually support the attached claim.
• Abstention rate: how often the model says "I don't know." A healthy abstention rate beats a confident fabrication.

For RAG pipelines, add context precision and recall — a poor retriever causes many "hallucinations" that the model never invented. It simply never saw the right document.

Metrics for Your Detector

Your detector makes predictions too, and those predictions can fail. Measure them against human labels:

• Precision: of the answers flagged as hallucinations, how many truly were?
• Recall: of the true hallucinations, how many did the detector catch?
• F1 score: one number that balances precision and recall.
• AUROC: how well the detector's scores separate good answers from bad across all thresholds.

def detector_metrics(flags, truth):
    """flags/truth: booleans per answer — detector verdict vs human label."""
    tp = sum(f and t for f, t in zip(flags, truth))
    fp = sum(f and not t for f, t in zip(flags, truth))
    fn = sum(not f and t for f, t in zip(flags, truth))
    precision = tp / max(tp + fp, 1)
    recall = tp / max(tp + fn, 1)
    f1 = 2 * precision * recall / max(precision + recall, 1e-9)
    return round(precision, 2), round(recall, 2), round(f1, 2)

print(detector_metrics([1, 1, 1, 0, 0], [1, 1, 0, 0, 0]))  # (0.67, 1.0, 0.8) — strict
print(detector_metrics([1, 0, 0, 0, 0], [1, 1, 0, 0, 0]))  # (1.0, 0.5, 0.67) — loose

Precision and recall trade off against each other. A strict threshold blocks more fabrications and also blocks good answers. Pick the balance based on risk: a medical or legal app should favor recall, while a casual assistant can favor precision.

How to Evaluate Your Detector

Don't trust a detector you haven't tested:

1. Collect 200 or more real outputs from your app, including some failures.
2. Have humans label each answer as supported or unsupported, with notes on which claims failed.
3. Run your detector on the same set.
4. Compute precision, recall, and F1 at several thresholds.
5. Review the misses. Missed hallucinations and false alarms both teach you something.
6. Repeat quarterly, since your data and models change.

Build a Layered Production System

Each method has a different cost, so route traffic smartly:

1. Run deterministic checks on every response. They're nearly free.
2. Run faithfulness scoring on every RAG response, or on a sample if volume runs high.
3. Use consistency or entropy checks on high-risk queries — pricing, medical, or legal topics.
4. Block or soften low-scoring answers at runtime, with a fallback such as "I couldn't verify that, so here's a link to the source."
5. Send borderline cases to human review, and feed the results back into your tests — the feedback loop that keeps the system honest.

Two-stage detection helps RAG apps most: check retrieval relevance first, because a bad retrieval guarantees a bad answer. Then check faithfulness against what the model actually saw.

Reduce Hallucinations While You Detect Them

Detection catches problems after the fact. These habits prevent many of them:

• Ground answers in retrieved sources, and require citations.
• Tell the model it may abstain when the context lacks an answer.
• Lower the temperature for factual tasks.
• Keep your knowledge base fresh, since stale documents produce stale answers.
• Constrain outputs with schemas and allowed value lists.

Common Pitfalls

• Trusting one detector — every method misses something; layer them.
• Never testing the detector — a judge with no human calibration is just another opinion.
• Scoring whole answers only — one fabricated claim hides inside a mostly correct paragraph.
• Ignoring retrieval failures — the retriever, not the model, causes many errors.
• Blocking too aggressively — users hate a bot that refuses to answer; tune toward your risk profile instead of maximum strictness.
• Believing vendor accuracy claims — test on your own data, as the guardrails guide advises for safety tooling generally.
• Forgetting abstention — a well-calibrated "I don't know" is a feature, not a failure.

Unlock AI That Actually Works

Get lifetime access to GPT-6 Astra, Claude Fable 5.1, Gemini 3.5, Grok 4.5, and more — all in one platform. Build websites, apps, videos, content, and digital products from a single command. No monthly fees. No tool-hopping.

Click here to get GPTAstra Max now — one-time payment, lifetime access.

Frequently Asked Questions

What are the main methods for detecting LLM hallucinations?

Production systems layer six, ordered cheap to expensive. Deterministic checks (schema validation, citation existence, entity and number checks against sources, banned-claim patterns) cost almost nothing and catch an impressive share of fabrications. Grounding and faithfulness checks compare the answer to retrieved context, ideally at claim level — break the answer into individual claims and verify each, because a whole-answer score hides one bad sentence inside four good ones — using NLI models or LLM-as-judge (one vendor roundup reports judges correlate about 81% with human raters on faithfulness; treat that as a hint and measure your own). Consistency checks (SelfCheckGPT) sample multiple responses and check whether claims recur — black-box friendly, but they multiply cost and a model can repeat the same wrong answer consistently. Semantic entropy groups sampled answers by meaning and measures spread; Nature-published research backs it for detecting arbitrarily varying answers, and it fits best as a targeted check on high-risk queries. Confidence signals from token log-probabilities are a cheap uncertainty hint where APIs expose them. Humans remain the gold standard: label samples, audit detectors, and review flagged cases.

What is the difference between intrinsic and extrinsic hallucinations?

Intrinsic (faithfulness) hallucinations contradict or ignore the source material given to the model, like a retrieved document. Extrinsic (factuality) hallucinations state something that conflicts with reality or that nobody can verify from any source. The distinction changes your approach: in a RAG app you can check answers against retrieved documents, making faithfulness checks practical, while open-ended questions have no document to check against and need other signals — consistency, entropy, or human review. A model can score well on faithfulness and still produce extrinsic errors, so measure both when stakes are high. Also watch retrieval: a poor retriever causes many "hallucinations" the model never invented — it simply never saw the right document. Two-stage detection helps most: check retrieval relevance first, then faithfulness against what the model actually saw.

How do you measure whether your hallucination detector works?

Your detector makes predictions too, so measure them against human labels: precision (of flagged answers, how many truly were hallucinations), recall (of true hallucinations, how many were caught), F1 (the balance), and AUROC (how well scores separate good from bad across thresholds). Precision and recall trade off — a strict threshold blocks more fabrications and also blocks good answers — so pick the balance by risk: medical or legal apps favor recall, casual assistants favor precision. The evaluation routine: collect 200 or more real outputs including failures; have humans label each answer supported or unsupported with notes on which claims failed; run the detector on the same set; compute precision, recall, and F1 at several thresholds; review the misses — both missed hallucinations and false alarms teach you; repeat quarterly since data and models change. For the app itself, track hallucination rate (share of answers with at least one unsupported claim), claim-level precision, groundedness, citation accuracy, abstention rate, and for RAG, context precision and recall.

How do you build a layered hallucination-prevention system?

Route traffic by cost: run deterministic checks on every response (nearly free); run faithfulness scoring on every RAG response or a sample if volume is high; use consistency or entropy checks only on high-risk queries like pricing, medical, or legal topics; block or soften low-scoring answers at runtime with a fallback such as "I couldn't verify that, so here is a link to the source"; send borderline cases to human review and feed results back into your tests. Then reduce hallucinations while you detect them: ground answers in retrieved sources and require citations, allow the model to abstain when context lacks an answer, lower temperature on factual tasks, keep the knowledge base fresh (stale documents produce stale answers), and constrain outputs with schemas and allowed value lists. Common mistakes to avoid: trusting one detector, never calibrating the judge against human labels, scoring whole answers only, ignoring retrieval failures, blocking so aggressively users get refusals, and believing vendor accuracy claims without testing on your own data.

Wrapping This Up

Hallucinations don't announce themselves, so you need layered detection that checks the facts. Start with cheap deterministic rules, add claim-level grounding checks for RAG, and reserve expensive consistency methods for high-stakes questions. Then validate every detector against human labels.

Start with 200 labeled answers and one citation check this week. You'll find out quickly how often your app invents things — and that knowledge beats any confident-sounding answer.

Related Articles

• LLM-as-a-Judge: How to Use One Model to Grade Another
• LLM Evaluation: Complete Guide to Testing and Measuring LLM Outputs
• Feedback Loops for LLM Apps: Turn User Signals into Better Outputs
• AI Agent Guardrails and Safety: Prevent Runaway Actions