Evaluating Structured Output: Measuring JSON Accuracy in LLMs

Contents

Green parser, wrong data — the dashboard nobody should trust alone

Your model returns valid JSON 100% of the time. Your parser never crashes. Your dashboard glows green.

Then someone opens a record and finds that the invoice total says $1,200 when the document says $12,000. The JSON was perfect. The data was wrong. That gap explains why measuring structured output takes more than checking whether the JSON parses. This guide shows you which metrics to track, how to score nested data fairly, and how to separate "valid" from "correct" — the field-level rigor of the dataset-building playbook applied to extraction tasks. Benchmarks cited here are named so you can check the evidence yourself.

Valid Is Not the Same as Correct

Structured output has two quality layers, and most teams measure only the first.

• Syntactic quality: does the output parse as JSON, and does it match the schema?
• Semantic quality: do the values inside the fields reflect the truth?

Constrained decoding, the technique behind most "structured output" features, mainly guarantees the first layer. It forces the model to emit tokens that fit a schema. Research on structured generation reports that constrained decoding consistently guarantees syntactic compliance but doesn't reliably improve semantic accuracy.

So you can hit 100% schema validity and still ship wrong data — that's the most important idea in this whole article.

The Metric Stack: From Cheap to Meaningful

Track these metrics in order. Each one catches a different kind of failure.

Level 1: Parse Rate

Parse rate measures the share of outputs that load as valid JSON. Failures here usually come from truncation (the model hit its token limit mid-object), stray text around the JSON, or unescaped characters.

Level 2: Schema Validity

Schema validity measures the share of parsed outputs that satisfy your JSON Schema: required fields present, types correct, enums respected, and no forbidden extra fields. Standard libraries like Python's jsonschema handle this check.

Researchers on JSONSchemaBench call a related idea the compliance rate. That benchmark also separates declared coverage (which schema features a framework claims to support) from empirical coverage (what actually works).

Level 3: Field-Level Accuracy

Field accuracy compares each predicted value against a gold value and reports the share that match. This metric finally tests meaning. For extraction tasks, report these per field type:

• Exact match: the predicted value equals the gold value after normalization.
• Precision: of the fields the model filled in, how many are correct?
• Recall: of the fields that should have values, how many did the model fill correctly?
• F1: the balance of precision and recall.

Level 4: Record-Level Accuracy

Record accuracy counts a record as correct only when every field matches. It runs far stricter than field accuracy: a model with 95% field accuracy across 10 fields gets roughly 60% of records fully right (0.95¹⁰ ≈ 0.60), so report both numbers.

Level 5: Task or Downstream Accuracy

The ultimate test asks whether the output works. Does the extracted order get created correctly in your system? Does the generated API call return the expected result? Downstream checks catch errors that field comparison misses.

A Minimal Scorer

This sketch covers parse rate, schema validity, and field accuracy. It flattens nested objects so you can compare field by field:

import json
from jsonschema import validate, ValidationError

def flatten(obj, prefix=""):
    out = {}
    if isinstance(obj, dict):
        for k, v in obj.items():
            out.update(flatten(v, f"{prefix}.{k}" if prefix else k))
    elif isinstance(obj, list):
        for i, v in enumerate(obj):
            out.update(flatten(v, f"{prefix}[{i}]"))
    else:
        out[prefix] = obj
    return out

def evaluate(pred_text, gold, schema):
    result = {"parsed": False, "valid": False, "field_acc": 0.0}
    try:
        pred = json.loads(pred_text)
        result["parsed"] = True
    except json.JSONDecodeError:
        return result
    try:
        validate(pred, schema)
        result["valid"] = True
    except ValidationError:
        pass
    gold_flat, pred_flat = flatten(gold), flatten(pred)
    correct = sum(pred_flat.get(k) == v for k, v in gold_flat.items())
    result["field_acc"] = correct / max(len(gold_flat), 1)
    return result

This version compares list items by position, which punishes harmless reordering. The next section shows how to fix that.

Score the Hard Parts Fairly

Real schemas include nested objects, lists, nulls, and messy strings. Each one needs a decision.

Normalize Before You Compare

Compare meaning, not formatting. Normalize values first:

• Strings: trim whitespace and lowercase where case doesn't matter.
• Numbers: parse "1,200.00" and 1200 to the same value, and allow small tolerances for floats.
• Dates: convert to one standard format before comparing.
• Units and currencies: convert to a canonical form.
• Booleans and enums: map "yes" and "true" to one representation.

import re
from datetime import datetime

def normalize(value):
    if isinstance(value, str):
        s = value.strip().lower()
        s = re.sub(r"^\$?", "", s).replace(",", "")   # "$1,200.00" -> "1200.00"
        try:
            return float(s)                           # "1200.00" -> 1200.0
        except ValueError:
            pass
        for fmt in ("%Y-%m-%d", "%m/%d/%Y"):          # "01/15/2026" -> "2026-01-15"
            try:
                return datetime.strptime(s, fmt).strftime("%Y-%m-%d")
            except ValueError:
                continue
        return s                                      # plain text, case-folded
    return value

Use Fuzzy Matching for Free Text

For fields like names or descriptions, exact match is too harsh. Use edit distance, token overlap, or an embedding or LLM check, and set a threshold. Report the exact and fuzzy scores separately.

Handle Lists Properly

Lists make scoring tricky. Decide whether order matters:

• Order doesn't matter: treat the list as a set, then match predicted items to gold items.
• Matching items: use an alignment step, such as the Hungarian algorithm, to pair each predicted item with its best gold partner before you score fields.
• Report list-level precision and recall: count extra items as false positives and missing items as false negatives.

Treat Null, Missing, and Empty Differently

These three states carry different meanings. A missing required field breaks the schema. A null may mean "not found." An empty string may mean the model guessed nothing. Score each case explicitly, and watch for hallucinated values where the correct answer is null — the failure mode hallucination detection is built to catch.

Penalize Extra Fields

A model that invents fields outside the schema adds noise. If your schema forbids additional properties, count extra keys as failures, and track how often they appear.

Build the Right Test Set

Your metrics only help when the data matches reality.

1. Collect real inputs, not only clean examples. Include typos, odd formatting, and missing information.
2. Create gold outputs with careful human labeling, and check agreement between labelers.
3. Cover the schema. Include cases that trigger every field, enum value, and optional branch.
4. Add negative cases. Include inputs where the right answer is null or "not present."
5. Slice by complexity. Score simple and deeply nested schemas separately.

Aim for at least 100 examples, and more when the schema is large — gold labeling is where custom evaluation datasets earn their keep.

Evaluate Constrained Decoding and Structured Output Features

Most providers and open-source tools now offer schema-constrained generation. Test your specific setup instead of assuming the guarantee covers your case.

The JSONSchemaBench study, which evaluated six frameworks across 10,000 real-world schemas, offers a useful lesson. Coverage differs a lot between tools, and complex real-world schemas expose gaps that simple examples hide. A secondary summary of the study notes that no framework covered every feature category in the official JSON Schema Test Suite, so check whether your schema uses features your tool supports.

Compare these three setups on your own data:

• Prompt-only: describe the format in the prompt, and parse the result.
• Native structured output: use your provider's schema-constrained mode.
• Constrained decoding library: use a tool like Guidance, Outlines, or XGrammar with an open model.

Measure parse rate, schema validity, field accuracy, latency, and cost for each. Constrained modes often win on validity and sometimes lose a little on content quality, so let the numbers decide — the same build-vs-buy discipline as the evaluation pipeline overview.

Figure 1: Under every green parse-rate dashboard sits a pipeline that can still be wrong at the field level

Diagnose Failures With an Error Taxonomy

A single accuracy number hides what's wrong. Tag each failure with a type:

• Truncation: the output ended early because of a token limit.
• Syntax error: broken JSON, such as trailing commas or unescaped quotes.
• Schema violation: wrong type, missing required field, or invalid enum.
• Hallucinated field: the model invented a key or a value.
• Wrong value: the field exists and holds the wrong data.
• Misplaced value: the right data landed in the wrong field.
• Over- or under-extraction: the model added or skipped list items.
• Refusal: the model declined and returned prose.

Count each type. Truncation points to a token budget fix. Misplaced values point to unclear field descriptions. Wrong values point to model or prompt quality.

Check Consistency Across Runs

Run each input several times. A model that gives different field values on different runs signals uncertainty, even when each output validates. Report the agreement rate across runs, and flag fields that change often. Those fields usually deserve human review in production — the same single-run trust problem judge evaluations face.

Improve Accuracy Once You Measure It

Once metrics reveal the weak spots, try these fixes:

• Write clear field descriptions in the schema. Models read them.
• Simplify the schema. Flatten deep nesting where you can.
• Add examples of tricky fields to the prompt.
• Allow a null or "unknown" option so the model doesn't guess.
• Add a validation and repair loop: if validation fails, send the error back and retry.
• Use a stronger model for the hardest fields.

Rerun your full evaluation after every change.

Common Pitfalls

• Reporting only schema validity — valid data can still be wrong.
• Skipping normalization — formatting noise tanks your scores.
• Scoring lists by position — reordering looks like failure.
• Ignoring nulls — models love to invent values for missing information.
• Testing on clean inputs only — production data arrives messy.
• Trusting one run — consistency matters.
• Never reporting record-level accuracy — 95% fields can mean 60% records.
• Skipping the error taxonomy — one blended number can't tell you what to fix.

Unlock AI That Actually Works

Get lifetime access to GPT-6 Astra, Claude Fable 5.1, Gemini 3.5, Grok 4.5, and more — all in one platform. Build websites, apps, videos, content, and digital products from a single command. No monthly fees. No tool-hopping.

Click here to get GPTAstra Max now — one-time payment, lifetime access.

Frequently Asked Questions

Why isn't valid JSON the same as correct JSON?

Structured output has two quality layers, and most teams measure only the first. Syntactic quality asks whether the output parses as JSON and matches the schema. Semantic quality asks whether the values inside the fields reflect the truth. Constrained decoding, the technique behind most structured-output features, mainly guarantees the first layer: research on structured generation reports that it consistently guarantees syntactic compliance but doesn't reliably improve semantic accuracy. So you can hit 100% schema validity and still ship wrong data — the invoice total reads $1,200 when the document says $12,000. That's why measurement needs a stack: parse rate (does it load), schema validity (does it satisfy your JSON Schema via standard libraries like Python's jsonschema), field-level accuracy (does each value match gold, with precision, recall, and F1 per field type), record-level accuracy (every field in the record correct — at 95% field accuracy across 10 fields, only about 60% of records are fully right), and finally downstream task accuracy (does the extracted order actually get created correctly in your system).

How do you score nested structures, lists, and nulls fairly?

Normalize before comparing: trim strings and lowercase where case doesn't matter, parse "1,200.00" and 1200 to the same value with small float tolerances, convert dates to one standard format, canonicalize units and currencies, and map "yes"/"true" to one representation — formatting noise otherwise tanks real accuracy. For free text like names or descriptions, exact match is too harsh: use edit distance, token overlap, or an embedding or LLM check with a threshold, and report exact and fuzzy scores separately. Lists need an explicit order decision: treat the list as set-like and match items when order doesn't matter, use an alignment step like the Hungarian algorithm to pair each predicted item with its best gold partner, and report list-level precision and recall (extra items are false positives, missing items are false negatives). Treat null, missing, and empty differently — a missing required field breaks the schema, null may mean "not found," an empty string may mean the model guessed nothing — and score hallucinated values where the correct answer is null as explicit failures. Penalize extra fields too: if your schema forbids additional properties, count invented keys as failures and track how often they appear.

What is an error taxonomy, and why tag failures?

A single accuracy number hides what's wrong, so tag each failed output with a type: truncation (the output ended early because of a token limit), syntax error (broken JSON such as trailing commas or unescaped quotes), schema violation (wrong type, missing required field, or invalid enum), hallucinated field (the model invented a key or a value), wrong value (the field exists but holds the wrong data), misplaced value (the right data landed in the wrong field), over- or under-extraction (the model added or skipped list items), and refusal (the model declined and returned prose). Each type points to a different fix: truncation to a token budget fix, misplaced values to unclear field descriptions, wrong values to model or prompt quality. Also check consistency across runs — run each input several times and report the agreement rate. A model that gives different field values on different runs signals uncertainty even when each output validates, and those flapping fields usually deserve human review in production. Fixes that follow from the metrics: write clear field descriptions in the schema (models read them), flatten deep nesting, add examples of tricky fields to the prompt, allow a null or "unknown" option so the model doesn't guess, and add a validation-and-repair loop that sends the schema error back and retries.

How do you test constrained decoding and structured output features?

Test your specific setup instead of assuming the guarantee covers your case. The JSONSchemaBench study, which evaluated six frameworks across 10,000 real-world schemas, found that coverage differs a lot between tools and that complex real-world schemas expose gaps that simple examples hide; a secondary summary notes that no framework covered every feature category in the official JSON Schema Test Suite, so check whether your schema uses features your tool supports. Compare three setups on your own data: prompt-only (describe the format in the prompt and parse the result), native structured output (your provider's schema-constrained mode), and a constrained decoding library such as Guidance, Outlines, or XGrammar with an open model. Measure parse rate, schema validity, field accuracy, latency, and cost for each — constrained modes often win on validity and sometimes lose a little on content quality, so let the numbers decide. Build the test set from real inputs, not only clean examples: include typos, odd formatting, and missing information; label gold outputs with humans and check labeler agreement; cover every field, enum value, and optional branch; add negative cases where the right answer is null; and slice by schema complexity, with at least 100 examples.

Wrapping This Up

Structured output gives you data your code can read. It doesn't guarantee that data tells the truth. Measure validity first, then measure accuracy field by field, normalize your comparisons, and score lists and nulls with care.

Start with 100 messy examples and the scorer above. You'll quickly see how much of your "100% valid" output is actually correct.

Related Articles

• How to Build a Custom Evaluation Dataset for Your LLM App
• LLM Evaluation: Testing and Measuring Outputs the Right Way
• LLM Hallucination Detection: Methods and Metrics That Work
• Bias in LLM-as-a-Judge: Position, Verbosity, and Self-Preference