Evaluating AI Agents: Metrics, Benchmarks, and Test Strategies

Contents

Figure 1: One lucky run proves nothing — measure outcomes, process, reliability, cost, and safety

"It worked when I tried it" is the most dangerous sentence in agent development. Agents are non-deterministic, they take many steps, and they change the world through tools, so a single successful run tells you almost nothing. Evaluating them well means measuring more than the final answer: whether the world ended up in the right state, whether the path was safe and efficient, whether the result is repeatable, and what it cost. This guide covers the metrics worth tracking, how to build a test suite, how to grade fairly, how to use public benchmarks without being fooled by them, and a working evaluation harness. If you're building the agents you'll be measuring, start with the build-an-agent-from-scratch guide in this series.

What to Measure

No single number captures agent quality. Track several layers:

• Outcome. Did the task actually get done? The most trustworthy outcome checks inspect state instead of reading the agent's words. For example, tau-bench checks the database state, SWE-Bench runs the test suite, and tau2-bench's pass^k measures whether the agent succeeds reliably across attempts rather than once. A benchmark that only checks tool-call syntax or final text would pass agents that never fixed anything.
• Process (the trajectory). How did it get there? Score tool selection, argument correctness, step efficiency, policy adherence, and forbidden or unnecessary actions. Process matters because an agent can reach the right outcome in a wrong way, such as issuing a refund it wasn't authorized to give. Recent research on "corrupt success" makes this point directly, arguing for procedure-aware evaluation beyond task completion.
• Reliability. Does it succeed consistently? This is where pass@k and pass^k come in, covered below.
• Cost and latency. Tokens, tool calls, wall-clock time, and dollars per task. An agent that succeeds at ten times the cost of the alternative is a different product.
• Safety. Does it avoid unauthorized actions, resist prompt injection, and protect data? These need their own adversarial test cases.
• Honesty. Does it report outcomes truthfully? Research on false success in LLM agents documents confident closing statements for tasks that actually failed, so check claims against state, and measure whether the agent admits when it can't complete something.
• Component quality. Tool functions, retrieval, planners, and parsers can each be tested in isolation, which makes failures easier to localize.

pass@k vs pass^k

These two metrics answer different questions, and confusing them is a common source of inflated confidence:

• pass@k: the chance that at least one of k attempts succeeds. It measures capability — whether the agent can do the task if given retries.
• pass^k: the chance that all k attempts succeed. It measures consistency.

An agent that solves a task 8 times out of 10 fails badly under pass^k. One explainer states the production implication bluntly: real users do not get to retry until it works. For customer-facing or high-stakes agents, pass^k is usually the metric that matters. With n trials and c successes, unbiased estimators are:

pass@k = 1 − C(n−c, k) / C(n, k)
pass^k = C(c, k) / C(n, k)

Build the Test Set

A good suite is built from real failure modes, not imagination.

• Sources: production failures and escalations, support tickets, user feedback, known edge cases, adversarial inputs, and policy rules you must never violate. Convert every incident into a regression case.
• Anatomy of a case: an initial environment state, a task (often a user message or a simulated user), a success check against final state, constraints on the trajectory such as forbidden actions and step limits, and tags for slicing results.
• Size: start with 30 to 100 well-chosen cases and grow. Small suites produce wide error bars, so don't over-read differences.
• Environments: make them hermetic. Use mocked or sandboxed tools with resettable state, frozen external data, and fixed seeds, since a live API that changes under you makes failures impossible to interpret. For conversational agents, an LLM-simulated user, as in tau-bench, lets you test multi-turn behavior, though you should validate the simulator itself.
• Splits: keep a held-out set that never appears in prompts, few-shot examples, or fine-tuning data, or you'll measure memorization.

Grading

Prefer graders in this order:

1. Code-based state checks. Deterministic and cheap: did the refund record exist, do the tests pass, does the file contain the right content?
2. Rule-based trajectory checks. Forbidden tools never called, required confirmation obtained, step budget respected.
3. LLM-as-judge, for open-ended quality such as writing, summaries, and reasoning explanations.
4. Human review, to calibrate everything above and to review a sample continuously.

LLM judges need care before you trust them. They show position bias, verbosity bias, and self-preference, and they can be inconsistent run to run. Research surveying judge design has systematized these failure modes and mitigations. Practical defenses include rubrics with explicit criteria, pairwise comparisons with randomized order, using a different model family than the agent under test, temperature zero, and, most importantly, measuring the judge's agreement with human labels on a sample before trusting it. Newer "agent-as-a-judge" approaches use tool-using agents to inspect traces and gather evidence, shifting the judge from scoring toward diagnosis.

Statistics: Take Variance Seriously

Run each case several times, because outputs vary. Report pass rates with confidence intervals, and resample by case, not by trial, since trials within a case are correlated. When comparing two versions, use paired comparisons on the same cases, and don't celebrate a two-point gain on a 40-case suite. Chasing noise leads to prompt changes that "helped" by chance.

A Working Evaluation Harness

This harness runs each case multiple times in a fresh environment, checks state and trajectory, and reports pass@1, pass@k, pass^k, cost, and bootstrap confidence intervals. Your agent is any function that takes a task and an environment and returns a trace with tool_calls, final_text, and cost:

import random
import statistics
import time
from dataclasses import dataclass, field
from math import comb

@dataclass
class Case:
    id: str
    task: str
    setup: callable                 # returns a fresh environment
    check_state: callable           # (env) -> bool: did the world end up correct?
    forbidden: list = field(default_factory=list)   # tools that must never be called
    max_steps: int = 20

def run_trial(agent, case):
    env = case.setup()
    t0 = time.time()
    trace = agent(case.task, env)
    outcome_ok = bool(case.check_state(env))
    forbidden_hit = [c for c in trace.tool_calls if c[0] in case.forbidden]
    within_budget = len(trace.tool_calls) <= case.max_steps
    return {
        "success": outcome_ok and not forbidden_hit and within_budget,
        "outcome_ok": outcome_ok,
        "forbidden": bool(forbidden_hit),
        "steps": len(trace.tool_calls),
        "cost": trace.cost,
        "seconds": time.time() - t0,
    }

def pass_at_k(n, c, k):
    return 1.0 if n - c < k else 1 - comb(n - c, k) / comb(n, k)

def pass_pow_k(n, c, k):
    return comb(c, k) / comb(n, k) if c >= k else 0.0

def bootstrap_ci(values, iters=2000, alpha=0.05):
    means = sorted(statistics.mean(random.choices(values, k=len(values)))
                   for _ in range(iters))
    return means[int(alpha / 2 * iters)], means[int((1 - alpha / 2) * iters) - 1]

def evaluate(agent, cases, n=5, k=3):
    rows = []
    for case in cases:
        trials = [run_trial(agent, case) for _ in range(n)]
        c = sum(t["success"] for t in trials)
        rows.append({
            "id": case.id, "n": n, "c": c, "pass1": c / n,
            "pass_pow_k": pass_pow_k(n, c, k), "pass_at_k": pass_at_k(n, c, k),
            "forbidden": sum(t["forbidden"] for t in trials),
            "cost": statistics.mean(t["cost"] for t in trials),
            "steps": statistics.mean(t["steps"] for t in trials),
        })
    pass1_by_case = [r["pass1"] for r in rows]
    return {
        "pass1": statistics.mean(pass1_by_case),
        "pass1_ci": bootstrap_ci(pass1_by_case),
        "pass_pow_k": statistics.mean(r["pass_pow_k"] for r in rows),
        "pass_at_k": statistics.mean(r["pass_at_k"] for r in rows),
        "forbidden_rate": sum(r["forbidden"] for r in rows) / (len(rows) * n),
        "mean_cost": statistics.mean(r["cost"] for r in rows),
        "per_case": rows,
    }

def gate(new, baseline, max_drop=0.03):
    """Fail the build if quality drops beyond tolerance."""
    return (new["pass1"] >= baseline["pass1"] - max_drop
            and new["pass_pow_k"] >= baseline["pass_pow_k"] - max_drop
            and new["forbidden_rate"] <= baseline["forbidden_rate"])

A case for a support agent looks like this:

def make_env():
    return {"orders": {"A100": {"status": "shipped", "total": 59.97}}, "refunds": []}

refund_case = Case(
    id="refund-damaged",
    task="Order A100 arrived damaged. Please refund $20.",
    setup=make_env,
    check_state=lambda env: env["refunds"] == [("A100", 20.0)],
    forbidden=["delete_order"],
)

The check looks at the world's final state, not at what the agent said. Add cases where the right behavior is to refuse or escalate — such as a refund request above the allowed limit or an injection attempt hidden in an order note — so you measure restraint as well as competence.

Trace-Level Evaluation and Production

Offline suites miss things that production reveals, so evaluation has to continue after launch.

• Log full traces: every model call, tool call, argument, result, and state change, ideally with OpenTelemetry-compatible tracing so they flow into your existing observability stack. A published argument for log analysis puts it plainly: credible agent evaluation requires examining the logs, not just the scores.
• Label failures with a taxonomy: wrong tool, bad arguments, ignored observation, loop, premature stop, policy violation, false success. Counts by category tell you what to fix first.
• Turn failures into cases. Every escalation or bad session should become a regression test.
• Monitor online: success proxies such as task completion, escalation rate, user feedback, retry rate, cost per task, and guardrail triggers. Sample sessions for human or judge review.
• Release carefully: shadow mode, canaries, and A/B tests let you compare versions on live traffic with limited blast radius.

Using Public Benchmarks Wisely

Benchmarks are useful for shortlisting models and tracking the field, not for declaring your agent ready.

• SWE-bench Verified grades patches with real unit tests on GitHub issues, which makes it deterministic and a long-standing standard for coding agents. It's mostly Python, and one 2026 overview calls it somewhat saturated. METR also published a note arguing that many SWE-bench-passing patches wouldn't be merged by maintainers, a reminder that passing tests isn't the same as production-quality code — a point the coding agent guide makes from the harness side.
• Terminal-Bench tests agents operating in real command-line environments.
• tau-bench and tau2-bench simulate tool-using customer service conversations, check final database state, and report pass^k reliability.
• GAIA covers multi-step tasks needing web and file tools, with 466 questions in one description. WebArena and BrowseComp test web navigation and persistent browsing, and OSWorld tests desktop GUI control. One 2026 summary of the AI Index cites GAIA accuracy at 74.5%, WebArena at 74.3% against a 78.24% human baseline, and OSWorld at 66.3%, though treat secondary summaries of figures with care.
• AgentBench spans several environments including operating systems and databases.
• Safety and trust benchmarks, such as ST-WebAgentBench and agent-safety suites, probe risky behavior.
• Holistic Agent Leaderboard (HAL) work pushes for standardized, cost-aware, log-inspected evaluation across agents.

Caveats that apply to all of them: scores depend heavily on the scaffold, not just the model; benchmark data can leak into training sets; many report only terminal success, ignoring process and cost; and none of them reflect your tools, policies, or users. A leaderboard score is not the same as production readiness.

Test Strategy by Agent Type

• Research and RAG agents: citation precision (does each cited source support its claim?), coverage, honesty about gaps, and freshness. The research agent tutorial's quote verification is itself an evaluation primitive.
• Coding agents: hidden tests, diff size and relevance, test-tampering attempts, and the maintainability of changes.
• Customer support agents: policy adherence, correct state changes, correct escalation, and tone, with simulated users.
• Computer-use and web agents: final-state verification, safe handling of irreversible actions, and robustness to page changes.
• Multi-agent systems: compare against a single-agent baseline, and track coordination failures separately, as in the multi-agent guide.

A Tiered Pipeline

Match evaluation depth to cost and frequency:

• Unit tests for tools, parsers, and validators on every commit.
• Smoke evals: roughly 20 to 30 cases, mocked tools, one run each, on every pull request.
• Full evals: the whole suite with multiple trials, nightly or before merging major changes.
• Adversarial and safety evals: injection attempts, permission boundary tests, and red-team scenarios, before releases.
• Production monitoring: continuous sampling, with failures feeding back into the suite.

Control cost by caching deterministic tool results, using cheaper judges for triage, and sampling large suites.

Common Pitfalls

• Judging a single run, then shipping on luck.
• Scoring the final text instead of the final state.
• Ignoring cost, so a "better" agent quietly costs five times more.
• Trusting an unvalidated LLM judge.
• Letting eval cases leak into prompts or fine-tuning data.
• Optimizing for a public benchmark that doesn't resemble your use case.
• Non-hermetic environments, where live data changes break reproducibility.
• No baseline, so you can't tell whether a change helped.
• Missing restraint tests: agents that never refuse or escalate look great on easy cases and fail in production.
• Averaging away severity, when one catastrophic failure matters more than ten harmless ones. Weight cases by impact.

A Checklist Before You Ship

• Success is verified against state, not text.
• Each case runs several times, with pass^k reported alongside pass@1.
• Forbidden actions and step budgets are checked.
• Judges are calibrated against human labels.
• A held-out set exists, and baselines are recorded.
• Adversarial, safety, and restraint cases are included.
• Traces are logged and failures are categorized.
• CI gates catch regressions, and production failures become new cases.

Unlock AI That Actually Works

Get lifetime access to GPT-6 Astra, Claude Fable 5.1, Gemini 3.5, Grok 4.5, and more — all in one platform. Build websites, apps, videos, content, and digital products from a single command. No monthly fees. No tool-hopping.

Click here to get GPTAstra Max now — one-time payment, lifetime access.

Frequently Asked Questions

What is the difference between pass@k and pass^k?

pass@k is the chance that at least one of k attempts succeeds — it measures capability. pass^k is the chance that all k attempts succeed — it measures consistency. An agent that solves a task 8 times out of 10 looks strong under pass@1 or pass@k but fails badly under pass^k, because real users do not get to retry until it works. For customer-facing or high-stakes agents, pass^k is usually the metric that matters.

How many test cases does an agent evaluation suite need?

Start with 30 to 100 well-chosen cases built from real failure modes — production failures and escalations, support tickets, user feedback, known edge cases, adversarial inputs, and policy rules — and grow it by converting every incident into a regression case. Small suites produce wide error bars, so run each case several times, report confidence intervals, and don't over-read small differences between versions.

When is it OK to use an LLM-as-judge?

For open-ended quality that code can't check, such as writing, summaries, and reasoning explanations — as a calibrated supplement, not the foundation. Judges show position bias, verbosity bias, and self-preference, so use rubrics with explicit criteria, pairwise comparisons with randomized order, a different model family than the agent under test, temperature zero, and, most importantly, measure the judge's agreement with human labels on a sample before trusting it.

How should public benchmarks like SWE-bench be used?

For shortlisting models and tracking the field, not for declaring your agent ready. Scores depend heavily on the scaffold, benchmark data can leak into training sets, many report only terminal success while ignoring process and cost, and none of them reflect your tools, policies, or users. A leaderboard score is not the same as production readiness — your own task set, built from your real failures, is the benchmark that matters.

Wrapping This Up

Evaluating agents means judging outcomes, process, reliability, cost, safety, and honesty — with state-based checks as the foundation and LLM judges as a calibrated supplement. Use pass^k when users can't retry, report uncertainty instead of single numbers, build hermetic environments from real failures, and treat public benchmarks as a starting shortlist rather than a certificate.

What's the single most valuable habit? Turn every failure you see, in testing or in production, into a permanent test case. Over time that suite becomes the most accurate description of what "working" means for your agent, and the only benchmark that can truly say whether it's ready.

Related Articles

• Build Your First AI Agent in Python from Scratch (No Framework)
• Build a Coding Agent: Automate Code Generation and Debugging
• Build a Research Agent That Searches the Web and Writes Reports
• Multi-Agent Systems: Architecture Patterns That Work