Evaluating Agentic AI in Production

Agentic systems are moving into production faster than the discipline to evaluate them. The capability benchmarks that vendors cite measure single attempts on curated tasks; production demands consistency across thousands of real ones — and the gap between the two is where most agent programs quietly fail. This research-backed guide explains why agent evaluation is fundamentally different from model evaluation, what to measure across the trajectory rather than just the output, and how to stand up a production evaluation system — grounded in findings from Sierra's τ-bench, the Pan et al. practitioner survey, OpenTelemetry, OWASP, NIST, and the EU AI Act.

A customer-service agent that resolves a refund correctly in a demo has cleared a low bar. The question that determines whether it belongs in production is different: will it resolve the same issue correctly for the next eight customers who phrase it eight different ways — without leaking data, looping on a tool call, or violating a policy buried on page nine of the operations manual? That is an evaluation question, and it is the one most agent programs answer last, if at all.

The industry has spent three years optimizing agents for capability. It is now discovering that capability and reliability are not the same property, that the benchmarks rewarding the former say little about the latter, and that the discipline of measuring an agent in production is its own engineering problem. This article lays out what that discipline looks like.

Reliability, Not Capability, Is the Production Barrier

The most rigorous evidence on this point comes from a benchmark built specifically to test it. In 2024, Sierra introduced τ-bench (Yao et al.), which evaluates agents on realistic multi-turn customer-service tasks in retail and airline domains, with a user simulated by a language model and a policy document the agent must follow. Crucially, τ-bench introduced a metric the capability benchmarks ignore: pass^k, the probability that an agent succeeds on all k repeated attempts at the same task, not just one.

The result was sobering. The best agent they tested, a GPT-4o function-calling agent, achieved above 60 percent average success on a single attempt (pass^1) but dropped below 25 percent at pass^8 in the retail domain. As Sierra put it, that means there is only about a 25 percent chance the agent resolves eight cases of the same issue with different customers — a number far behind the expectation of a real-world user-facing agent. All twelve models they tested degraded the same way as k increased.

Even for the best-performing gpt-4o function calling agent which has a > 60% average task success, pass^8 drops to < 25%. In real-world scenarios, it is important and challenging not just to build agents with high average success (pass^1), but with more robustness and consistency. — Yao et al., τ-bench (2024)

This is the core finding that reframes agent evaluation: a single-attempt success rate systematically overstates how an agent behaves in production, because production is the repeated trial. A 70-percent-in-testing agent that performs at 30 percent consistently creates a poorer user experience than a 60 percent agent with reliable performance.

Practitioners building real systems have reached the same conclusion from the field. The first large-scale study of agents actually in production — Pan et al. (2025), surveying 306 practitioners across 26 domains with 20 in-depth case studies — found that reliability remains the top development challenge, driven by the difficulty of ensuring and evaluating agent correctness. Their most telling finding is what builders do in response: rather than reach for sophisticated techniques, they constrain their agents. In their sample, 68 percent of agents execute at most 10 steps before requiring human intervention, 70 percent rely on prompting off-the-shelf models rather than fine-tuning, and 74 percent depend primarily on human evaluation. Teams are trading away autonomy to buy back reliability — because they cannot yet measure the reliability of more autonomous systems well enough to trust them.

That trade-off is rational, but it caps the value agents can deliver. The way past it is not less autonomy; it is better evaluation. And evaluating an agent requires measuring things a model benchmark never touches.

Why Agent Evaluation Is a Different Problem

Evaluating a language model and evaluating an agent answer different questions. A model benchmark like MMLU tests a capability in isolation: given a prompt, is the answer correct? An agent, by contrast, is an autonomous system that operates in a loop — it plans, selects tools, calls them with arguments, observes results, recovers from errors, and chains many such steps toward a goal. As NVIDIA's evaluation guidance puts it, two agents can produce the identical final answer while behaving completely differently: one with three precise tool calls, the other thrashing through dozens of irrelevant steps. Final-answer grading treats them as equal. Production does not.

This is why the field has converged on evaluating agents at three levels, a model articulated clearly by the teams at Langfuse, Confident AI, and others:

• Final-response evaluation asks whether the agent accomplished the user's goal — task completion, also called goal accuracy. It tells you *what* went wrong, but not where or why.
• Trajectory evaluation scores the path the agent took: which tools it selected, in what order, with what arguments, and how it recovered from failures. It tells you *where* it went wrong.
• Single-step evaluation scores an individual decision — one tool call, one reasoning step — in isolation. It tells you *why* it went wrong.

The reason all three are necessary is a failure mode unique to agents: an agent can reach the right answer through the wrong reasoning, or the wrong answer through mostly-right reasoning. An outcome-only score gives the first agent a perfect grade and hides a structural flaw that surfaces the moment the task shifts slightly. Trajectory evaluation is what catches it.

Within the trajectory, the metrics that matter most cluster around tool use, because most production agents succeed or fail on how they use tools rather than how they phrase text:

• Tool-selection precision and recall — were the right tools called, and the wrong ones avoided? Precision is correct calls over total calls; recall is correct calls over required calls.
• Argument accuracy — were tools invoked with correct parameters? For multi-parameter tools, often scored as the fraction of correct arguments rather than all-or-nothing.
• Order and dependency satisfaction — were interdependent steps executed in a valid sequence? Trajectory-aware benchmarks such as TRAJECT-Bench formalize this with exact-match and inclusion metrics over the required tool sequence.
• Trajectory efficiency — how many steps and tokens did success cost? An agent that gets the right answer but burns 50,000 tokens where 3,000 would do is likely misbehaving — stuck in a loop or re-reading context. Efficiency is both a cost metric and a reliability signal.

These trajectory metrics are not available from a final-answer benchmark. They require that the agent be instrumented to emit its full decision graph — which is where production evaluation meets observability.

The Benchmark-to-Production Gap

Public benchmarks, useful as they are, cannot be the basis for a production go/no-go decision, for three reasons.

The first is saturation and non-comparability. Several headline agent benchmarks are near their ceiling — GAIA's high score has reached roughly 90 percent, and on SWE-bench Verified, vendor-reported scores climbed from under 2 percent in 2023 to above 80 percent by late 2025. But those SWE-bench numbers, as independent analysis stresses, are not directly comparable across vendors due to scaffold, tool, and evaluator differences. A benchmark score is a property of an entire agentic system — model plus scaffolding, tools, and harness — not of a model you can drop into your own stack and expect to reproduce.

The second is environment drift and limited transfer. A 2026 review of agentic evaluation in Artificial Intelligence Review documents the tension directly: real-world web-agent datasets like Mind2Web offer ecological validity but suffer drift, with a substantial share of tasks going invalid within months as websites change; simulated environments like WebArena are reproducible but transfer imperfectly to live sites. No public dataset simultaneously maximizes realism, reproducibility, and trajectory richness.

The third is the gap between lab and deployment itself. Industry analyses of 2026 evaluation practice report a large divergence between benchmark scores and real-world deployment performance, alongside order-of-magnitude cost variation between systems achieving similar accuracy. The lesson is not that benchmarks are worthless — they are valuable for tracking the field and for initial model selection — but that they are a starting point, not evidence that a specific system is production-ready on a specific workflow. That evidence has to be generated against your own data, tools, and policies.

Evaluation in Production: The Online Loop

Offline evaluation — running a curated dataset of test cases before release — is necessary but not sufficient. Production traffic contains inputs no test set anticipated, and agent behavior drifts as models are updated, documents change, and user populations shift. The discipline that closes this gap is online evaluation: continuously scoring live production traces to detect drift, regressions, and new edge cases, then feeding what you learn back into the offline suite.

Google Cloud's agent-evaluation guidance describes the loop concretely. In production you monitor operational metrics (tool-call error rates, latencies, token consumption per interaction), quality and engagement metrics (thumbs-up/down, task-completion rates), and drift — significant changes in query types or a gradual decrease in performance over time. The final step is the one that compounds: feed production data back into your evaluation assets, so the suite becomes a living entity that learns from real use. Databricks describes the same pattern as a continuous loop of production traces feeding updated datasets.

A practical sequencing principle comes from the Langfuse team's field experience: start with tracing, not scoring. Before building automated evaluators, review agent traces manually; the failure patterns you observe tell you which metrics actually matter. Then define success criteria explicitly — the expected final answer, tool sequence, and arguments — because vague criteria produce unreliable evaluations.

This online loop cannot exist without two things most pilots lack: comprehensive tracing of every agent step, and a scalable way to score outputs for which there is no ground-truth label. The first is an observability problem. The second is the LLM-as-a-judge problem.

Observability: The Foundation Is Now a Standard

You cannot evaluate what you cannot see, and agents are unusually hard to see into. An agent can fail gracefully from a systems perspective — no exception thrown, an HTTP 200 returned — while producing a wrong or harmful result. Observability for agents therefore has to capture not just whether it ran, but what it decided and why.

The significant development of the past two years is that this is no longer a bespoke undertaking. The OpenTelemetry GenAI Semantic Conventions, developed by a working group that began in April 2024, now define a standard, vendor-neutral schema for AI telemetry: a consistent vocabulary for spans, metrics, and events covering model calls, token usage, tool calls, and agent operations. For an agent, a trace resolves into a span tree — a top-level invoke_agent span with child chat spans for each LLM call and execute_tool spans for each tool invocation — annotated with standardized attributes such as gen_ai.request.model, gen_ai.usage.input_tokens, and gen_ai.response.finish_reasons.

The conventions have moved quickly to cover agent-specific structure. The GenAI agent-span conventions define operations for agent creation and invocation, and the in-progress extensions add the concepts an agent system needs to be observable: orchestration and workflow spans, agent-to-agent handoffs, memory operations, guardrail checks, and human-in-the-loop review points. The schema is designed to capture the decision graph of an agent rather than just its input/output boundary. Adoption is broad — major observability vendors including Datadog support the conventions natively, and frameworks such as LangGraph, CrewAI, and AutoGen emit compatible spans — so a span from one framework looks like a span from another, and telemetry becomes comparable across the stack.

One caveat defines the boundary of what the standard gives you: OpenTelemetry standardizes the data plane — how telemetry is captured and moved — but, as Fiddler's analysis notes, it stops at output evaluation and safety scoring. Capturing a faithful trace is the prerequisite; judging whether what the agent did was correct, safe, and policy-compliant is a separate layer you still have to build on top.

LLM-as-a-Judge: Powerful, Scalable, and Biased

Most agent outputs have no ground-truth label. There is no answer key for whether a support response was helpful, whether a summary was faithful, or whether a multi-step research trajectory was sound. To evaluate these at production scale, teams increasingly use a strong language model to score the outputs of another — the LLM-as-a-judge pattern.

The approach has real empirical support. The foundational work by Zheng et al. (2023) found that GPT-4 as a judge agreed with human evaluators more than 80 percent of the time — roughly the rate at which human evaluators agree with each other. That made automated evaluation mainstream almost overnight: it offered human-approximate judgment at API speed and cost, on exactly the open-ended, semantic qualities that older metrics like BLEU and ROUGE fail to capture.

But the research that followed has been a steady catalogue of the ways LLM judges fail. Multiple peer-reviewed surveys now document a consistent taxonomy of judge biases:

• Position bias — judges favor responses by their position in the prompt; swapping the order of two candidates can flip the verdict.
• Verbosity bias — longer answers are preferred irrespective of quality.
• Self-enhancement bias — a model tends to rate its own outputs more favorably than those of other models.
• Authority bias — answers that cite sources are favored even when the citations are fabricated.

These are not edge cases. A survey on LLM-as-a-judge catalogues them as systematic, and notes that judges are also vulnerable to adversarial manipulation, including prompt-injection attacks designed to influence the score. Industry analysis of 2025–2026 bias research reached a blunt summary: no LLM judge evaluated in one major study was uniformly reliable across benchmarks.

The implication is not to abandon LLM-as-a-judge — at production volumes there is often no alternative — but to engineer around its weaknesses. The research points to mitigations that work: provide a reference answer when one exists, since reference-based scoring is consistently more reliable; use detailed, explicit rubrics rather than a holistic score; control for position by swapping candidate order and averaging; and — the governing principle — calibrate judges against human labels on a sample, treating the judge as a measurement instrument whose own sensitivity and specificity must be validated, not assumed. The reliable production posture starts with human review on a sample, uses it to align the automated judge, then scales the judge while continuing to spot-check it on high-stakes decisions. Automated pipelines catch predictable failures; human experts catch the edge cases.

Agentic Evaluation Is Also a Security Problem

For agents, evaluation and security are not separable. An agent that can browse the web, query databases, execute code, and call APIs has a blast radius a chatbot does not, and evaluation has to cover adversarial behavior, not just correctness on cooperative inputs.

The reference catalogue is the OWASP Top 10 for Agentic Applications, released in December 2025 as the first peer-reviewed framework targeting autonomous-AI security specifically, developed with input from over 100 security experts. It identifies the failure modes that distinguish agentic security: agent goal hijacking (manipulating objectives through malicious instructions in processed content), tool misuse (legitimate tool access induced to act destructively), identity and privilege abuse, and cascading failures across connected agents. Prompt injection — ranked the top risk in the OWASP Top 10 for LLM Applications — is amplified in agentic settings because a single injected instruction can trigger real-world actions, not just unintended text.

These are not theoretical: OWASP's 2026 exploit reporting documents agentic risks exploited in the wild, including tool-misuse and code-execution vectors through agent frameworks and Model Context Protocol tooling. The consequence for evaluation is that a production agent's test suite must include adversarial cases — injection attempts embedded in retrieved content, tool-misuse scenarios, attempts to exceed authority — scored in the same harness that measures task success. OWASP's recommended posture is defense-in-depth: least-privilege tooling, input and output filtering, human approval for high-risk actions, and regular adversarial red-teaming. Reliability evaluation answers *does it work*; security evaluation answers *can it be made to do harm*. Production requires both answers.

Governance: Evaluation Is the Evidence Regulators Will Ask For

The evaluation artifacts described above are increasingly the documentation that governance frameworks and regulators expect. Treating evaluation as audit evidence from the start is far cheaper than reconstructing it later.

The NIST AI Risk Management Framework (AI RMF 1.0, 2023) organizes practice around four functions — GOVERN, MAP, MEASURE, MANAGE — and its Generative AI Profile (NIST AI 600-1), released July 2024, maps GenAI-specific risks such as confabulation and information integrity into that structure with more than 200 suggested actions. The MEASURE function is, in effect, a mandate for exactly the continuous evaluation discipline this article describes.

ISO/IEC 42001:2023, the first certifiable standard for an AI management system, requires documented controls across the AI lifecycle — including the monitoring and measurement an online evaluation loop produces — and is increasingly requested by enterprise buyers during vendor due diligence.

The EU AI Act (Regulation (EU) 2024/1689) makes the connection explicit for high-risk systems: its substantive obligations include risk management, logging, human oversight, accuracy and robustness, and post-market monitoring — none of which can be satisfied without the tracing, evaluation, and drift-detection machinery of a production evaluation system. The Act's phased obligations for general-purpose models became binding in August 2025; high-risk-system obligations follow on the timeline set by the subsequent simplification process. A system designed now should already emit the evaluation logs and monitoring records those obligations will require, because producing them retroactively is markedly more expensive.

The throughline across all three frameworks is the same: the evaluation system is the governance system. The traces that let you debug an agent are the logs an auditor will request; the evaluation suite that gates your releases is the evidence of testing a regulator expects; the drift monitors that protect your reliability are the post-market monitoring the law requires. Built once, they serve both purposes.

A Reference Architecture for Production Agent Evaluation

A workable evaluation system has five components. They are rarely a single product; most teams assemble them from open-source frameworks, cloud services, and a small amount of custom code. What matters is that all five are present and owned.

Tracing and instrumentation

Every agent run emits a full trace — session, agent invocation, each LLM call, each tool call with arguments and results, memory operations, guardrail checks, and human-review points — using the OpenTelemetry GenAI conventions so the telemetry is standard and portable. This is the substrate everything else reads from.

Offline evaluation harness

A curated, versioned set of datasets — golden cases, regression cases captured from past failures, and adversarial cases including injection and tool-misuse attempts — scored at all three levels. The harness runs in CI on every prompt, model, or tool change, and a regression blocks the release.

Online evaluation loop

Sampling of live production traffic, scored by automated judges and human reviewers, with results monitored for drift in inputs, outputs, and cost. New failure modes discovered in production are promoted into the offline harness, closing the loop.

Judge and scoring layer

The LLM-as-a-judge configuration — rubrics, reference answers where available, position-bias controls — calibrated against human labels and continuously spot-checked. This is the layer OpenTelemetry explicitly does not provide and the team must own.

Reliability and security metrics plane

The dashboards and thresholds that turn raw scores into operational signals: pass^k-style consistency metrics rather than single-run success, tool-call accuracy and trajectory efficiency, cost per resolved task, adversarial-case pass rates, and the SLOs and alerts built on top. The specific tools matter less than two things: that each component has a named owner, and that the five are wired into one loop rather than living as disconnected scripts.

A 90-Day Path to Production Evaluation

For a team with an agent that already works on a slice of real traffic, a focused quarter is usually enough to reach a defensible evaluation baseline. The sequencing matters: instrument before you score, and score before you scale.

Days 1–30: See everything

Instrument the agent end-to-end with OpenTelemetry GenAI tracing so every run produces a full span tree. Review traces manually — start with tracing, not scoring — to learn the actual failure modes. Stand up the first offline harness from golden and regression cases, scored at the final-response and trajectory levels. Exit criterion: a named owner can answer what the agent did on a given run, and where it went wrong, from the trace alone.

Days 31–60: Judge and gate

Build the judge layer — explicit rubrics, reference answers where they exist, position-bias controls — and calibrate it against human labels on a sample. Add tool-selection, argument-accuracy, and trajectory-efficiency metrics. Add adversarial cases from the OWASP agentic catalogue. Wire the harness into CI so a regression blocks release. Exit criterion: no change ships without passing the harness, and the judge's agreement with humans is measured, not assumed.

Days 61–90: Close the loop and harden the number

Stand up the online loop: sample production traffic, score it, monitor input/output/cost drift, and promote new failures into the offline set. Replace single-run success reporting with a consistency metric (pass^k-style repeated trials on representative tasks) so the reliability number reflects production reality. Produce the first governance artifacts as a by-product. Exit criterion: a weekly evaluation review is on the calendar, and reliability and cost-per-task trends are published to the people accountable for the workflow.

A team that completes this arc has not finished — evaluation is permanent — but it has crossed from "the demo worked" to "we can tell, continuously, whether the system works."

The Discipline Is the Moat

The most consistent signal across the evidence is that the hard part of agentic AI is not building an agent that can do the task once. τ-bench shows that even capable agents are inconsistent across repeated trials; the Pan et al. survey shows practitioners already know this and constrain their agents to compensate; the benchmark literature shows public scores do not transfer cleanly to specific deployments; and the governance frameworks are converging on continuous measurement as a requirement.

What separates the agents that reach durable production from the ones that stall is therefore not a model choice. It is whether the team built the boring, compounding machinery the demo skips: full tracing, a three-level evaluation harness, a calibrated judge, an online loop that learns from production, adversarial coverage, and a reliability number that reflects the repeated trial rather than the lucky single run. Each compounds — every captured regression makes the next release safer, every promoted production failure makes the harness sharper, every calibration makes the judge more trustworthy.

The agent is replicable. The evaluation system around it — tuned to your workflow, your tools, your policies, and your risk posture — is not. That is where the durable advantage lives.

References

• Yao, S. et al. *τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains.* 2024. arxiv.org · sierra.ai
• Pan, M. et al. *Measuring AI Agents in Production* (survey of 306 practitioners). 2025. arxiv.org
• *Beyond Accuracy: A Multi-Dimensional Framework for Evaluating Enterprise Agentic AI Systems (CLEAR).* 2025. arxiv.org
• *From Benchmarks to Deployment: A Comprehensive Review of Agentic AI Evaluation.* Artificial Intelligence Review, 2026. link.springer.com
• *TRAJECT-Bench: A Trajectory-Aware Benchmark for Evaluating Agentic Tool Use.* 2025. arxiv.org
• Zheng, L. et al. *Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.* 2023. arxiv.org
• *From Generation to Judgment: Opportunities and Challenges of LLM-as-a-Judge.* 2024. arxiv.org
• OpenTelemetry. *Semantic Conventions for GenAI Agent and Framework Spans.* opentelemetry.io
• Google Cloud. *A Methodical Approach to Agent Evaluation.* November 2025. cloud.google.com
• Langfuse. *Agent Evaluation: How to Evaluate LLM Agents.* langfuse.com
• NVIDIA. *Mastering Agentic Techniques: AI Agent Evaluation.* 2026. developer.nvidia.com
• Databricks. *What is AI Agent Evaluation?* databricks.com
• Fiddler AI. *OpenTelemetry for AI Observability: What It Covers and Where It Stops.* 2026. fiddler.ai
• OWASP GenAI Security Project. *Top 10 for Agentic AI Applications (2026).* December 2025. genai.owasp.org
• NIST. *AI Risk Management Framework (AI RMF 1.0).* 2023. nist.gov · *Generative AI Profile (NIST AI 600-1).* July 2024. nvlpubs.nist.gov
• ISO/IEC. *ISO/IEC 42001:2023 — AI Management System.* 2023. iso.org
• European Union. *Regulation (EU) 2024/1689 (Artificial Intelligence Act).* In force August 1, 2024. artificialintelligenceact.eu