How to Evaluate AI Agents in Production: A Trace-First System

The short answer is that a production agent should be evaluated as a complete system, not as a model answering a test set. The unit under test includes the model, prompt, tools, orchestration, permissions, data, runtime, and the environment the agent changes. The primary evidence is not the final sentence. It is the final state plus the trace that produced it.

That distinction changes the entire evaluation program. A support agent can say it issued a refund while leaving the payment record untouched. A coding agent can announce that tests pass after running only one test. A research agent can produce polished prose with citations that do not support its claims. In each case, grading the response rewards performance theater. Inspecting the environment and trace measures the work.

A production agent eval is a replayable test of the whole agent-environment loop. Grade the final state first, use the trace to explain the result, and treat every consequential production failure as the seed of a regression test.

The agent, not the model, is the system under test

A model benchmark holds the surrounding system fixed and scores model answers. It can inform model selection, but cannot tell you whether a deployed agent is safe or dependable.

The deployed result depends on at least six coupled parts:

1. Model: reasoning capability, tool-use behavior, calibration, and variance.
2. Instructions: system policy, task prompt, examples, and retrieved context.
3. Harness: the loop that selects tools, handles errors, compacts context, and decides when to stop.
4. Tools: schemas, implementations, timeouts, error messages, and side effects.
5. Environment: files, databases, network services, credentials, CPU, memory, and time limits.
6. Policy boundary: what the agent may read, change, send, spend, or delete.

Change any one of these and you have changed the system. A model upgrade can regress because a tool description now elicits a different argument shape. A prompt improvement can increase task success while causing more policy violations. A stricter container can make an otherwise capable agent time out. The release artifact is therefore a versioned tuple of model, harness, tools, policy, and environment, not a model name.

This is also why a single leaderboard number is insufficient. Public benchmarks are useful reference points, but the production question is local: does this exact system complete the work your users ask for, under the constraints your organization must preserve?

Separate outcome, trace, and report

Anthropic's agent evaluation guide (opens in a new tab) makes a useful distinction between a transcript and an outcome. The transcript, also called a trace or trajectory, records model messages, tool calls, intermediate results, and interactions. The outcome is the state of the environment after the run.

Add a third object: the report, meaning what the agent tells the user. These three can disagree.

Evidence	Question it answers	Typical grader
Final state	Did the requested change actually happen?	Database assertion, test, diff, API query
Trace	How did the agent reach that state?	Event rules, cost metrics, rubric, human review
Report	Did the agent explain the result accurately and clearly?	Schema check, groundedness check, calibrated rubric

The order matters. Grade final state first. If the state is wrong, elegant reasoning does not rescue the run. Then inspect the trace for policy adherence, recovery behavior, unnecessary actions, and the cause of failure. Finally, grade whether the report matches reality.

That is the production version of not letting an agent grade itself. An agent's claim is evidence about what it believes, not proof that the work is complete.

Build a scorecard that cannot hide failure

An aggregate pass rate is useful only when its components remain visible. Keep success, safety, efficiency, and uncertainty separate.

Dimension	Example measure	Why it stays separate
Task success	Required final-state assertions passed	The core outcome cannot be averaged away
Policy compliance	Forbidden actions and missing approvals	A successful unsafe run is still a failure
Reliability	Success across repeated trials	One lucky run is not dependable behavior
Recovery	Correct response to tool errors and partial failure	Normal operation hides brittle loops
Efficiency	Tool calls, tokens, latency, and external cost	Success at unbounded cost may not be viable
Report fidelity	Claims supported by state and trace	Prevents false completion messages
Escalation quality	Stops and asks for help at the right boundary	Abstention can be the correct outcome

Do not collapse hard constraints into a weighted average. If an agent exposes private data in one run and succeeds quickly in nine, a high average is a misleading release signal. Define blocking conditions such as any critical policy violation, any destructive action outside scope, or any report that asserts an irreversible action occurred when it did not.

For stochastic behavior, report more than the mean. Run repeated trials, publish the count, and show confidence intervals or raw results. For high-consequence tasks, measure the probability of at least one failure across a sequence. Reliability compounds when an agent performs many tasks every day.

Use three layers of evaluation

No single suite covers the production surface. Use three connected layers.

1. Capability and contract tests

These are controlled tasks with explicit initial state, allowed actions, and expected final state. They answer whether the agent can perform representative work. Keep tasks narrow enough to diagnose but realistic enough to exercise the loop.

For each task, define a contract:

• initial fixtures and clock state;
• tools and permissions available;
• required end-state assertions;
• forbidden actions and data flows;
• acceptable escalation conditions;
• resource and time budgets.

Use deterministic graders wherever the property is deterministic. Query the database after a booking task. Run tests after a code change. Resolve every citation after a research task. A model grader is appropriate for qualities such as clarity or completeness, but it should not replace a state check that code can perform exactly.

2. Regression and adversarial suites

Every fixed production defect should leave behind a smaller, reproducible task. Preserve the relevant initial state, the triggering input, the tool responses, and the violated invariant. Label the underlying failure class, not just the surface wording.

Then create variants. Change names, ordering, irrelevant detail, and error timing while preserving the causal structure. This prevents the suite from becoming a collection of memorized strings. Include tool timeouts, malformed results, stale records, ambiguous requests, conflicting policies, prompt injection, and partial side effects.

This is where the loop engineering mindset becomes measurable. A loop is not reliable because it succeeds on the happy path. It is reliable when its gates, retries, budgets, and escalation paths remain correct under pressure.

3. Production monitoring and sampled review

Offline suites cannot reproduce the full distribution of real users, live data, and changing dependencies. Production monitoring should capture schema-valid trace events, outcomes, policy decisions, latency, cost, and user corrections without retaining more sensitive content than necessary.

Monitor leading signals as well as visible failures: repeated tool calls, escalating retries, sudden changes in tool mix, missing citations, policy denials, and unusually long traces. Sample both successes and failures for human review. A system that only inspects reported failures will miss silent errors and users who simply leave.

Promote new production failures into the offline suite. This closes the evaluation loop: production discovers the distribution, the suite preserves what was learned, and the next release must prove it did not forget.

Make the trace useful, not merely large

Trace-first does not mean collecting every byte forever. Record enough structured evidence to reconstruct consequential decisions.

At minimum, record:

• run, task, user, agent, model, prompt, tool, and policy versions;
• a timestamped sequence of tool requests and normalized results;
• source and trust labels for external content;
• authorization and approval decisions;
• state snapshots or hashes before and after consequential actions;
• retry, timeout, cancellation, and escalation events;
• grader versions and verdicts.

Use stable event schemas so releases can be compared. Redact secrets at ingestion. Apply retention and access controls. A trace containing credentials or personal data can create a second security problem while trying to explain the first.

The trace should support two directions of analysis. Forward analysis asks what happened after a particular input or tool response. Backward analysis starts from a bad outcome and identifies the decision, data, and permission chain that produced it. Both are much easier when actions have identifiers and state changes carry provenance.

Control the environment or disclose the noise

Agentic evaluations are unusually sensitive to infrastructure because the agent interacts with the runtime. CPU, memory, network stability, package mirrors, tool latency, and time limits can alter both completion and strategy.

Anthropic reported a six percentage point difference (opens in a new tab) across resource configurations in its Terminal-Bench 2.0 experiments with the same model, harness, and task set. In the strictest configuration, infrastructure errors reached 5.8 percent; uncapped resources reduced them to 0.5 percent. These are results from one benchmark setup, not universal correction factors. They show why environment configuration belongs in the eval record.

Separate at least three verdicts:

• task failure: the agent completed the run but produced the wrong state;
• policy failure: the agent crossed a forbidden boundary;
• infrastructure failure: the environment prevented a valid trial.

Do not quietly remove infrastructure failures until the score improves. Report them, retry according to a written policy, and publish the policy with the result. Pin dependencies and fixtures, isolate runs, reset state between trials, and record resource limits. If the production environment differs, run a smaller validation suite there too.

Use model graders as instruments, not authorities

Model graders scale nuanced review, but they are non-deterministic instruments. Give them atomic rubrics, direct evidence, and an uncertainty option. Hide irrelevant metadata, randomize pairwise answer order, and compare them periodically with expert human labels.

Avoid asking one broad question such as "Was this a good run?" Split it into verifiable claims: Did every factual statement cite retrieved evidence? Did the agent disclose uncertainty? Did it follow the refund policy? Did its report match the final database state?

Research benchmarks illustrate the move toward richer environments. τ-bench (opens in a new tab) evaluates tool-using agents in multi-turn interactions while checking domain policies. AlphaEval (opens in a new tab), a 2026 preprint, proposes 94 tasks across six occupational domains and multiple business environments. These are useful designs to study, but neither substitutes for local tasks, and a recent preprint should be treated as provisional evidence until methods and results receive broader scrutiny.

A release gate for an agent change

Before shipping a model, prompt, tool, or harness change, require a versioned evaluation report with:

1. no critical policy or containment failures;
2. all deterministic contract assertions passing at the required threshold;
3. repeated-trial reliability within a predefined confidence bound;
4. no material regression in named task and failure categories;
5. resource, latency, and cost changes disclosed;
6. model-grader calibration still within its accepted range;
7. a canary plan, rollback trigger, and owner for production review.

Keep the gate category-aware. A one-point aggregate gain should not ship if authentication tasks regress or escalation becomes less reliable. The decision should show which users and workflows improve, which become worse, and which remain uncertain.

OpenAI's account of building a tax agent with Codex (opens in a new tab) is useful here because it frames improvement as a cycle of task execution, grading, error analysis, and targeted changes. The general lesson is not that an agent can safely optimize itself. It is that improvement becomes tractable when evidence is structured and external evaluation remains in control.

Production agent eval checklist

• Version the model, prompts, harness, tools, policies, fixtures, and graders.
• Define initial state, required final state, forbidden actions, and budgets per task.
• Grade environment state before the agent's report.
• Keep task success and policy compliance as separate release gates.
• Run repeated trials and report counts, variance, and retry rules.
• Inject tool errors, stale data, ambiguity, and adversarial content.
• Record structured traces with provenance, redaction, and retention controls.
• Distinguish task, policy, and infrastructure failures.
• Calibrate model graders against human experts.
• Sample successful production runs, not only reported failures.
• Convert consequential incidents into regression tasks and structural variants.
• Canary releases and define rollback triggers before exposure grows.

Sources

• Anthropic: Demystifying evals for AI agents (opens in a new tab)
• Anthropic: Quantifying infrastructure noise in agentic coding evals (opens in a new tab)
• OpenAI: Building a self-improving tax agent with Codex (opens in a new tab)
• τ-bench: A benchmark for tool-agent-user interaction (opens in a new tab)
• AlphaEval: A benchmark for agent evaluation in real-world environments (opens in a new tab)