Agent evals: grade the path, not just the answer

If you only score an agent's final answer, you are grading luck as generously as skill. An agent that answers correctly in three clean steps and one that answers correctly after eight dead-ends, a retry loop, and a call it should never have been allowed to make are indistinguishable to an outcome-only eval — they both score 1.0. The second agent is not working. It is failing in a way that costs you tokens, latency, and eventually an incident, and your test suite is telling you everything is fine. Trajectory evaluation is the fix: score the intermediate steps, not just the output.

The gap is measurable, and it's large

The cleanest quantification of this comes from mathematical reasoning rather than tool use — which makes it the friendliest possible case for outcome-only grading, because in maths "the answer" is unambiguous and trivially checkable. Even there, the outcome signal is badly lossy.

In Step-wise Rubric Rewards for LLM Reasoning (May 2026), the authors analysed 1,000 problems and found:

• 18.2% of steps inside correct-answer responses are wrong — yet positively rewarded, because the answer came out right.
• 49.9% of steps inside incorrect-answer responses are correct — yet penalised, because the answer came out wrong.

Read those two numbers together. Roughly one step in five inside your "passing" runs is bad, and half the work inside your "failing" runs is fine. Outcome-only grading isn't a slightly noisy signal about process quality; on this evidence it's close to a coin flip at the step level.

This is not a new observation, just a newly quantified one. OpenAI's Let's Verify Step by Step established the direction in 2023: process supervision significantly outperformed outcome supervision, with the process-supervised model solving 78% of a representative MATH test subset — and it took 800,000 human step-level labels to build the dataset that proved it.

Agents are strictly harder than maths problems. There is no single verifiable answer, the action space includes side effects on real systems, and a run can be twenty steps long. If outcome-only grading loses this much signal on arithmetic, it loses more on an agent that touches your database.

What outcome-only grading actually measures

Here is the mechanism, stated plainly: an outcome-only eval compresses an entire multi-step run into one bit. Pass or fail. Everything about how the agent got there — which tools it chose, what it passed them, what order it ran them in, how many times it went in a circle — is discarded before the grader sees it.

That compression has three consequences that show up in production:

Silent inefficiency. A run that takes 18 tool calls instead of 4 passes the eval identically. You discover the cost in your bill and the latency in a support ticket, not in CI.

Silent policy violations. A correct final answer reached via a call that shouldn't have been permitted — a full refund where a partial was authorised, a write where a read was scoped — is a green tick. The eval had no opinion about intermediate actions because it never looked at them.

Regressions that hide. Change a prompt and the agent starts taking a worse path to the same answer. Your pass rate is flat. Nothing alerts. Three weeks later the path degrades enough to break, and the change that caused it is long merged.

Google's Agent Development Kit documentation puts the distinction crisply:

Unlike evaluating generative models, where the focus is primarily on the final output, agent evaluation requires a deeper understanding of the decision-making process.

This is the same reasoning as why traces beat logs in observability — an aggregate outcome tells you that something is wrong; the sequence tells you where.

What a trajectory grader scores

TRAJECT-Bench, a trajectory-aware benchmark for agentic tool use, makes the omission explicit in its abstract: existing work "largely focus[es] on the final answers yet overlook[s] the detailed tool usage trajectory, i.e., whether tools are selected, parameterized, and ordered correctly." That sentence is a specification. Four dimensions, each independently scoreable:

1. Tool selection — did it reach for the right tool for that step? The benchmark's authors identify similar tool confusion as a distinct failure mode: with a large tool surface, agents pick a plausible neighbour of the correct tool.
2. Arguments — were the parameters right? Their second named failure mode is parameter-blind selection: the right tool, called with wrong or unconsidered arguments.
3. Order and dependencies — did prerequisite steps actually run first, or did the agent guess at a value it should have fetched?
4. Efficiency — loops, dead-ends, redundant calls. The Step-wise Rubric Rewards work found self-correction looping in 48.1% of runs under one setup; the point is that it's common enough to be worth a metric.

TRAJECT-Bench's most useful finding for planning purposes is where agents actually break: the bottleneck isn't long trajectories, it's the transition from short to mid-length ones. A tool-use pipeline that looks solid at three steps is not evidence it will hold at eight.

Making it concrete

You don't need a research harness to start. The metric families are simple and mostly already implemented in the major agent platforms — Google's Gen AI evaluation service ships trajectory_exact_match, trajectory_in_order_match, trajectory_any_order_match, trajectory_precision and trajectory_recall as built-ins, and ADK exposes tool_trajectory_avg_score alongside rubric-based tool-use judges.

Pick the strictness that matches the task:

The honest catch: most of these need a reference trajectory — a human-written ideal path per test case. That's real work, and it's the reason teams skip this. Two ways to make it affordable. First, don't write references for everything; write them for the ten flows where a wrong path is expensive, and leave the rest on outcome-only. Second, use rubric-based judging where a canonical path doesn't exist — score "did it avoid unauthorised writes?" as a per-step assertion rather than diffing against a reference. Deterministic assertions on individual steps are cheap, exact, and catch the class of failure you care most about.

Our opinion

Trajectory evals are worth the effort exactly where a wrong path costs something, and nowhere else. We don't think every agent needs a reference trajectory for every case — that's the kind of completionism that gets an eval suite abandoned in month two. But if your agent can spend money, write to a database, send a message, or touch anything a regulator would ask about, then "it got the right answer" is not a passing grade and never was.

The sharper point: the four dimensions above are not four separate metrics, they're one question asked four ways — would a competent colleague have done it this way? An agent that picks a near-miss tool, passes it a guessed argument, runs it before its prerequisite, and loops twice is not "mostly working with some inefficiency." It is a system with no reliable model of the task that happened to land on the answer. Outcome-only grading can't see the difference, and the difference is the whole thing.

We'd also treat trajectory data as an operational asset, not just a test artefact. The trace you capture to grade an agent in CI is the same trace you need to debug it in production — build it once, use it in both places.

How Ashvara helps

We build agents that do real work, which means we build the evaluation harness alongside them rather than after. That's step-level assertions for the actions that matter, reference trajectories for the flows where the path is the requirement, and traces instrumented so the same data grades the agent offline and explains it in production.

It's part of how we approach AI solutions generally — the same discipline behind why you can't ship AI you can't evaluate, applied one layer deeper. If you have an agent that passes its tests and still surprises you in production, tell us what it's doing and we'll help you see the path it's taking.

Sources: TRAJECT-Bench: A Trajectory-Aware Benchmark for Evaluating Agentic Tool Use (He et al.); Step-wise Rubric Rewards for LLM Reasoning (Xie et al., 2026); Let's Verify Step by Step (Lightman et al.); Google ADK agent evaluation docs. The 18.2% / 49.9% figures are measured on mathematical reasoning steps, not tool-call trajectories — cited here because it is the easiest case for outcome-only grading, not the hardest.