How do you evaluate a multi-step agent that calls tools, not just a single LLM response?
Answer it out loud first, then compare. That's what makes it stick.
I grade two things: the outcome and the trajectory. Outcome means checking the end state, like whether the refund was actually issued in the database, not whether the agent said it was. Trajectory means checking tool calls: right tools, right arguments, required steps in order, no forbidden actions like writing before approval. I use a simulated user for multi-turn tasks, run each task several times, and report pass^k for consistency, because customers need it to work every time, not once.
- Grade the end state, not the message
- Grade the trajectory: tools, args, order, forbidden actions
- Simulated users for multi-turn tasks
- Multiple trials: pass@k vs pass^k
- Capability vs regression suites
Agents fail in ways single-response evals miss: the right answer reached through a dangerous action, or success once and failure on the next run.
Outcome grading
- Check the environment state after the run: was the order cancelled in the sandbox DB, the ticket updated, the file written? Agents often claim success they did not achieve.
- Use code graders where possible: Anthropic's agent-evals guide lists outcome verification, tool-call verification and transcript analysis (turns, tokens) as code-based graders.
Trajectory grading
- Google's agent evaluation docs split final-response from trajectory evaluation, with metrics such as exact match, in-order match, any-order match, precision, recall and single-tool use against a reference trajectory.
- Prefer flexible matches (in-order, any-order) because valid paths vary; insist on exact rules only for policy: 'identity check must precede account change', 'never call refund above limit without approval'.
- Add an LLM judge for policy adherence in the conversation, calibrated like any other judge.
Realistic inputs
- Multi-turn tasks need a simulated user (an LLM with a persona and a goal) plus a sandboxed copy of the tools and data. Sierra's tau-bench is the reference design: realistic APIs, a policy document and an LLM-simulated user.
Consistency: pass@k vs pass^k
- pass@k: at least one of k tries succeeds (fine for a coding assistant that proposes options).
- pass^k: all k tries succeed (what a customer-facing agent needs). At 75% per trial, pass^3 is about 42%.
- In tau-bench's first results, a GPT-4o agent was under 50% on pass^1 and fell to about 25% on pass^8 in retail.
Suites
- Capability evals start with a low pass rate and give a hill to climb.
- Regression evals should pass near 100%; anything less means something broke.
Practical extras
- Track cost and step count per task; a correct answer in 40 tool calls may be a failure.
- Log full transcripts so failing runs can be read, not just scored.
- How do you build a sandbox when the customer's tools are production APIs?
- How many trials per task is enough?
- The simulated user is unrealistic. How would you know?
- Demystifying evals for AI agents (official, 2026-01-09) Defines code, model and human graders, capability vs regression evals, and pass@k vs pass^k (75% per trial gives about 42% across 3 trials).
- Evaluate your AI agents with Vertex Gen AI evaluation service (official, 2025-01-25) Separates final-response from trajectory evaluation and lists trajectory metrics: exact, in-order, any-order match, precision, recall, single-tool use.
- tau-Bench: Benchmarking AI agents for the real-world (official, 2024-06-20) Introduces pass^k; a GPT-4o agent dropped to about 25% pass^8 in the retail domain versus under 50% pass^1.