A key quality metric for our customer-facing AI agent dropped last week. How do you diagnose it and decide what to fix first?
Answer it out loud first, then compare. That's what makes it stick.
First I confirm the drop is real and not a measurement change: did the judge, the eval set or the logging change? Then I find when it started and line it up against four suspects: our deploys and prompt edits, a provider model update, a change in the customer's data or tools, and a shift in user traffic. I slice the metric by intent, channel and customer segment to localise it, read failing traces, and prioritise fixes by volume times business impact.
- Is the drop real? Check the measurement
- Timeline vs deploys, model, data, traffic
- Slice to localise
- Read failing traces; find first failure
- Prioritise by volume x impact; add cases to evals
This is a debugging question disguised as an evals question. Show a calm, ordered process.
1. Is the drop real?
- Did the measurement change? New judge model or prompt, rubric edit, logging or sampling change, a new feedback widget. Re-score last week's traces with the current grader: if old traffic now scores lower too, the instrument moved, not the agent.
- Check volume: a drop on 40 conversations may be noise.
2. When did it start, and what changed then? Line up the onset against four suspects:
- Our changes: deploys, prompt edits, tool schema changes, retrieval index rebuilds.
- Provider model update or a silent alias change.
- Customer-side changes: new policy docs, an API returning a new error, a product launch.
- Traffic mix: a new campaign bringing a new intent, a new language or channel.
3. Slice to localise
- Break the metric by intent, channel (chat vs voice), customer tier, language, tool used, and conversation length. A global drop that is really a 40% drop on 'cancel subscription' changes the plan.
- Split into stages: was the intent right, was retrieval right, was the tool call right, was the final message right?
4. Read the traces
- Pull 30-50 failing conversations from the worst slice and note the first failure in each. Group and count.
- Compare with a sample from before the drop.
5. Prioritise
- Rank by affected volume x cost per failure x ease of fix. A wrong refund is worse than a clumsy greeting.
- Ship a fast mitigation if needed (roll back the prompt, pin the previous model version, route a slice to humans) while the real fix is built.
6. Institutionalise
- Add the failing cases to the golden set and regression suite, run the full suite to catch collateral damage, and set an alert on that slice.
- Close the loop with the customer: what happened, what changed, before/after on their own conversations.
Why sources flag this: a public candidate report says a Sierra take-home asked how you would diagnose a drop in an agent metric, which data you'd pull and how you'd prioritise fixes.
- The provider says nothing changed on their side. How do you prove or rule it out?
- You can only fix one thing this week. How do you choose?
- How would you have caught this before the customer noticed?
- Forward Deployed Engineer Interview Guide 2026: Every Round (prep, 2026-10-16) Reports an OpenAI FDE system-design prompt asking what evaluation would catch failures before customers see them, and a Sierra take-home on diagnosing a drop in an agent metric.
- Forward Deployed Engineer Interview Questions (2026, by Round) (prep, 2026-10-16) Lists reported applied-AI design questions by company, including the OpenAI evals question and the Sierra metric-drop question.
- 50 Forward Deployed Engineer Interview Questions and Answers (prep, 2026-07-15) FDE question bank including building evals when the customer has nothing, and the eval harness you run after a prompt change.
- Demystifying evals for AI agents (official, 2026-01-09) Defines code, model and human graders, capability vs regression evals, and pass@k vs pass^k (75% per trial gives about 42% across 3 trials).