Your eval scores keep going up, but the customer says the product isn't getting better. Which do you trust, and what do you do?
Answer it out loud first, then compare. That's what makes it stick.
I trust the customer's outcome and treat the gap as a bug in my eval. Usually the set has drifted from real traffic, the judge rewards the wrong thing, or we optimised a proxy that doesn't drive their business metric. I'd pull recent real conversations, have their experts grade a sample, and compare with my scores. Then I fix the eval, and if the gap is the model's fault, I package examples and frequency for the provider's product team.
- Assume the eval is wrong first
- Check drift: set vs live traffic
- Re-grade a sample with customer experts
- Re-link proxy metrics to the business KPI
- Route model gaps to product/research with evidence
An eval is a proxy for the customer's outcome. When the two disagree, the proxy loses until proven otherwise.
Likely causes
- Distribution drift. The golden set reflects month one; users now ask different things. Compare tag mix in the set vs the last 30 days of traffic.
- Overfitting to the set. Dozens of prompt iterations against the same cases. Check the frozen holdout and a fresh sample: if those don't move, the gains were memorisation.
- Judge misalignment. The judge rewards length, confident tone or format. Re-measure judge-human agreement on recent outputs; a judge or rubric change can move pass rates by several points with no system change.
- Wrong proxy. You optimised 'answer correctness' but the customer's pain is latency, too many clarifying questions, or answers agents have to rewrite.
- Workflow gap. The model is fine, but the output lands in the wrong place, needs too many clicks, or users don't trust it.
What to do
- Pull 50-100 recent real interactions, stratified, and have the customer's experts grade them blind with your rubric.
- Compare their grades with your judge and your eval score on the same items. Read the disagreements.
- Interview 3-5 end users: what made them edit, ignore or escalate?
- Fix the eval: add new categories, refresh the set, recalibrate the judge, and add the online signal that matches their pain (edit distance on drafts, time to resolve).
- Re-agree the success metric and its link to the business KPI, in writing.
Feeding the provider
- If the gap is a model limitation, not your integration, package it: 20-50 examples, frequency in production, business cost, and a minimal reproduction. OpenAI's FDE role explicitly measures success partly by eval-driven feedback that changes product and model roadmaps; vague 'the model is bad at X' feedback goes nowhere.
Say this out loud: the customer's experience is the ground truth; my eval is a tool I owe them to keep accurate.
- How would you package a model failure so the provider's research team acts on it?
- What online metric would have caught this earlier?
- The customer's feedback comes from one loud manager. How do you weigh it?
- OpenAI Forward Deployed Engineer Interview (2026) (prep, 2026-10-25) Lists an OpenAI FDE question on using evals to show a deployment works and to feed product/model teams, with a follow-up on eval vs customer metric disagreement.
- Forward Deployed Engineer (FDE) - SF (OpenAI careers) (official, unknown) Role description: success measured through adoption, workflow impact and eval-driven feedback that changes product and model roadmaps.
- Real Interview Questions for the Forward Deployed Engineer at OpenAI (prep, 2026-11-15) Lists 'How do you know your AI system is actually working well?' as reported in the OpenAI FDE process, with a layered offline/online/human-review answer.
- An LLM-as-Judge Won't Save The Product - Fixing Your Process Will (prep, 2025-04-20) Evals as the scientific method: look at data, annotate, hypothesize, experiment against a baseline; keep human sampling even with automated judges.