You've built an AI feature for a customer. How do you know it's actually working?
Answer it out loud first, then compare. That's what makes it stick.
I don't trust demos, so I define 'working' with the customer first: a short list of pass/fail criteria tied to their workflow. Then I measure in three layers. Offline, a golden set of real cases scored by code where possible and a calibrated LLM judge where not. Online, signals like human override rate, escalations and cost per task. And a weekly human review sample that also keeps the judge honest. If the layers disagree, I trust the human and business signals and fix the eval.
- Define 'working' with the customer as pass/fail criteria
- Offline: golden set + graders, release gate
- Online: override rate, escalations, latency, cost
- Human review sample every week
- Reconcile when the layers disagree
'Working' is not a property of the model. It is a property of the deployed workflow measured against criteria the customer agreed to. A strong answer gives a measurement plan before it touches prompts or model choice.
1. Write the contract. Sit with the customer's domain experts and turn 'good' into 4-8 binary checks, for example 'every allergy in the note appears in the summary' or 'never promises a refund date'. Anthropic's docs ask for criteria that are specific and measurable; 'good performance' is not a criterion.
2. Offline layer. A versioned golden set of 200-400 real, de-identified cases, stratified by intent, plus deliberately nasty edge cases. Grade with:
- Code for anything checkable: schema valid, field exact-match, citation exists in source.
- An LLM judge per criterion for things code cannot check, only after it agrees with expert labels.
- A release gate: a change that drops any guardrail check does not ship.
3. Online layer. Offline scores describe the day you measured. In production track:
- Human override / edit rate (the most honest free label)
- Escalation or hand-off rate, re-queue rate
- Schema-valid rate, tool-call error rate
- p95 latency and cost per resolved task
4. Human review. A stratified sample (say 50-100 traces a week) graded by the customer's reviewers with the same rubric. This measures quality and recalibrates the judge.
5. Reconcile. If offline pass rate rises but override rate is flat, the grader is wrong, not the system. If both look fine but the business KPI (handle time, resolution rate) does not move, you built the wrong thing.
Numbers to quote. A 92% pass rate on 200 cases has a 95% interval of roughly plus or minus 4 points, so do not claim a 2-point win on that set. Report results per category, not just one average.
What to say at the end: the eval set is a deliverable. You hand it to the customer so they can re-test after every model or prompt change without you.
- The customer has no labeled data. Where do your correct answers come from?
- Which single online metric would you put on the exec dashboard, and why?
- How would you know your LLM judge is wrong?
- Real Interview Questions for the Forward Deployed Engineer at OpenAI (prep, 2026-11-15) Lists 'How do you know your AI system is actually working well?' as reported in the OpenAI FDE process, with a layered offline/online/human-review answer.
- Forward Deployed Engineer Interview Guide 2026: Every Round (prep, 2026-10-16) Reports an OpenAI FDE system-design prompt asking what evaluation would catch failures before customers see them, and a Sierra take-home on diagnosing a drop in an agent metric.
- LLM evaluation interview: 'how do you know it works?' (prep, 2026-09-25) FDE prep guide on the 'how do you know it works' question, eval sets without labels, and post-launch monitoring signals.
- Define success criteria and build evaluations - Claude Platform Docs (docs, unknown) Official guidance: specific, measurable success criteria; task-specific evals with edge cases; prefer code grading, then LLM grading tested for reliability, human grading last.