What's the difference between offline and online evaluation, and what would you monitor after launch?
Answer it out loud first, then compare. That's what makes it stick.
Offline evals run a fixed golden set before release: repeatable, cheap, good for catching regressions, but they only describe the cases you thought of. Online evaluation measures real traffic after launch: user edits and overrides, escalations, thumbs-down, task completion, plus sampled LLM-judge scores and a weekly human review. I monitor input mix for drift, quality signals, latency and cost per task, and every metric gets an owner, a threshold and an action.
- Offline: frozen set, pre-release gate
- Online: real traffic, implicit and explicit feedback
- Sampled judges + human review in production
- Drift, latency, cost
- Owner + threshold + action per metric
Offline evaluation answers 'is this version good enough to ship?' on a fixed dataset.
- Strengths: repeatable, comparable across versions, runs in CI, no user risk.
- Weakness: it only covers inputs you collected. It goes stale as users, documents and models change.
Online evaluation answers 'is it working for real users right now?'
- Strengths: true distribution, real outcomes.
- Weakness: noisy, often no reference answer, and you learn after users are affected.
What to monitor after launch
- Input mix / drift: share of each intent or topic, new clusters, language mix. A new topic means the golden set no longer represents reality.
- Implicit feedback (free labels): human override or edit rate on drafts, re-queues, hand-offs to a person, retries, session abandonment. Usually more reliable than thumbs.
- Explicit feedback: thumbs, flags, CSAT. Sparse and biased toward angry users, but cheap.
- Reference-free automated checks on sampled traces: schema validity, PII leakage, groundedness or policy judges. LangSmith-style online evaluators run on a sampling rate (for example 5-10% of traffic) to control judge cost.
- Human review: a fixed stratified sample each week, graded with the same rubric as the golden set.
- Operations: p95 latency, error and timeout rate, tokens and cost per resolved task.
- Business KPI: the number the customer bought this for (handle time, resolution rate, hours saved).
Wire them together
- Every metric has an owner, threshold and action: 'if refund misroutes exceed 2% over 24 hours, fall back to rules and page the on-call FDE'.
- Production failures flow back into the offline set, so offline coverage grows from real mistakes.
- For risky changes, add stages between the two worlds: shadow mode (new version runs silently), then a canary or A/B with guardrail metrics.
Line to remember: offline evals tell you a change is safe to try; online signals tell you it was worth doing. OpenAI's own guidance on evals says measurement continues after launch with real inputs and user signals.
- Override rate is flat but thumbs-down doubled. What happened?
- How do you evaluate online when there is no reference answer?
- When would you use an A/B test versus shadow mode?
- Forward Deployed Engineer Interview Questions (2026) (prep, 2026-10-22) FDE question list including evaluating a RAG system before launch, defining success metrics, and monitoring an LLM app in production.
- LLM evaluation interview: 'how do you know it works?' (prep, 2026-09-25) FDE prep guide on the 'how do you know it works' question, eval sets without labels, and post-launch monitoring signals.
- How evals drive the next chapter in AI for businesses (official, 2025-11-19) Describes Specify, Measure, Improve: golden sets from real situations, costly edge cases, LLM graders audited by domain experts, and continuous measurement after launch.
- LangSmith Evaluation docs (docs, unknown) Describes online evaluators that run on production traces without reference outputs, with filters and sampling rates to control cost.