Evals & measuring LLM qualityReported 2×mediumChecked 2026-11-17
You changed the system prompt and the customer asks, 'Did it get better?' Walk me through how you answer.
Answer it out loud first, then compare. That's what makes it stick.
30-second answer
I run the old and new versions on the same frozen eval set and compare item by item, not just the average. Deterministic checks first, then calibrated judges, broken down per category. I look at what flipped from pass to fail, not only the net gain, because a plus three overall can hide a new failure in a critical category. I check whether the difference is bigger than noise, then show the customer the before-and-after on their own examples.
Framework to remember
- Same frozen set, both versions
- Code checks, then calibrated judges
- Per-category, item-level diff
- Inspect pass-to-fail flips
- Significance check, then show examples
Deep answer
This is a regression-testing question. The answer is a harness that makes 'did it get better' a 10-minute command, not a debate.
The harness
- A versioned golden set (dev + frozen holdout), each case tagged by category and risk.
- A runner that executes the candidate config (prompt, model, retrieval settings, temperature) on every case, ideally 2-3 times for non-deterministic flows.
- Graders in order: code checks (schema, exact fields, citation exists), then one calibrated judge per criterion, then latency and cost per case.
- Output: a table of baseline vs candidate per category, plus the list of changed items.
Reading the result
- Item-level diff. Count pass-to-fail and fail-to-pass separately. A net +3 points made of +8 fixes and -5 new breaks is a different story, especially if the breaks are in refunds or compliance.
- Per category. Averages hide regressions in small but critical slices.
- Noise check. Because both versions ran on the same items, use a paired test (McNemar on flips, or a paired bootstrap). Anthropic's statistics guidance recommends paired comparisons for exactly this reason: they cut variance versus comparing two independent averages.
- Cost and latency. A longer prompt might win 2 points and add 30% tokens.
Guardrail vs north-star
- Some checks are ship-blockers (no PII leak, required disclaimer present): any regression blocks.
- Others are aspirational (helpfulness): must not drop beyond a pre-agreed margin.
Make it continuous
- Wire the harness into CI: OpenAI's guide calls this continuous evaluation, running evals on every change. Tools like OpenAI Evals, Braintrust, LangSmith or promptfoo all support this; the tool matters less than the frozen set and the item diff.
Close the loop with the customer
- Show 3-5 concrete before/after examples from their own data, the per-category table, and anything that got worse. Honesty about the regressions is what earns trust for the next change.
Strong answer
Describes a repeatable harness, paired item-level comparison on a frozen set, per-category breakdown, explicit pass-to-fail review and a significance check.Weak answer
Tries a handful of prompts by hand, eyeballs outputs, and reports 'it looks better'.The trap
Comparing only the aggregate score; a small net gain often hides new failures in the categories the customer cares most about.Likely follow-ups
- The new prompt wins overall but breaks two compliance cases. Do you ship?
- How many runs per case do you need for a non-deterministic agent?
- How do you stop people from tuning the prompt to the eval set?
Sources (4) · confidence high
- 50 Forward Deployed Engineer Interview Questions and Answers (prep, 2026-07-15) FDE question bank including building evals when the customer has nothing, and the eval harness you run after a prompt change.
- LLM Evals for Customer Deployments (prep, 2026-10-12) FDE course lesson framing 'we changed the prompt, is it worse?' as a deep-dive interview question; covers judge calibration with kappa and item-by-item release gates.
- Evaluation best practices (OpenAI API docs) (docs, unknown) Lists eval anti-patterns (generic metrics, biased datasets, vibe-based evals, ignoring human feedback) and recommends continuous evaluation on every change; notes judge position and verbosity bias.
- A statistical approach to model evaluations (official, 2024-11-19) Recommends reporting standard errors and 95% CIs (mean plus or minus 1.96 SEM), clustering errors for related questions, and paired comparisons between models.