Your eval score went from 82% to 85% on 300 examples. Is that a real improvement?
Answer it out loud first, then compare. That's what makes it stick.
Probably not on its own. On 300 items, a single score near 83% has a 95% interval of about plus or minus 4 points, so two independent scores 3 points apart overlap. Because both versions ran on the same items, I'd do a paired test on the flips: if 15 items improved and 6 got worse, McNemar says that's borderline. I'd also cluster errors if items share documents, rerun for noise, and check whether the gain is concentrated in one category.
- Compute the standard error and 95% CI
- Use a paired test on the same items
- Cluster errors for related items
- Account for run-to-run noise
- Size the set for the effect you need
Interviewers use this to see whether you treat evals as measurements with error bars or as scoreboards.
Step 1: the naive interval
- Standard error of a proportion = sqrt(p(1-p)/n). At p = 0.83, n = 300: sqrt(0.83 x 0.17 / 300) is about 0.022.
- 95% CI = plus or minus 1.96 x SE, about plus or minus 4.3 points. Anthropic's statistics paper recommends reporting exactly this alongside every score.
- Treating the two runs as independent, the SE of the difference is about 0.031, so a 3-point gap is well inside noise.
Step 2: use the pairing
- Both versions saw the same 300 items, so compare flips. Suppose 15 items went fail-to-pass and 6 went pass-to-fail (net +9 = 3 points).
- McNemar's test only uses the 21 discordant items: chi-square with continuity correction = (|15-6|-1)^2 / 21 = 64/21, about 3.0, p about 0.08. Suggestive, not conclusive.
- Paired analysis is much more sensitive than comparing two averages because shared easy and hard items cancel out.
Step 3: clustering
- If several questions come from the same document or conversation, they are not independent. Anthropic found clustered standard errors on popular evals can be over 3x the naive ones. Cluster by the unit you sampled (document, customer, conversation).
Step 4: noise in the system and the judge
- Rerun each version 3 times. If the same config varies by 2 points run to run, a 3-point gain is meaningless.
- If an LLM judge grades, its disagreement with humans adds error too.
Step 5: size the set for the decision
- To detect a 3-point change on individual proportions you need well over 1,000 items; paired designs and focusing on the relevant slice need far fewer.
- Practical rule to state: on a few hundred items, only trust changes of roughly 5+ points, or paired results with clear flip asymmetry.
What to tell the customer: 'Directionally better, not yet proven. Here are the 15 cases it fixed and the 6 it broke; I'll confirm on the holdout and a week of shadow traffic.'
- How many examples would you need to detect a 2-point change?
- How would you build a confidence interval for an LLM-judge score?
- The CEO wants one number. What do you show?
- A statistical approach to model evaluations (official, 2024-11-19) Recommends reporting standard errors and 95% CIs (mean plus or minus 1.96 SEM), clustering errors for related questions, and paired comparisons between models.
- Real Interview Questions for the Forward Deployed Engineer at OpenAI (prep, 2026-11-15) Lists 'How do you know your AI system is actually working well?' as reported in the OpenAI FDE process, with a layered offline/online/human-review answer.
- LLM Evals for Customer Deployments (prep, 2026-10-12) FDE course lesson framing 'we changed the prompt, is it worse?' as a deep-dive interview question; covers judge calibration with kappa and item-by-item release gates.