What is a golden set, and how do you keep it useful over the life of a deployment?
Answer it out loud first, then compare. That's what makes it stick.
A golden set is a versioned collection of representative inputs with expert-approved expected outputs or rubrics, used to gate every change. Keeping it useful is the hard part. I keep a frozen holdout nobody tunes against, add every production failure as a new case, retire cases that no longer reflect reality, re-sample from traffic each quarter, and record provenance and version so scores stay comparable. Mature capability cases graduate into a regression suite that should pass near 100%.
- Representative + edge cases with expert answers
- Version, tag, record provenance
- Frozen holdout vs dev slice
- Grow from production failures; retire stale cases
- Graduate cases into a regression suite
A golden set is a contract in data form: the inputs that matter and what correct looks like, agreed with the customer.
What goes in
- A stratified sample of real traffic, so frequencies resemble production.
- Deliberate edge and adversarial cases: ambiguous asks, missing fields, out-of-scope requests that should be refused, injection attempts, and rare-but-costly scenarios.
- For each case: input, expected output or per-case rubric, tags (category, risk, language), source and date, and who approved it.
How it decays (and the fixes)
- Overfitting by iteration. After 50 prompt tweaks against the same 200 cases, the score measures memorisation of that set. Fix: a frozen holdout (20-30%) only run at release time, refreshed periodically.
- Distribution drift. New products, policies and user behaviours appear. Fix: quarterly re-sampling from production and comparing the tag mix against live traffic.
- Stale answers. Policies change, so yesterday's correct refund answer is now wrong. Fix: an owner on the customer side reviews expected answers on a schedule; cases carry a 'valid as of' date.
- Saturation. Once everything passes, it no longer separates versions. Anthropic's agent-evals guide describes capability evals that start with low pass rates, then graduate into a regression suite expected to pass near 100%, while new hard cases form the next capability set.
- Contamination. If cases appear in prompts as few-shot examples or in fine-tuning data, scores inflate. Keep eval data out of training and prompt examples, and track that rule.
Operations
- Store as JSONL in git or an eval platform, with semantic versions. A score is always 'v3.2 of the set', or comparisons over time are meaningless.
- Every production incident becomes a case before the fix ships, so the bug cannot return quietly.
- Track size: a few hundred cases is typical for a first deployment; per-slice counts matter more than the total.
Handoff: the golden set, its rubric and its runner are part of what you leave behind, so the customer can re-validate when the model provider ships a new version.
- How large should the holdout be, and how often do you refresh it?
- A policy change makes 40 expected answers wrong. What is your process?
- How do you stop few-shot examples from leaking into the eval?
- FDE Interview Questions and Enterprise AI Scenarios (prep, 2026-08-05) FDE question list including how to build a golden dataset, whether an LLM can judge another LLM, and what your release threshold is.
- Demystifying evals for AI agents (official, 2026-01-09) Defines code, model and human graders, capability vs regression evals, and pass@k vs pass^k (75% per trial gives about 42% across 3 trials).
- How evals drive the next chapter in AI for businesses (official, 2025-11-19) Describes Specify, Measure, Improve: golden sets from real situations, costly edge cases, LLM graders audited by domain experts, and continuous measurement after launch.
- Evaluation best practices (OpenAI API docs) (docs, unknown) Lists eval anti-patterns (generic metrics, biased datasets, vibe-based evals, ignoring human feedback) and recommends continuous evaluation on every change; notes judge position and verbosity bias.