The customer has no labeled data and no eval set. How do you build one in the first two weeks?
Answer it out loud first, then compare. That's what makes it stick.
I mine what they already have: tickets with resolutions, documents a human already extracted, emails they actually sent. Those historical pairs are my seed set, maybe 50 to 100 real cases. I stratify by task type, add hard and adversarial cases on purpose, and run a labelling hour with two of their experts. Where they disagree, a named owner decides and the reason becomes a rubric line. I use synthetic inputs only to fill gaps, and freeze a holdout I never tune on.
- Mine historical input-output pairs
- Stratify and add edge/adversarial cases
- Expert labelling session; resolve disagreements
- Fill gaps with structured synthetic inputs
- Freeze a holdout, version everything
Almost every enterprise workflow you automate has been done by hand for years. That history is your first eval set.
Week 1: real data first
- Pull 50-150 historical cases from logs, tickets, CRM, past documents, with what a human eventually did. 50 real cases beat 5,000 invented ones because real data carries the mess: typos, missing fields, mixed languages.
- Stratify to match the real mix (by intent, document type, channel), then deliberately add the hard tail: ambiguous requests, missing data, out-of-scope asks the system should refuse, and anything rare but expensive.
- Get data access sorted early: de-identify PII, agree retention, and record provenance per row.
Labelling session
- One or two hours with two senior operators grading outputs on 20-30 cases with a draft rubric.
- Their disagreements are the most valuable output: they surface unwritten rules ('we never quote refund timelines in writing') that become both prompt instructions and rubric lines.
- Appoint one decider (Hamel Husain calls it a benevolent dictator) so the rubric does not stall in committee.
- Expect criteria drift: research on EvalGen found people refine criteria while grading, so plan a second pass.
Week 2: fill gaps with synthetic inputs
- Define dimensions (feature x scenario x persona), write ~20 tuples by hand, then have an LLM expand them and turn each into a natural query.
- Generate inputs, not expected outputs, so the set does not inherit the generator's blind spots. Experts still write or approve the answers.
- Check each synthetic case triggers the scenario it claims ('no matches' really returns zero results).
Structure the set
- Split: a dev slice you iterate on and a frozen holdout (20-30%) you only touch at release.
- Store as JSONL in git or an eval tool with id, input, expected output or rubric, tags, source, and version.
What to tell the customer: a usable first number in two weeks, a trustworthy one after ~300 labelled cases, and the set grows from every production failure.
- Two experts disagree on 30% of cases. What do you do?
- When is synthetic data unreliable?
- How do you handle PII in the eval set?
- 50 Forward Deployed Engineer Interview Questions and Answers (prep, 2026-07-15) FDE question bank including building evals when the customer has nothing, and the eval harness you run after a prompt change.
- LLM evaluation interview: 'how do you know it works?' (prep, 2026-09-25) FDE prep guide on the 'how do you know it works' question, eval sets without labels, and post-launch monitoring signals.
- FDE Interview Questions and Enterprise AI Scenarios (prep, 2026-08-05) FDE question list including how to build a golden dataset, whether an LLM can judge another LLM, and what your release threshold is.
- AI Evals: Everything You Need to Know (prep, 2026-09-18) Error analysis via open and axial coding, about 100 diverse traces, 100-200 labels per failure mode for a judge, structured synthetic data, and retrieval metrics like Recall@k and MRR.
- A Field Guide to Rapidly Improving AI Products (prep, 2025-03-24) Synthetic test data from features, scenarios and personas; generate inputs not outputs; verify scenarios actually trigger.
- Who Validates the Validators? (UIST 2024) (docs, 2024-10-11) Identifies criteria drift: people refine their grading criteria while grading outputs, so criteria cannot be fully fixed in advance.