Exact match, code checks, LLM judges, human review: how do you decide which grader to use for what?
Answer it out loud first, then compare. That's what makes it stick.
I pick the cheapest grader that's reliable for that criterion. If there's one right answer, like a label or an extracted amount, exact or normalised match. If a rule can decide it, like valid JSON, a citation that exists, or a tool called in order, a code check. If it needs judgment, like tone or whether a claim is supported, a calibrated LLM judge. Humans define the rubric, calibrate the judges, and make the call on high-stakes cases.
- One right answer: exact/normalised match
- Rule-checkable: code assertion
- Needs judgment: calibrated LLM judge
- Humans: rubric, calibration, high stakes
- Combine per task: all-must-pass or weighted
The principle from Anthropic's docs: choose the fastest, most reliable, most scalable grader that can actually judge the criterion. Each step down the list adds cost and noise.
1. Exact or normalised match
- Classification labels, extracted fields, yes/no answers. Normalise first (case, whitespace, date formats, currency), or you under-count correct answers.
- OpenAI's Evals API has string-check graders for exactly this.
2. Code assertions
- Schema validity, required fields present, numbers within range, quoted text exists in the source, URL resolves, SQL executes and returns the expected rows, tool called with valid arguments, required disclaimer present.
- Cheap, deterministic, no bias. Weakness: brittle to valid variations (Anthropic's agent guide calls this out).
- Older text-overlap metrics (BLEU, ROUGE) are a weak fit for most product tasks; OpenAI's guidance lists relying only on such generic metrics as an anti-pattern.
3. LLM judge
- Tone, policy adherence, groundedness, completeness, helpfulness.
- Binary per criterion, with reference material, calibrated against expert labels. Variants: pointwise rubric, pairwise comparison, reference-guided. Google's evaluation service generates per-prompt rubrics with pass/fail verdicts, a useful pattern when every input is different.
4. Human review
- Defines the rubric, labels calibration sets, audits a weekly sample, and handles anything where one mistake is expensive (medical, legal, money movement).
- Slowest and costliest: roughly a few minutes and a dollar or more per item for expert review, versus about a cent per judge call.
Combining graders
- Per task, either all must pass (guardrails) or a weighted score above a threshold. Anthropic's agent guide describes binary, weighted and hybrid scoring.
- Run cheap graders first and skip the judge when code already failed the case.
Example: invoice extraction. Exact match on totals and dates, code check that line items sum to the total, a judge only for the free-text 'reason for dispute' field, and a human for any invoice above a value threshold.
- Your exact-match grader says 70% but manual review says 90%. What's wrong?
- When would you pay for expert human grading at scale?
- How do you grade SQL generation?
- Define success criteria and build evaluations - Claude Platform Docs (docs, unknown) Official guidance: specific, measurable success criteria; task-specific evals with edge cases; prefer code grading, then LLM grading tested for reliability, human grading last.
- Demystifying evals for AI agents (official, 2026-01-09) Defines code, model and human graders, capability vs regression evals, and pass@k vs pass^k (75% per trial gives about 42% across 3 trials).
- Working with evals (OpenAI API docs) (docs, unknown) Shows the Evals API flow: describe the task, run with test inputs, analyze and iterate, with graders such as string_check against a labeled answer.
- Gen AI evaluation service overview (Google Cloud docs) (docs, 2026-11-12) Describes rubric-based metrics and adaptive rubrics: per-prompt pass/fail tests generated, then each response checked with a rationale.
- LLM-as-a-judge pitfalls to name in an FDE interview (prep, 2026-09-25) Lists judge pitfalls (position, verbosity, self-preference, rubric drift, noise) and checks such as order swaps and kappa against human labels.