Can you use an LLM to grade your outputs? How do you make sure the judge is trustworthy?
Answer it out loud first, then compare. That's what makes it stick.
Yes, but I treat the judge as a measuring instrument that needs calibration. First I use code for anything checkable. For the rest I write one narrow pass/fail judge per criterion, have experts label 100 to 200 examples, tune the judge prompt on a dev split and test it on a held-out split. I report agreement with Cohen's kappa and recall on failures. I pin the judge model and prompt, and re-check agreement whenever either changes.
- Code first; judge only what code can't check
- One binary judge per criterion
- Expert labels, dev/test split
- Measure kappa, precision, recall on fails
- Pin versions; re-calibrate on change
An LLM judge is a classifier you built. Like any classifier, it needs a labelled test set before you trust its numbers.
Decide if you need one
- If code can check it, use code: valid JSON, exact field match, quote exists in the source, test passes. It is cheaper, deterministic and has no bias. Anthropic's docs rank code grading first and say to test LLM grading for reliability before scaling it.
- Use a judge for tone, groundedness, policy adherence, helpfulness.
Build it
- One judge per criterion. A single 'god judge' scoring faithfulness, tone and completeness at once is hard to align and impossible to debug.
- Binary pass/fail with a short written definition and 2-3 pass and fail examples. 1-10 scales drift and humans disagree on them.
- Ask for a short rationale before the verdict, and give it the source context it needs (retrieved passages, policy text).
Calibrate it
- Have domain experts label 100-200 examples per criterion, with a healthy share of real failures (a set that is 98% passes tells you nothing about recall).
- Split ~75/25: iterate the prompt on dev, report on test, so you do not overfit the judge to the labels.
- Report Cohen's kappa (chance-corrected), plus recall on the fail class (does it catch defects?) and precision (does it cry wolf?). Compare to human-human agreement: that is the ceiling, not 100%.
- Read every disagreement. Often the rubric was ambiguous, not the judge.
Keep it honest
- Pin judge model version, prompt and rubric; version them like code.
- Re-run calibration when the judge model changes; a judge upgrade alone can shift pass rates.
- Keep a weekly human sample in production; Google's guidance is to evaluate the autorater against human ratings, and OpenAI's is to have experts audit graders regularly.
Where not to rely on it: when its error rate is larger than the effect you are measuring, or for single high-stakes decisions (clinical, financial), where it can triage but a person decides.
- What kappa would you accept before using the judge?
- Should the judge be the same model as the system under test?
- The judge model is deprecated next quarter. What happens to your metrics?
- FDE Interview Questions and Enterprise AI Scenarios (prep, 2026-08-05) FDE question list including how to build a golden dataset, whether an LLM can judge another LLM, and what your release threshold is.
- FDE AI / LLM Application Interview: Evals, RAG, Agents (prep, unknown) Describes 'Can you use a model to grade the outputs?' as a common follow-up in the FDE AI application round.
- 50 Forward Deployed Engineer Interview Questions and Answers (prep, 2026-07-15) FDE question bank including building evals when the customer has nothing, and the eval harness you run after a prompt change.
- Product Evals in Three Simple Steps (prep, 2025-11-23) Label binary pass/fail data, align one judge per dimension on a dev/test split, swap order for pairwise, score judges with precision, recall and Cohen's kappa.
- Define success criteria and build evaluations - Claude Platform Docs (docs, unknown) Official guidance: specific, measurable success criteria; task-specific evals with edge cases; prefer code grading, then LLM grading tested for reliability, human grading last.
- How to evaluate your gen AI at every stage (official, 2025-06-13) Recommends evaluating the autorater itself against a human-rated benchmark set before trusting it.