What biases do LLM judges have, and how would you detect and control each one?
Answer it out loud first, then compare. That's what makes it stick.
The big ones are position bias in pairwise comparisons, verbosity bias toward longer answers, self-preference for the judge's own model family, sensitivity to small prompt changes, and run-to-run noise. I test for each directly: swap answer order, pad a good answer with fluff, compare judge-versus-human agreement per model family, reword the prompt, and judge the same set twice. Then I mitigate: run pairs both ways and call flips a tie, cap length effects, and use a judge from a different family.
- Position: swap order, flips = tie
- Verbosity: padded-answer probe
- Self-preference: cross-family judge
- Prompt sensitivity and noise: rerun and reword
- Track all with a human-labelled holdout
The classic evidence is the 2023 MT-Bench study (Zheng et al.). Model versions have moved on, but the bias types still show up and interviewers expect you to name and test them.
Position bias
- In pairwise comparisons judges favour a slot. In that study one judge preferred the first answer about 70% of the time when content was swapped.
- Control: run every pair twice with order swapped. If the verdict flips, record a tie rather than forcing a noisy win. Report the flip rate itself as a health metric.
Verbosity bias
- Judges reward longer answers. When answers were padded with rephrased, no-new-information text, two judges preferred the longer one over 90% of the time.
- Detect: a padding probe, the same correct answer with and without filler. Control: rubric lines that penalise unneeded length, and pointwise binary checks instead of 'which is better'.
Self-preference (self-enhancement)
- Judges rate their own outputs higher (the study saw roughly +10% and +25% win-rate inflation for two models).
- Control: use a judge from a different model family than the system under test, or measure judge-human agreement separately on each family's outputs.
Prompt sensitivity and rubric drift
- Small wording changes move scores. Control: pin model, prompt and rubric; re-run the human-labelled golden set on any judge change.
Run-to-run noise
- Verdicts change even at temperature zero. Detect: judge the same set twice and report the disagreement rate. Control: majority vote of 3 on borderline items, or treat differences smaller than the noise as no change.
Leniency and reference leakage
- Judges are often too lenient on confident wrong answers. Give them the source or reference answer and ask claim-by-claim questions.
Tie it together: all of these are caught by one discipline, a human-labelled holdout you re-score whenever anything about the judge changes. Eugene Yan's summary of the same study is a good reference: judge-human agreement reached about 85%, above the 81% human-human figure, which is why judges are useful after calibration.
- How do you report a pairwise win rate when 20% of pairs flip on swap?
- Would multi-judge ensembles fix these biases?
- How do you check self-preference if the customer only allows one model vendor?
- Evaluating the Effectiveness of LLM-Evaluators (aka LLM-as-Judge) (prep, 2024-08-18) Summarizes MT-Bench findings: about 85% judge-human agreement vs 81% human-human, plus position, verbosity and self-enhancement biases.
- Product Evals in Three Simple Steps (prep, 2025-11-23) Label binary pass/fail data, align one judge per dimension on a dev/test split, swap order for pairwise, score judges with precision, recall and Cohen's kappa.
- LLM-as-a-judge pitfalls to name in an FDE interview (prep, 2026-09-25) Lists judge pitfalls (position, verbosity, self-preference, rubric drift, noise) and checks such as order swaps and kappa against human labels.
- Evaluation best practices (OpenAI API docs) (docs, unknown) Lists eval anti-patterns (generic metrics, biased datasets, vibe-based evals, ignoring human feedback) and recommends continuous evaluation on every change; notes judge position and verbosity bias.