LLM-as-Judge

Using an LLM to evaluate outputs for subjective failures (where code-based checks — assertions, regex, schema — don't apply). Powerful but load-bearing only once calibrated against human judgment: the judge is itself a model that can be wrong, so its verdicts must be validated before they gate anything.

Discipline (from the AI-evals plan, Phase 3):

Prefer binary pass/fail + a written critique over 1–5 numeric scales — sharper, more reliable, and the critique surfaces why.
Use few-shot example critiques in the judge prompt (raises human/LLM agreement ~15–20%).
Validate the judge: partition data, measure generalization to unseen data, track agreement with a human expert, iterate until high (a documented Honeycomb case took 3 iterations to clear 90%). Re-check over time — ongoing, not one-and-done.
Criteria drift: eval criteria only fully crystallize after judging many outputs; treat them as living documents.

Claims

Calibrate the judge before you trust it. principle — durable: an automated evaluator must be validated against human ground truth before its verdicts gate changes. Corroborates P4 in llmops-continual-improvement.
Binary pass/fail + written critique beats 1–5 scales. best practice — context: subjective LLM-judge evals with current (2024–26) models; numeric scales invite false precision and inter-rater noise.
Few-shot example critiques in the judge prompt raise human agreement ~15–20%. best practice — context: prompt-level tuning of a judge, today's models.
Judge criteria only crystallize after judging many outputs (criteria drift). observation — a property of the task, informs treating criteria as living docs.
Use the most powerful model you can afford as the judge; run a mini-eval system for the judge itself. best practice — context: critiquing needs advanced reasoning, and you can afford a slower/stronger model offline than in production; the judge is a meta-problem whose own quality must be tracked.
Don't gate on raw human/judge agreement when classes are imbalanced — measure precision and recall separately. best practice — context: validating a judge on skewed data (e.g. failures are rare); raw agreement is misleading when one class dominates, so a high-accuracy-looking judge can be useless.
Route the verdict to a different agent or model than the one that produced the work — a separate sub-agent or a foreign model (e.g. a Codex plugin) gives a less-biased opinion than self-grading. best practice — context: in-loop verification of an agent's own output; independence reduces the same-model bias that makes a producer a poor judge of itself. A practitioner source (Austin Marchese) reaches for sub-agents / cross-model review inside loops; note he treats a 1–10 score as interchangeable with binary — prefer binary pass/fail + written critique per the claim above.
Judges carry structured biases — self-bias (preferring their own model's output), position bias (favoring the first answer compared), and verbosity bias (favoring longer answers) — and are non-deterministic across runs. observation — from the AI Engineering textbook survey; independently names the failure modes the "route to a different model" and binary-critique disciplines above are mitigating. Mitigate position bias by randomizing response order (at added cost).
Judges do better at classification than at numerical scoring, since models handle text better than numbers. best practice — context: choosing a judge output format; a textbook-source reason to prefer categorical verdicts over 1–5/1–10 scales, converging on the binary-pass/fail claim above.

Error Analysis — supplies the subjective failure modes a judge is built to score.
Eval-Driven Development — the judge is one gate in the eval flywheel.
Levels of Evaluation — the judge is the automated half of Level 2.
Agentic Workflow Patterns — the evaluator-optimizer "evaluator" call and parallelization's "voting" variant are LLM-as-judge applied online (feedback loops and vote thresholds), not just offline gating.
Distillate: AI Evals — Self-Study Plan (Hamel & Shreya).
Distillate: Your AI Product Needs Evals.
Distillate: Building Effective Agents.
Distillate: Loop Engineering, Illustrated: Triggers, Skills, Verification, Memory — independent-agent verification inside a loop (sub-agents / Codex plugin) for a less-biased verdict.
Distillate: AI Engineering in 76 Minutes — Chip Huyen's Book, Speedrun — textbook corroboration plus the self/position/verbosity bias taxonomy and classification-over-scoring guidance.
External: Who Validates the Validators? (arXiv 2404.12272); Eugene Yan's AlignEval.

LLM-as-Judge

Claims

Related

Linked from