Some questions about an AI’s output can’t be checked by a simple rule. Did the answer actually solve the customer’s problem, not just technically respond to it? Did the agent’s reasoning stay coherent across ten steps? LLM-as-a-judge answers these by giving a second model the full transcript, a scoring rubric, and instructions to explain its reasoning before giving a numeric grade — the same way a human reviewer would, but at a scale no human review team can match.
The technique is only as reliable as the judge itself. Judge models carry their own biases, favouring longer or more confident-sounding answers over correct ones, and can drift as the underlying model updates. Teams that rely on it without checking should calibrate the judge’s scores against real human graders periodically, and treat a judge that consistently disagrees with human reviewers as broken, not as the ground truth.
The discipline that separates a trustworthy judge setup from a rubber stamp is temperature set to zero and a rubric with worked examples. A judge asked to freely decide “is this good” without either will produce scores that look precise and mean very little.