"LLM-as-judge: useful, noisy, and how to keep it honest"

A judge score of 8.2 lands in the dashboard and nobody in the room can say what it means. Is 8.2 good? Was last month's 8.5 meaningfully better? Would a human agree with any of it? Teams building AI features with free-text output, summaries, drafts, answers, end up here quickly, because you can't string-compare a summary, so you ask another model to grade it and hope the number is telling the truth.

Let me defend the technique before criticising it. LLM-as-judge is genuinely useful, and for open-ended output it is often the only scoring approach that scales. A human can review thirty outputs a week; a judge can review every output of every eval run. I use judges in most systems I build that produce free text. The mistake isn't using one, it's treating an uncalibrated instrument as if it were a ruler.

Calibration is the unskippable step. Take fifty to a hundred real outputs, have a human label each one pass or fail against your quality bar, then run the judge over the same set and measure agreement. Before you judge the judge, know your ceiling: have two humans label a subset independently, because the judge cannot meaningfully agree with you more than you agree with each other, the study that put LLM-as-judge on the map, Zheng et al.'s MT-Bench work in 2023, found strong judges matching human preferences a little over eighty per cent of the time, which was roughly the rate at which the humans matched one another. On a narrow task with a clear rubric you should do better; at seventy per cent you are mostly measuring noise. Just as important: read the disagreements one by one. The pattern, the judge forgives factual errors, or punishes short answers, or rewards confident tone, tells you exactly what your judge cannot see.

Rubric design moves that agreement number more than model choice does. Vague instructions produce vague grades: "rate the quality from 1 to 10" invites the judge to invent its own standard on every call, and humans can't even agree with each other on what a 7 means, which is why practitioner guidance has settled firmly on binary pass/fail. Decompose instead into narrow yes-or-no questions. Does the summary contain any claim not present in the source? Does the reply answer the question that was actually asked? Is anything from the required-points list missing? The judge answers each of those far more consistently than a global score, and each failure is directly actionable.

Judges also carry biases that have been documented in the research literature since that same 2023 study. In pairwise comparisons they favour whichever answer appears first, so run both orderings and reconcile. They prefer longer, more confident answers regardless of correctness. They tend to rate output from their own model family more kindly. None of this makes judges useless; it makes them instruments with known error characteristics, like any sensor.

Then there is drift. Your judge is a model behind an API, and when the provider updates it, its grading standard shifts, scores move even though your system didn't change. Pin the judge's model version explicitly. When you're eventually forced to migrate the judge, re-run the calibration against the same human-labelled examples before trusting a single new score. It's the same discipline as re-baselining a lab instrument after a service.

And frequently the honest move is not to use a judge at all. Most "unstructured" outputs have structured parts hiding in them. Length limits, required fields mentioned, banned phrases absent, every citation resolving to a real source, every number matching the source document, all of these are deterministic asserts, cheap and unambiguous. Layer them first, and let the judge handle only the residue that genuinely needs judgement. In several systems I've worked on, the exact-check layer caught most of the regressions and the judge caught almost none.

Finally, be careful what you let a judge gate. A CI build that fails because a judge score dropped from 0.86 to 0.84 will train your team to ignore CI, because that difference is inside the noise. Gate releases on exact checks and human-labelled cases; treat judge scores as a trend to watch across weeks, and investigate when the trend moves, not the point.

The overhead is real but bounded: building the human-labelled calibration set is a day of someone's time, and re-calibration is an hour whenever the judge changes. Compared to months of decisions steered by a number nobody validated, that's cheap.

Calibrating judges, layering exact checks, and building eval pipelines that stay honest is the work I do. If your dashboard has a judge score nobody quite trusts, get in touch via the contact form.