Your AI judge needs a disagreement log
An automated judge that agrees with a human-reviewed reference 85 times in 100 can still accept half of the answers the reference rejects, so look at the disagreements, not just the average.
Using an automated judge makes a large evaluation practical, but its pass count shouldn't be trusted for comparing models until it has been checked against a reviewed reference set. The grader is one more part of the test, with its own configuration and its own ways of going wrong.
Anthropic's evaluation guide, read on 14 November 2026, describes several grading methods and recommends testing LLM-grading reliability before scaling. Our proposed starting point is a disagreement log: one row per case, holding the answer, the rubric, the reference decision and the judge decision.
Where do the judge and the reference part ways?
The table uses one hundred invented answers. A reviewed reference accepts ninety and rejects ten. The automated judge accepts eighty-five and rejects fifteen. Those two totals alone would hide which answers changed category.
| Reviewed reference | Automated judge | Invented cases |
|---|---|---|
| Accept | Accept | 80 |
| Accept | Reject | 10 |
| Reject | Accept | 5 |
| Reject | Reject | 5 |
Eighty answers are accepted by both and five rejected by both, which gives 85% agreement. The fifteen disagreements split into ten reference passes rejected by the judge and five reference failures accepted by it. Of the ten reference failures, the judge accepts half.
That does not make the reference infallible, only worth comparing against. The table describes a chosen trial standard. It is not an independent measurement of truth or a result for a real judge.
A rubric the judge can actually check
A good rubric (the written marking scheme the judge applies) states what earns a pass and which failures override otherwise good work. If an answer must cite supplied evidence, the judge needs that evidence and a definition of what counts as support. Asking whether an answer is “good” leaves too much of the standard unsaid.
Exact conditions, such as a required field or an unambiguous value, are often better handled by ordinary code, with judgement kept for the conditions that need it. Saving the intermediate checks lets a final label be explained afterwards.
The judge's model identifier, prompt, settings and rubric version also need recording. A new grader release or a revised instruction can change the score while the candidate answers stay exactly the same, so a fixed reference set should be re-scored whenever the judging setup changes.
Reading the disagreements before changing anything
Each disagreement ends in one of three decisions: the reference label, the rubric or the judge needs correcting. The reason should be saved. A changed reference label should leave the earlier version in place, so a previous model comparison can still be interpreted.
Where they fit the task, the reference set should also hold difficult but legitimate answers, concise correct answers and polished wrong answers. That coverage plan is our proposal. We have not measured whether any judge favours a particular style or length.
Some reviewed cases should be held back while the rubric is improved. A judge tuned repeatedly against the same examples may agree well on them without showing that the repair generalises, so tuning and held-back results belong in separate columns.
Why the smaller group needs its own numbers
The invented example has many reference passes and few reference failures, so overall agreement is dominated by the larger group. When either kind of error affects the decision, false accepts and false rejects each need their own denominators.
Suppose the release policy treats an unsupported critical answer as a blocker. That category then gets inspected explicitly, and a high aggregate agreement rate does not waive the condition. The judge helps apply the release rule and should not quietly rewrite it.
Your own results log should say which judge scored each run and how it agreed with the reviewed labels. When a model's score changes, the disagreement log gives the next reviewer something concrete to inspect.