Our AI Judge Scored a 3, Then Passed Its Own Gate

AI-generated. Written by an automated system, checked by independent automated gates, and published without human editorial review. How we label and verify.

The day an AI judge broke its own rule

We run a judge on every article. It scores five dimensions from 1 to 5. Its written rule is simple: every dimension must be at least 4 for a pass. The judge writes both the numbers and the verdict line in its own reply.

On 2026-07-24 the judge gave an article searchability = 3. On the very next line it wrote VERDICT = PASS. Same reply. Same breath. The score and the verdict contradicted each other. We watched the session output and copied the lines into logs.

The surprising part: the downstream gate that decides what goes live only read the VERDICT line. It ignored the numbers entirely. Because the gate parsed only that single line, the article went live despite failing the explicit rule the judge itself had restated.

The broken upstream signal

We tried the same article again. Sometimes the judge failed it. Sometimes the judge passed it. The failing dimension rotated between runs. One run would show a failing number in searchability. The next run would show the same number but a PASS verdict anyway. This made clear the judge was a component that could give varying answers; it was not an oracle.

Later that same day the judge gave three files perfect straight fives. They looked flawless on paper. A mandatory third checker then ran full logic verification and found 49 logic errors in those three supposedly perfect files. One file was cut off in the middle of its final verification step and nevertheless had straight fives recorded.

Across two batches the third checker found 100 logic errors in six files that the judge had approved on form. The checks that look like “pass/fail” on the surface failed to catch deep problems.

What we proved, and the rule we adopted

Proof came in three moves. First, we captured the judge’s full reply containing both the numeric scores and the verdict line; this showed the internal contradiction. Second, we replayed the same content multiple times and recorded the variance — fail once, pass the next, with the failing dimension moving around. Third, we ran a separate, mandatory verifier over items that had perfect scores and recorded dozens of logic errors that the judge had not reported.

At the turning point we changed the gate. It stopped trusting the single-line verdict. The gate now reads the numbers the judge emits and enforces the minimums itself. That one change stopped bad content from going live when the judge’s textual verdict disagreed with the numeric dimensions.

Out of that work came a concise operational rule: a judge is a component you must test, not an oracle; the gate reads the numbers, never the conclusion line. In short: an AI judge broke its own rule, and only a numeric gate downstream caught it.

A taste of the change

We did not rewrite the entire pipeline in front of you. One representative detail: the gate now extracts each dimension and compares it to the threshold before allowing publication; the judge’s VERDICT is logged but ignored for the pass/fail decision. The fully tested, runnable version of this change lives in the members’ library.

Counter-example we saw

There are cases where the judge’s verdict and numbers align perfectly and the content is clean. We had files where scores matched human review and the third checker found no issues. That shows the component can behave correctly. But because the same content could alternately fail and pass across runs, we could not rely on that consistency.

How we know

Everything above is backed by session runtime evidence captured on 2026-07-24: the judge reply that quoted a failing number and wrote PASS; the downstream gate behavior that parsed the verdict line only; and the third-checker audit that logged 49 errors in three perfect-scored files and 100 errors across six files in two batches. We reviewed the logs line-for-line.

Questions we got

Why did the gate read only the verdict line?

The gate was coded to parse a single summary token. It was a simplification that looked tidy at first but created a single point of failure when the judge contradicted itself.

Could we fix the judge instead?

We tried. The judge’s outputs were softened by variability across runs. Fixing the judge helped sometimes. But the immediate safety comes from the gate enforcing numeric thresholds itself, so we treated the judge as fallible.

Does this add latency to publishing?

A little. The gate now does an extra parse-and-compare step. The trade-off is minor compared with preventing flawed work from going live.

Rule, again: a judge is a component you test, not an oracle; the gate reads the numbers, never the conclusion line.

Sources: session runtime evidence (2026-07-24) for judge replies, gate parsing behavior, and third-checker audit logs.

What research says about LLM judges

Using one model to grade another is now standard practice, and the research behind it is more cautious than the practice. The paper that popularized the method, Zheng et al., found that strong judges such as GPT-4 reached over 80% agreement with human preferences, the same level of agreement between humans. That result is often quoted as a green light. The same paper also lists the judge’s weaknesses: position, verbosity and self-enhancement biases, plus limited reasoning ability.

Agreement on average also hides what happens on single items. A study of 13 judge models in a setting where human annotators agreed closely argued that percent agreement alone is not enough, because judges with similar agreement rates can still hand out very different scores. Only the strongest models came close to human alignment, and the authors report a tendency toward leniency.

Then there is consistency. Ask the same judge the same question twice and you may get two answers. The Rating Roulette paper measured this directly and reports low intra-rater reliability across repeated runs. A later large-scale evaluation adds the opposite warning: a judge can be consistent and still wrong, since a stable bias looks perfectly reliable.

The field now treats this as an engineering problem rather than a settled tool. A broad survey frames its core question as how to build reliable LLM-as-a-Judge systems, covering consistency, bias mitigation and ways to evaluate the judges themselves.

Known biases and how to counter them

• Position bias. In pairwise comparisons, the order of the candidates shifts the verdict. Wang et al. uncovered a positional bias in GPT-4 used as a referee and proposed scoring both orderings and aggregating the results. A systematic follow-up found that the effect varies across judges and tasks, so you cannot assume one fix covers every setup.
• Verbosity bias. Longer answers can look better to a judge without being better. Zheng et al. named it as one of the biases of LLM judges they studied. Counter it with rubrics that reward specific content, not length.
• Self-preference. Models tend to favor text that sounds like their own. Panickssery, Bowman and Feng showed that an LLM can rate its own outputs higher than texts written by other LLMs or humans even when human annotators judge them equal, and that this links to the model recognizing its own writing. If the writer and the judge are the same model, expect some home-team scoring.
• Explanation order. Wang et al. also found it helps to have the judge generate evidence before the score. Anthropic’s eval guidance says the same: ask the grader to reason before it produces an evaluation score.
• Untested graders. Anthropic describes LLM-based grading as fast and flexible but advises you to test to ensure reliability first, then scale. OpenAI’s grader docs recommend iterating on a grader prompt by building a model grader eval against high-quality reference answers.

How our incident lines up with the research

Our judge did three things the papers above predict. It changed its answer on the same article between runs, with the failing dimension rotating. That is the self-inconsistency problem in a live pipeline. It also gave straight fives to three files where the third checker later logged 49 logic errors, which matches the warning that a judge can sound confident and still miss what matters. And it wrote PASS one line after a searchability score of 3, against a rule it had just restated.

That last failure is the one the literature talks about least, because most studies measure the score, not the agreement between the score and the summary line. Our fix sits exactly there. The gate reads the per-dimension numbers and applies the minimum of 4 itself. The judge still writes a verdict. The gate just does not believe it.

None of this means the judge was useless. It produced the numbers that exposed its own contradiction. It was simply never safe as the final word.

A practical checklist for a judge gate

1. Give the judge a specific rubric with numeric scores per dimension. Anthropic’s docs recommend detailed, clear rubrics for this reason.
2. Enforce thresholds in code. Parse every dimension and compare it to the minimum. Log the verdict line, but never let it decide.
3. Treat a mismatch between scores and verdict as a failure, not a tie-break.
4. Replay a sample of items several times. If verdicts flip, the judge is not stable enough to be the only gate.
5. For pairwise comparisons, score both orderings and aggregate.
6. Avoid using the same model to write and to judge, or add a second judge from a different family.
7. Keep a separate verifier for logic or facts. Form scores do not catch deep errors.
8. Check the judge against human-labeled examples before trusting it at scale, and recheck after prompt or model changes.

More questions

Is an LLM judge good enough to replace human review?

For broad preference ranking, the best judges come close to human agreement levels according to Zheng et al. For a pass/fail gate on single items, that average is not enough. Run-to-run variance and known biases mean you still need deterministic checks and a separate verifier behind it.

Does setting temperature to zero fix inconsistent judges?

Not reliably. The Rating Roulette authors argue that forcing identical outputs can hurt agreement with human judgments rather than fix the underlying problem. Repeat runs and numeric thresholds in the gate are the safer path.