Our AI Judge Scored a 3, Then Passed Its Own Gate

That day it passed a failing article

We run a judge on every article. It scores five dimensions from 1 to 5. Its written rule is simple: every dimension must be at least 4 for a pass. The judge writes both the numbers and the verdict line in its own reply.

On 2026-07-24 the judge gave an article searchability = 3. On the very next line it wrote VERDICT = PASS. Same reply. Same breath. The score and the verdict contradicted each other. We watched the session output and copied the lines into logs.

The surprising part: the downstream gate that decides what goes live only read the VERDICT line. It ignored the numbers entirely. Because the gate parsed only that single line, the article went live despite failing the explicit rule the judge itself had restated.

The broken upstream signal

We tried the same article again. Sometimes the judge failed it. Sometimes the judge passed it. The failing dimension rotated between runs. One run would show a failing number in searchability. The next run would show the same number but a PASS verdict anyway. This made clear the judge was a component that could give varying answers; it was not an oracle.

Later that same day the judge gave three files perfect straight fives. They looked flawless on paper. A mandatory third checker then ran full logic verification and found 49 logic errors in those three supposedly perfect files. One file was cut off in the middle of its final verification step and nevertheless had straight fives recorded.

Across two batches the third checker found 100 logic errors in six files that the judge had approved on form. The checks that look like “pass/fail” on the surface failed to catch deep problems.

What we proved, and the rule we adopted

Proof came in three moves. First, we captured the judge’s full reply containing both the numeric scores and the verdict line; this showed the internal contradiction. Second, we replayed the same content multiple times and recorded the variance — fail once, pass the next, with the failing dimension moving around. Third, we ran a separate, mandatory verifier over items that had perfect scores and recorded dozens of logic errors that the judge had not reported.

At the turning point we changed the gate. It stopped trusting the single-line verdict. The gate now reads the numbers the judge emits and enforces the minimums itself. That one change stopped bad content from going live when the judge’s textual verdict disagreed with the numeric dimensions.

Out of that work came a concise operational rule: a judge is a component you must test, not an oracle; the gate reads the numbers, never the conclusion line.

A taste of the change

We did not rewrite the entire pipeline in front of you. One representative detail: the gate now extracts each dimension and compares it to the threshold before allowing publication; the judge’s VERDICT is logged but ignored for the pass/fail decision. The fully tested, runnable version of this change lives in the members’ library.

Counter-example we saw

There are cases where the judge’s verdict and numbers align perfectly and the content is clean. We had files where scores matched human review and the third checker found no issues. That shows the component can behave correctly. But because the same content could alternately fail and pass across runs, we could not rely on that consistency.

How we know

Everything above is backed by session runtime evidence captured on 2026-07-24: the judge reply that quoted a failing number and wrote PASS; the downstream gate behavior that parsed the verdict line only; and the third-checker audit that logged 49 errors in three perfect-scored files and 100 errors across six files in two batches. We reviewed the logs line-for-line.

Questions we got

Why did the gate read only the verdict line?

The gate was coded to parse a single summary token. It was a simplification that looked tidy at first but created a single point of failure when the judge contradicted itself.

Could we fix the judge instead?

We tried. The judge’s outputs were softened by variability across runs. Fixing the judge helped sometimes. But the immediate safety comes from the gate enforcing numeric thresholds itself, so we treated the judge as fallible.

Does this add latency to publishing?

A little. The gate now does an extra parse-and-compare step. The trade-off is minor compared with preventing flawed work from going live.

Rule, again: a judge is a component you test, not an oracle; the gate reads the numbers, never the conclusion line.

Sources: session runtime evidence (2026-07-24) for judge replies, gate parsing behavior, and third-checker audit logs.