A judge model returned 5-5-5-5-5 on a file that stopped in the middle of a section. The final heading had nothing under it and the last sentence broke off mid-clause. A second judge gave top marks to a file whose summary described a different picture than the one that judge had verified minutes earlier. A third approved a publishing gate that would pass an article with no H1. Three separate passes of AI code review, three clean verdicts, and in all three the form of the artifact was in order. Form is what a judge actually reads.
What we assumed about AI code review
We assumed independence came from the prompt: give a model the reviewer’s instructions, tell it to be strict, hand it a rubric. We had already learned once that an approving reviewer can sit on top of thirteen real logic errors, and we responded by adding a dedicated logic checker. What we never asked was where the reviewer was standing.
The truncated file makes the mistake visible. A file ending mid-section is not subtle — a character count and a glance at the last line catch it. The judge missed it because everything before the break looked like a good file: headings in order, sections opening as the rubric expected, prose in the house register. It scored what it had been oriented to find, found it, and reported agreement.
Recognition wearing the clothes of verification
This is the mechanism, and it is not a flaw in any particular model. A reviewer that knows what a change was supposed to do reads it looking for that. When it finds it, the reading feels like verification. It was recognition. A confirmed expectation produces the same output as a defect that does not exist.
The second case closes the loop tighter. That model had itself established a set of facts, then read a file presenting a different account of them, and passed it. Having produced the premises, it re-derived the same conclusions and called that a check. A model reviewing work it participated in has no position outside that work to review it from: standard and artifact came from the same place, so there is nothing to compare against.
The SEO gate is the version that costs money later. It was reviewed for whether its rules read sensibly, not for what they let through. Nobody asked which input the gate accepts today that yields a wrong verdict — a different question from “does this look right,” and the only one that would have caught it.
A finding without a failure scenario is noise
The other half of the problem shows up as volume rather than silence. Reviews that do produce findings produce far too many, and most are observations: a name that could be clearer, a structure someone would have organised differently, a restatement of what a tool already prints. None can be acted on without arguing about taste.
The requirement that fixes this is one sentence long. Every candidate must come with a specific input, the changed line it reaches, and the wrong result it produces. If you cannot construct that, you do not have a finding. We keep a count of what got dropped for lacking one, because a review that publishes only what it flagged is indistinguishable from a review that flagged whatever it noticed. Noise does not sit beside the blockers; a reader sorting thirty observations to find two real problems reads neither carefully.
Deterministic checkers must never spend reviewer attention
Formatters, linters, type checkers and schema validators already return an exact, repeatable, cheap verdict. Handing that to a model is not redundancy — it spends the scarce thing. Every line of attention used restating what a validator printed is a line not spent on the consumer three files away that never learned about the new enum value.
What follows for anyone building AI code review
The principle is narrow. A logic leg must be something other than the instance that wrote the change. Independence is a property of the dispatch, not of a label: a separate invocation receiving the artifact and the procedure and nothing else — no rationale, no summary, no account of what the change was for. A fresh context of the same model qualifies on those terms; a renamed reader does not, and neither does asking the same invocation twice.
Two things follow. The surface under review has to be pulled out of stored state rather than from a description of it: a commit message and a status code are reports about a request, not statements about what is stored. And the fix has to be re-read by someone who did not write it, since a fix is authored by a party just told what the defect was — the exact condition under which a reviewer confirms what it expected.
A clean review and a review that never happened produce identical output. What separates them is a record of what was examined and what was rejected — which is why the dropped set belongs in the verdict. Without it, “reviewed” is a word attached to a file.
The runnable procedure — the classes, the decision rule separating a blocker from a comment, and the guards that catch these failures — lives in the member library.
How we know
Grounded in: our own judge-calibration runs behind change-review-gate v1.1, 2026-09-04 — a 5-5-5-5-5 verdict on a truncated file, a top score on a file contradicting the judge’s own verified account, and an approved SEO gate that admitted an article without an H1. Verified: each artifact was re-read from stored state after the verdict and compared against what the verdict claimed about it. The runnable procedure lives in the member library.
Leave a Reply