AI-generated. Written by an automated system, checked by independent automated gates, and published without human editorial review. How we label and verify.
A judge model returned 5-5-5-5-5 on a file that stopped in the middle of a section. The final heading had nothing under it and the last sentence broke off mid-clause. A second judge gave top marks to a file whose summary described a different picture than the one that judge had verified minutes earlier. A third approved a publishing gate that would pass an article with no H1. Three separate passes of AI code review, three clean verdicts, and in all three the form of the artifact was in order. Form is what a judge actually reads.
What we assumed about AI code review
We assumed independence came from the prompt: give a model the reviewer’s instructions, tell it to be strict, hand it a rubric. We had already learned once that an approving reviewer can sit on top of thirteen real logic errors, and we responded by adding a dedicated logic checker. What we never asked was where the reviewer was standing.
The truncated file makes the mistake visible. A file ending mid-section is not subtle — a character count and a glance at the last line catch it. The judge missed it because everything before the break looked like a good file: headings in order, sections opening as the rubric expected, prose in the house register. It scored what it had been oriented to find, found it, and reported agreement.
Recognition wearing the clothes of verification
This is the mechanism, and it is not a flaw in any particular model. A reviewer that knows what a change was supposed to do reads it looking for that. When it finds it, the reading feels like verification. It was recognition. A confirmed expectation produces the same output as a defect that does not exist.
The second case closes the loop tighter. That model had itself established a set of facts, then read a file presenting a different account of them, and passed it. Having produced the premises, it re-derived the same conclusions and called that a check. A model reviewing work it participated in has no position outside that work to review it from: standard and artifact came from the same place, so there is nothing to compare against.
The SEO gate is the version that costs money later. It was reviewed for whether its rules read sensibly, not for what they let through. Nobody asked which input the gate accepts today that yields a wrong verdict — a different question from “does this look right,” and the only one that would have caught it.
A finding without a failure scenario is noise
The other half of the problem shows up as volume rather than silence. Reviews that do produce findings produce far too many, and most are observations: a name that could be clearer, a structure someone would have organised differently, a restatement of what a tool already prints. None can be acted on without arguing about taste.
The requirement that fixes this is one sentence long. Every candidate must come with a specific input, the changed line it reaches, and the wrong result it produces. If you cannot construct that, you do not have a finding. We keep a count of what got dropped for lacking one, because a review that publishes only what it flagged is indistinguishable from a review that flagged whatever it noticed. Noise does not sit beside the blockers; a reader sorting thirty observations to find two real problems reads neither carefully.
Deterministic checkers must never spend reviewer attention
Formatters, linters, type checkers and schema validators already return an exact, repeatable, cheap verdict. Handing that to a model is not redundancy — it spends the scarce thing. Every line of attention used restating what a validator printed is a line not spent on the consumer three files away that never learned about the new enum value.
What follows for anyone building AI code review
The principle is narrow. A logic leg must be something other than the instance that wrote the change. Independence is a property of the dispatch, not of a label: a separate invocation receiving the artifact and the procedure and nothing else — no rationale, no summary, no account of what the change was for. A fresh context of the same model qualifies on those terms; a renamed reader does not, and neither does asking the same invocation twice.
Two things follow. The surface under review has to be pulled out of stored state rather than from a description of it: a commit message and a status code are reports about a request, not statements about what is stored. And the fix has to be re-read by someone who did not write it, since a fix is authored by a party just told what the defect was — the exact condition under which a reviewer confirms what it expected.
A clean review and a review that never happened produce identical output. What separates them is a record of what was examined and what was rejected — which is why the dropped set belongs in the verdict. Without it, “reviewed” is a word attached to a file.
The runnable procedure — the classes, the decision rule separating a blocker from a comment, and the guards that catch these failures — lives in the member library.
How we know
What the research says
The failures above have names in the literature, and most of them were measured before anyone wired a model into a merge pipeline.
Start with the reviewer grading its own side. The paper that popularized model judges, Zheng and colleagues’ MT-Bench study, lists the known weaknesses up front: position bias, verbosity bias, self-enhancement bias, and limited reasoning ability. A later paper went after the mechanism. Panickssery, Bowman and Feng found that models can tell their own text from other text with non-trivial accuracy, and reported a linear correlation between self-recognition capability and the strength of self-preference bias. A model that recognizes its own work tends to like it more. That is the measured version of what this article calls recognition wearing the clothes of verification.
Asking the same model to look again does not fix it. Huang and colleagues tested intrinsic self-correction on reasoning tasks and concluded that LLMs struggle to self-correct their responses without external feedback, with performance sometimes getting worse after the attempt. The useful word there is external. A second pass by the same invocation adds no new information.
Long inputs are a separate problem, and a quieter one. The “Lost in the Middle” study found that performance significantly degrades when models must access relevant information in the middle of long contexts. The RULER benchmark pushed further: models that score nearly perfectly on simple needle-in-a-haystack retrieval still exhibit large performance drops as the context length increases. Truncation is the extreme case. A reviewer cannot weigh the part of the file it never received, and nothing in its output will say so. API documentation is clear that the truncation signal lives in metadata, not in the text: Anthropic’s guide to stop reasons tells developers to check whether a response reached the max_tokens limit and handle that case explicitly. The same discipline applies to the input side. Check length before you trust a score.
Field data on automated review in production is mixed. An industry study Automated Code Review In Practice, presented at ICSE 2025, followed an LLM-based review tool across projects at Beko and reported that developers resolved most of the automated comments, yet the pull request closure duration increased from five hours 52 minutes to eight hours 20 minutes on average. The authors also reported faulty reviews and irrelevant comments. Volume costs reviewer time even when individual comments are fine.
How this maps to what our judges did
Each of the three verdicts in this article lines up with one finding above. The 5-5-5-5-5 on a truncated file is the long-context problem at its limit: the judge scored the part it saw. The top score on a file that contradicted the judge’s own earlier account fits self-preference and the weak record of self-correction. The SEO gate that let an article without an H1 through is what happens when a reviewer is asked whether rules read sensibly instead of what input they accept.
The research also supports the remedy this article describes. Independence comes from the dispatch, not from the instructions, and evidence about what was examined belongs in the verdict.
A checklist for AI code review you can trust
- Before the model sees anything, check the artifact’s length and its last line. A cut-off input fails the review outright.
- Pull the artifact from stored state, not from a commit message or summary.
- Send the logic review to a separate invocation that gets the artifact and the procedure only.
- Check the response’s stop reason. A truncated verdict is not a verdict.
- Require each finding to name an input, the changed line it reaches and the wrong result.
- Run formatters, linters, type checkers and schema validators first, and keep their output out of the model’s attention budget.
- Record the dropped findings next to the kept ones.
More questions
Does a bigger context window solve truncated or ignored input?
Not by itself. RULER found that although the models tested all claimed 32K tokens or more, only half could maintain satisfactory performance at the length of 32K, and the “Lost in the Middle” results show that position inside the window matters too. Measure what the reviewer received.
Is it enough to use a different model as the judge?
It helps with one problem. The research ties self-preference to self-recognition, so a judge from a different model has less of its own text to recognize. It does not fix truncation or a reviewer primed with the author’s rationale. What the reviewer receives matters as much as which model it is.
Grounded in: our own judge-calibration runs behind change-review-gate v1.1, 2026-09-04 — a 5-5-5-5-5 verdict on a truncated file, a top score on a file contradicting the judge’s own verified account, and an approved SEO gate that admitted an article without an H1. Verified: each artifact was re-read from stored state after the verdict and compared against what the verdict claimed about it. The runnable procedure lives in the member library.