AI-generated. Written by an automated system, checked by independent automated gates, and published without human editorial review. How we label and verify.
One model gave its own prose a clean bill of health. That was the moment everything changed.
We had a model write articles and then grade them — a pure self-review loop with no outside check. It almost always passed its own drafts. Even when the prose read as obviously machine-made.
The weird part wasn’t the facts. The facts were fine. Both the writer and an independent reviewer caught invented details. The blind spot was rhythm, cadence, the humanness of voice.
How the experiment played out
Step one: we used a single model to generate an article and to score it on several axes—accuracy, clarity, voice. It handed its own work high marks nearly every time. Fluency got A’s. Tone got A’s. Even when a human reader winced at the phrasing.
Step two: we replaced the grader with a different model. Same capability level. Different initial weights and prompt history. The outcomes shifted. Drafts the writer had passed were suddenly failing. Not because facts were wrong—those stayed solid. Because the reviewer flagged the “AI-sounding” rhythm and repeated patterns: phrasing that looped, flat sentence variety, predictable transitions.
Both models were close in raw capability. What mattered was independence, not intelligence. A model is a weak judge of the exact patterns it has just produced.
Concrete examples from our runs
One article about small business taxes: the writer graded its draft 9/10 for voice. The independent reviewer scored voice 5/10 and annotated ten sentences as “robotic.” The human editor agreed with the independent reviewer.
Another piece on customer research: self-review passed it. Independent review flagged a repeating phrase structure used five times in four paragraphs and marked the lead as “too generic.” We rewrote the lead. Traffic and engagement rose. No factual corrections were needed.
We saw the same pattern repeatedly. Self-review waved through AI-sounding prose. An independent reviewer—different model, different perspective—caught it.
The rule that came out of doing this
The model that writes never makes the publish decision; a different model must sign off, and we only trust a check we have seen reject something.
We turned that into a hard rule in our content pipeline. The writer model produces. It may mark obvious failures to save time. But the final gate is an independent model’s approval or rejection. If that model rejects, we fix the draft. We only publish after an independent pass that has explicitly accepted the content.
Why independence beats extra smarts
We tried upgrading the reviewer to a higher-capability model. That helped a little. But the biggest jump came when the reviewer was different, not smarter. Two close models with different priors and different prompt histories spot each other’s blind spots.
Think of it like proofreading: someone who wrote the paragraph rarely catches the rhythm they used. A different reader hears it immediately.
Don’t toss self-review—use it where it helps
We kept self-review as a cheap first pass. It removes broken output—hallucinations, missing sections, malformed lists—before the expensive independent reviewer runs. That saves tokens and human time.
But we never let self-review act as the final arbiter. It performs triage, not judgement. When the independent reviewer rejects something, we pay attention; rejections are where the learning happens.
How we know this isn’t just noise
We repeated the cycle across dozens of articles and different topics. The writer model’s pass rate on its own drafts stayed high. The independent reviewer consistently downgraded voice in a predictable way. Human editors confirmed the independent reviewer’s calls. That pattern held across business and product content, and across short and mid-length pieces.
Simple operational rule to take away
Make the writer and the final reviewer different systems. Keep self-review as a quick filter. Require a visible rejection from the independent reviewer to trigger a rewrite. Treat an approval from the independent reviewer as meaningful—especially when it has flagged things first.
FAQs
Can a model usefully grade its own writing?
Useful as a fast filter. It catches broken output and saves time. It fails at judging its own voice and stylistic patterns. Rely on it for coherence checks, not for final sign-off.
Does a more capable reviewer solve the problem?
A more capable reviewer helps, but the biggest gain came from using a different reviewer. Independence, not just raw capability, uncovers the rhythmic and stylistic blind spots that the writer repeats.
What counts as “independent” in practice?
Independence can be another model instance with different initialization or a different model family. The key is different priors and prompt history so it doesn’t share the writer’s blind spots.
We learned this by letting models grade their own work until they didn’t. The moment a different reviewer began failing drafts the writer had passed is when the rule hardened.
Sources: observations and repeated tests run in our own content pipeline.
What the research says about models grading themselves
Our pipeline was not an outlier. Several research groups have measured the same weakness, and they describe it in plainer terms than we did at first.
Self-preference is a named bias
In LLM Evaluators Recognize and Favor Their Own Generations, Panickssery, Bowman and Feng define self-preference as the case where an LLM evaluator scores its own outputs higher than other texts while human annotators consider them of equal quality. They went a step further and asked whether the model knows which text is its own. Out of the box, models such as GPT-4 and Llama 2 had non-trivial accuracy at telling their own text apart from other models and humans. When the authors fine-tuned that ability, they found a linear correlation between self-recognition capability and the strength of the bias. Their conclusion is direct: self-recognition can interfere with unbiased evaluations.
Judges have other habits too
A study of model judges, Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, lists position, verbosity, and self-enhancement biases along with limited reasoning ability. The same paper reports that strong judges reached over 80% agreement with human preferences, the same level of agreement between humans. Both findings matter. A model judge can be useful and still be tilted toward certain answers, including its own.
Self-correction without outside input is weak
Another team tested whether models can fix their own reasoning in Large Language Models Cannot Self-Correct Reasoning Yet. Their finding: models struggle to self-correct their responses without external feedback, and at times their performance even degrades after self-correction.
A second study, Pride and Prejudice: LLM Amplifies Self-Bias in Self-Refinement, looked at the loop where a model critiques and revises its own draft. It found that self-refinement improves fluency and understandability but further amplifies self-bias. What helped was external feedback with accurate assessment, which the authors say can significantly reduce the bias.
Guidance from model providers
Anthropic’s guide to building evaluations covers model-based grading, and one of its tips is to have detailed, clear rubrics. A rubric does not remove self-preference. It does make the grader’s decision easier to inspect, which is a precondition for catching it.
How this lines up with what we saw
Our runs showed three things, and each has a counterpart in the papers above.
First, the writer graded its own drafts high nearly every time, including the tax article it scored 9/10 for voice while the independent reviewer gave 5/10. That is the pattern the self-preference work describes: higher scores for one’s own text that a human reader does not agree with. In our case the human editor sided with the independent reviewer.
Second, the blind spot was voice and rhythm, not facts. Both the writer and the independent reviewer caught invented details. The research does not tell us why the gap lands on style in particular. What it does show is that a model can recognize its own output, and a judge that recognizes its own phrasing is badly placed to call that phrasing robotic.
Third, swapping in a different reviewer helped more than upgrading to a smarter one. The self-correction studies point the same way. Outside feedback is what moves the result, not another pass by the same system.
We also kept self-review as a cheap first filter for broken output. Nothing in the research argues against that. The papers warn against using a model as the final judge of its own work, which is the role we removed.
Checklist for setting up an independent reviewer
- Use a different model for the final publish decision than the one that wrote the draft.
- Keep self-review, but only for triage: missing sections, malformed lists, obviously broken output.
- Give the reviewer a written rubric with separate scores for facts, structure and voice, so you can see which axis it is grading on.
- Log every rejection and the reason. A reviewer that never rejects has not been tested.
- Have a human read a sample where writer and reviewer disagree. That is where bias shows up first.
- Re-check the setup after any model or prompt change. Self-preference depends on the model, so a swap can bring it back.
Is self-preference proven, or just a hunch?
It has been measured. The self-preference study found models favor their own outputs over texts that human annotators rated as equal quality, and linked the effect to the model’s ability to recognize its own writing.
Does asking the model to revise its draft fix the problem?
Not on its own. Research on self-refinement found that the revise loop improved fluency but amplified the model’s bias toward its own output. Feedback from an independent, accurate source is what reduced it.