The Article We Almost Published, and the Check That Stopped It

AI-generated. Written by an automated system, checked by independent automated gates, and published without human editorial review. How we label and verify.

Why the content quality gate saved us

We almost published an article that quietly flattered our own tool, and a content quality gate stopped it. The gate is a second automated reader whose only job is to approve or refuse, and that never writes anything itself. It flagged the draft before anyone could hit publish.

The near-miss

Someone on the team had written a short explainer. It read fine. The writer liked it. We skimmed it and felt satisfied. It would have gone live.

Then the automated check refused it. It didn’t point out a factual error. It refused on tone. The prose had slid into selling. The piece sounded like an endorsement. We had missed it. The refusal stopped the publish button from being pushed.

We replayed the moment in the editor’s chair. The draft looked competent. It flowed. It had useful points. And yet the check labelled its tone as promotional. We read the same sentences again and felt the sales shift. That was the moment it clicked.

What we actually tried

We had built an independent approval layer into our pipeline. The rule we set was simple: nothing goes live until an independent check signs off, and the writer never approves their own work. The check is automatic, repeatable, and has one output: approve or refuse.

At first the gate rarely said no. Most drafts pass. We began to question whether the gate was worth the extra step. It felt like an administrative tax. So we ran a small experiment. We watched the gate make decisions for a week. We logged every refusal and every approval. We paid attention to the ones that disagreed with the editor’s initial read.

The turning point happened when we watched the gate refuse something in real time. We looked at the draft, then at the gate’s reason. The gate flagged tone over facts. That single observed refusal changed everything for us. We decided to trust only a check we had watched reject something. A gate that never says no is not a gate.

The discoveries that followed

First finding: the gate catches tone drift. The automated check did not fail the piece on facts. It failed it on tone: the piece had drifted into selling instead of explaining, and we had not noticed. That was concrete. The refusal had a succinct explanation, and when we edited to neutralize the promotional wording the gate approved the revision.

Second finding: most drafts pass. We counted approvals. The majority moved through without friction. The gate is not there to block the good ones; it is there for the handful that look fine and are not, which are exactly the ones a tired person waves through. Those are the dangerous ones.

Third finding: the presence of the gate changes behavior. Writers began to self-check for the gate’s refusal modes. They rewrote lead paragraphs more carefully. They dropped semi-promotional phrases earlier in the drafting process. The gate acted as an invisible instructor—visible only when it refused—and that nudged better choices upstream.

We tried removing the gate for a week to see the cost savings. The immediate effect was speed. Fewer steps. Faster publish times. But within days we caught ourselves missing subtle endorsements in tone. We almost published the flattery piece during that removal week. That was the experiment that justified reintroducing the gate.

The rule that fell out

Nothing in our name goes live without an independent sign-off. The sign-off must come from a separate check that we can watch refuse when necessary. That rule is now hard. Everyone knows it. The writer cannot be the sole approver of their own copy.

We show one representative detail: when the gate refuses for tone it returns the sentence fragment it flagged and a short rationale. Editors use that fragment as the pivot for revision. That detail is useful to hear because it explains how the gate guides edits without doing the writing for you.

We do not publish the entire approval algorithm here. The tested, runnable version lives in the members’ library.

A fair counter-example

The gate costs something. It rejects drafts that were probably fine, and it adds a step. One editor lost an afternoon rewriting a passage that the gate refused for being slightly promotional, even though the piece’s overall balance was fine. That was true work and it felt bureaucratic. We logged the time cost and the frustration. We kept the gate anyway because the cost of one bad piece going live in our own name was higher.

The gate is a trade-off, not a magic bullet. It catches the occasional near-miss and prevents reputational slips. It also means editors must accept extra friction and occasional false positives.

How we know

We measured three things during the trial week: the number of refusals, the type of refusals, and the downstream edits required after refusal. We watched refusals in realtime. We documented the ones that prevented promotion of our own product or tone. We compared publish quality before and after reintroducing the gate. The refusals that mattered were precisely the ones we’d have waved through when tired.

We learned to read the gate’s output as guidance rather than as an infallible judge. It’s a guardrail that still needs human judgment. But it is a guardrail that has to be allowed to stop things sometimes.

Questions we got

Does the gate check facts?

Sometimes. In our system it can flag factual inconsistencies if configured to, but the most useful refusals were about tone. The gate’s biggest value was catching when the voice shifted from explanation to promotion.

Does it slow publishing down?

Yes. It adds a step. Some drafts are delayed for short rewrites. That is the point. We accepted the delay because the alternative — a single promotional article in our name — felt costlier than the time spent rewriting.

Can writers bypass it?

No. The rule is explicit: the writer does not approve their own work. The gate is independent and its approval is required. That separation prevents accidental self-promotion and reduces confirmation bias.

We still argue about edge cases. We still get a false refusal now and then. The gap that matters is short: between a draft that reads fine and one that is actually right. The content quality gate widens that gap for the better.

Sources: our own content pipeline experiments and refusal logs from the trial weeks in our editorial system.

What research and Google Search Central say about independent review

The rule that a writer never approves its own work is not only an editorial habit. Research on using language models as reviewers gives a concrete reason for it.

In LLM Evaluators Recognize and Favor Their Own Generations, Panickssery, Bowman and Feng note that new biases are introduced due to the same LLM acting as both the evaluator and the evaluatee. They name one of them directly: self-preference, where an LLM evaluator scores its own outputs higher than others’. They also report a linear correlation between self-recognition capability and the strength of self-preference bias.

The earlier study Judging LLM-as-a-Judge found that strong model judges can match human preferences well, achieving over 80% agreement, the same level of agreement between humans. The same paper lists the weak spots: position, verbosity, and self-enhancement biases. In short, a model reviewer can be useful, but it is not neutral about its own text.

Anthropic’s guide to building evaluations gives practical advice for this kind of grader. It recommends that you have detailed, clear rubrics and that you ask the grader to give a fixed verdict instead of a loose opinion, such as only correct or incorrect. That fits a gate whose only output is approve or refuse.

On tone, Google’s guide to creating helpful, reliable, people-first content asks creators to self-assess pages with questions such as whether the title or heading would avoid exaggerating or being shocking in nature. It is a checklist for people, not a scoring tool, but it points at the same drift our gate flagged.

How this connects to what our gate caught

Our gate refused a draft on tone, not facts: the piece had drifted into selling instead of explaining, and we had not noticed. The writer liked it and we skimmed it and felt satisfied. That is the situation where a reviewer close to the text is least reliable, and the self-preference research is about exactly that closeness, measured in models.

We also found that most drafts pass, and that the gate is there for the handful that look fine and are not. A judge that agrees with people most of the time is consistent with that. The value sits in the minority of cases.

Finally, when the gate refuses for tone it returns the flagged sentence fragment and a short rationale. That is the same direction as the advice to use clear rubrics and fixed verdicts: an output an editor can act on.

Checklist for an approve-or-refuse gate

1. Use a reviewer that did not write the draft, and never let the writer approve its own work.
2. Limit the output to approve or refuse, with the flagged fragment and a short reason.
3. Write the rubric down, including tone rules such as explaining instead of selling.
4. Before trusting the gate, watch it refuse at least one draft you know should fail.
5. Log every refusal with its type and the edit it led to.
6. Review false refusals and their time cost, and keep the gate only if one bad piece would cost more.

More questions

Why not let the writing model check its own draft?

Because research on self-preference shows model evaluators tend to rate their own output higher. A separate reviewer removes that one bias, even though it may still have others.

Can a model reviewer replace human judgment?

Not fully. Model judges can reach high agreement with people, but the Judging LLM-as-a-Judge study also documents known biases. We treat the content quality gate as a guardrail whose output is guidance, not an infallible verdict.