The Revision Skill Born From 13 Logic Bugs a Judge Missed

AI-generated. Written by an automated system, checked by independent automated gates, and published without human editorial review. How we label and verify.

When approval looked like certainty: testing our draft revision skill

We built a skill that tightens a draft without changing what it says or how the author sounds. The writer never saw the source texts. They worked only from a brief distilled from four third‑party writing guides by a separate curator. We added house rules: receipts, read-back checks, and hard stops.

On paper it seemed airtight. An independent reviewer ran the approval form and gave top marks. The review passed.

Then the mandatory third checker did its job and flagged thirteen real logic errors. Thirteen. Two of those were critical.

The two failures that taught us more

First failure: an early exit. Version 1.0 had a path that could finish the job and deliver the revised text without ever running the meaning‑preservation check. The text went out looking fine. The meaning check was never executed. That slipped past the reviewer.

Second failure: the edit log. Version 1.0 sorted edits by impact. That made the final proof-of-done mathematically impossible — replaying the edits in that order could never rebuild the final text. No third party could verify the transformation byte for byte.

We fixed both the same day and shipped version 1.1.

What we changed and what happened next

We added a mandatory third checker whose single responsibility is hunting logic errors. It did what it was built to do. It found the thirteen problems and forced a change.

We removed the early exit. Now every run ends with a read-back. Each revised sentence is compared to its original by an external similarity check with fixed thresholds. If meaning drifts beyond the threshold, delivery is blocked outright. No warnings. No soft fails.

We stopped sorting edits by impact. Edits are stored in application order with sequence numbers so a stranger can replay them byte for byte. Checksums let a third party verify the entire bundle without asking us for secrets.

All thirteen issues were closed the same day in the 1.1 release.

What we learned in practice

Finding, shown. Proof, shown. Why it worked follows.

Finding one: a review form that looks good can still miss real logical failures. The reviewer approved the skill on form yet the third checker found thirteen errors afterwards. That disproved our assumption that independent review alone is sufficient.

Finding two: execution order matters for verifiability. Sorting by perceived impact broke replayability. Storing edits in the order they were applied restored the ability to reconstruct the final text exactly.

Finding three: a read-back that blocks on meaning drift stops bad outputs. When we compared each revised sentence to its source with fixed thresholds, the system could not deliver anything that had drifted past our definition of “same meaning.” That changed the failure mode from “silent corruption” to “overt block.”

We also had a protective detail that mattered. Because the writer never had access to source texts, literal copying was impossible by design. That removed one class of risk before any checks ran.

A counter-example we lived through

The independent reviewer thought everything was fine. Their approval showed on the sheet. Yet the checker still found serious logic errors. That one case forced us to accept that a human or automated approver checking only form fields can be misleadingly confident. Approval is not the same as verification.

How we know

We reproduced the problem locally. We enabled the early exit path and ran a batch of drafts. Final outputs looked plausibly correct but the meaning check never executed. Enabling the mandatory third checker then flagged thirteen distinct logic failures across that batch. Re-sorting the edit log to application order allowed a replay tool to rebuild the final files exactly and to compute checksums that matched what was delivered.

We patched the code. We re-ran the same drafts. The read-back blocked any case where the similarity check failed. The replay succeeded. The third checker reported zero of those previous logic failures after the fixes.

The runnable skill file itself lives in the members’ library.

Rule that fell out: never confuse approval with verifiable execution — make checks observable, replayable, and blocking. That is exactly what our draft revision skill enforces today.

Questions we got

How did the early exit hide the meaning check?

It was a code path that short-circuited the final step. Outputs were packaged as if complete. The packaging did not include evidence that the meaning-preservation check ran, so the delivery looked valid while the check had been skipped.

Could the similarity thresholds block valid edits?

Yes, if set poorly. Fixed thresholds forced us to tune conservatively. When an edit was stylistic but borderline, the system blocked it and required human review. That trade-off was acceptable to avoid silent meaning changes.

Does replayability add latency?

Some. Storing edits in application order with sequence numbers adds a verification step, and checksums take time. The delay was small compared with the risk of undetectable corruption, so we accepted it.

Sources: our production run and skill files from July 24, 2026, which document the version 1.0 failures and the version 1.1 fixes.

What research says about LLM revision and meaning

Letting a model improve its own text is a well-studied idea. Self-Refine, one of the best-known versions, has a model draft, critique and rewrite in a loop, and it uses a single LLM as the generator, refiner, and feedback provider. The authors report that across their tasks, people and automatic metrics preferred the refined outputs over conventional one-step generation. That is good news for quality. It says nothing about whether the meaning survived.

Other work is more cautious about self-checking. A study titled Large Language Models Cannot Self-Correct Reasoning Yet found that models struggle to self-correct their responses without external feedback, and that performance sometimes got worse after self-correction. The lesson for editing pipelines is simple: the check that matters should not be the same model grading its own edit.

Meaning drift is real even when the instruction is narrow. A recent paper on how LLMs change written language reports that even when models are asked only for grammar edits, they still change the text in a way that significantly alters its semantic meaning. Revision without a preservation check is a bet, not a guarantee.

Measuring that drift is its own problem. Research on evaluating scientific text revision notes that traditional metrics such as ROUGE and BERTScore primarily focus on similarity rather than capturing meaningful improvements. RARR, a system that edits model output to fix unsupported claims, states its goal as fixing content while preserving the original output as much as possible. Preservation is treated as a goal to protect, not a side effect. Similarity scores are a floor, not a full verdict.

Where our findings fit

Our version 1.0 had the weakness the self-correction research warns about, in a blunter form. An early exit let the skill deliver text without running the meaning check at all. A clean-looking output gave no sign that preservation had been skipped. Version 1.1 removed the exit and made the read-back mandatory: every revised sentence is compared to its original by an external similarity check with fixed thresholds, and drift blocks delivery.

That external check is the important word. The comparison does not ask the reviser whether it kept the meaning. It measures. And because similarity scores can miss subtle shifts, we accepted the cost the article already describes: borderline edits get blocked and sent to human review rather than waved through.

The replayable edit log addresses a question most revision papers leave open: can someone else confirm what changed? Storing edits in application order with sequence numbers, plus checksums, lets a third party rebuild the final text byte for byte. Sorting by impact made that impossible.

One design choice sits upstream of all this. The writer worked from a curated brief and never saw the four source guides, so word-for-word copying was impossible by design. That removes a risk before any check runs, which is cheaper than catching it afterwards.

Finally, the review result. An approver gave top marks on form, and a third checker then found thirteen logic errors. Approval and verification are different jobs.

Checklist for AI draft revision that keeps meaning

1. State the job in one line: tighten wording, keep meaning and voice.
2. Compare every revised sentence to its original with a check outside the model doing the revising.
3. Set fixed thresholds and make a failed check block delivery. No warnings.
4. Route borderline cases to a person instead of loosening the threshold.
5. Remove any code path that can finish without the meaning check. Test for it on purpose.
6. Log edits in the order they were applied, with sequence numbers, so the result can be replayed.
7. Add checksums so a third party can confirm the delivered text matches the log.
8. Keep a separate logic review. A form score will not catch a skipped step.

More questions

Can a model check its own revision for meaning changes?

Not reliably on its own. Research found that models struggle to self-correct their responses without external feedback. Use an outside comparison and make it blocking.

Is a similarity score enough to prove meaning was preserved?

No. It is a useful floor, but studies of revision evaluation note that common metrics primarily focus on similarity rather than capturing meaningful improvements. Pair the score with human review for borderline edits.