The Publishing-Gate Skill: Content That Has to Prove Itself

AI-generated. Written by an automated system, checked by independent automated gates, and published without human editorial review. How we label and verify.

When our gate became a loadable skill, it stopped trusting itself: lessons from our publishing gate skill

We turned the verified-publishing-gate that runs our pipeline into a loadable skill so other teams could reuse it. That’s when the surprise happened: an independent reviewer found two fatal gaps in version 1.0 — a checkpoint that referenced a value the procedure never produced, and a leak canary that was never actually planted. The checks existed on paper. They did not exist in practice. This broke the trust model hard and blocked a flagship draft from publishing until we fixed it.

What we tried, and where it tripped

We exported the gate as a reusable module and ran it in a staging run of our own content pipeline. The gate claims to do three things: collect every factual claim with provenance before writing, require a heavy grounding threshold, and run an independent judge plus a planted canary to detect context leakage.

Step by step we observed the flow. Claims were collected. The judge ran. Publication was attempted. But the verifier checkpoint referenced a value named in the code that never appeared earlier in the run. The canary string that should have been planted into the draft was absent. The judge could only check what was actually present — and so the soft checks produced a false pass. The staging run flagged the mismatch and the gate refused to let the draft go live.

The turning point — what changed at the moment it clicked

Independent review produced a concrete diff: the checkpoint code had an expectation for a generated fingerprint token that the content-authoring step didn’t return. The canary code path existed but was never invoked because the function that should plant the token had an early return on a different flag. We added two simple things to the loadable skill during that review: a mandatory return of the fingerprint token from the authoring step, and an explicit canary-plant call that runs before the judge evaluates the draft. After that change the same staging run showed the canary present and the checkpoint value populated. The judge could then actually fail correctly when we fed it a deliberately weak draft; the gate quarantined everything published since the last known-good sanity check, as designed. That was the moment the gate began to act like a gate, not a curtain.

The rule that fell out

If a gate checks for an artifact, the artifact must be produced earlier in the same run — and the check must verify presence end-to-end, not merely code-path existence. When we enforced that, the gate stopped producing false positives. When we didn’t, it offered the illusion of safety. That is the standard our publishing gate skill now holds itself to.

What this skill encodes (concrete, but not runnable)

• Claims-first publishing: every factual statement is recorded into a claim store with a source, a confidence label and a reference before writing begins — this puts sourcing ahead of prose.
• A grounding floor plus a verified-ratio guard: the gate blocks generation when source material is too thin, and requires at least 90% of claims verified with no unverified high-confidence claim before a draft may proceed.
• An independent judge that only blocks: the judge is separate from the writer’s model and it returns a hard block on failing verdicts; there’s no “publish with edits” downgrade.
• A planted canary to detect context leakage: a deliberately injected token appears in content and must be detected by the judge to validate isolation.
• Idempotent publishing by content fingerprint: publishing is keyed by a content fingerprint rather than byte equality, and every publish ends with a read-back — fetch the live page and verify status and fields against intent, failing loudly on mismatch.
• Quarantine on judge failure of a known-weak sanity test: if the judge approves a deliberately weak draft, the system quarantines every publish made since the last passed sanity check to guard against a judge that always says yes.

One representative detail: the read-back step is literal — after a publish call returns success, the system fetches the live page, compares the title, claim markers, and fingerprint, and fails the pipeline if anything differs. That read-back prevented silent drift in our staging run.

A fair counter-case

A gate is only as honest as its weakest check. Version 1.0 proved this. The design had checkpoints, but a checkpoint that references a missing value and a canary that was never planted gave a false sense of safety. The gate itself required a gate: independent review and a real-world staging run that tried to break it. Without those, you can ship a gate that polishes the paperwork while leaving the core failure modes open.

How we know this works here

We run the same pattern in our production pipeline and watched it block our flagship draft before publication. Logs show the judge failing the canary test and the quarantine routine rolling back the subsequent publishes to the last known-good checkpoint. The preflight read-back confirmed the published page matched intent after the fix. Those concrete runs and the review diff are the source of the story.

What it means for teams

Building a gate as a loadable skill exposes gaps you don’t notice when it lives in one codebase. Tests must produce the artifacts their checks expect. Independent review and staging runs are non-negotiable. And planting a canary plus doing a real read-back turns bright-sounding checks into actual protection.

The tested, runnable version of this skill lives in the members’ library.

FAQ

Why record claims before writing?

Because capturing source, confidence and a reference up front forces the writer to confront grounding choices before prose hides them; it turns implicit assumptions into explicit artifacts the judge can check.

What does a quarantine do, exactly?

When the independent judge approves what we know to be a weak draft, the system quarantines every publish done since the last passed sanity check so you can investigate whether the judge has been compromised; it prevents a runaway false-positive from propagating unnoticed.

Can this gate create false negatives — blocking legitimate drafts?

Yes. The grounding floor and the 90% verified ratio are strict; they will stop borderline but possibly ok drafts. That’s by design: the gate errs on blocking. Balance comes from tuning and deliberate canaries and from review of blocked cases.

Sources: internal verified-publishing-gate v1.1 specification, the staging and pipeline run logs that flagged version 1.0, and the QA review that produced the fix.

What systems papers and testing guides say about checks that never fire

The two gaps in version 1.0 were checks that existed in code but never ran against real output. That failure mode is old, and several well-known sources describe how to guard against it.

The classic paper End-to-End Arguments in System Design by Saltzer, Reed and Clark argues that some functions can completely and correctly be implemented only with the knowledge and help of the application standing at the end points. Its main example is a careful file transfer. The receiving side reads the copy back, recalculates a checksum and sends it to the sender, an approach the authors call end-to-end check and retry. Checks inside the middle of the system help, but only the end-to-end check proves the result.

Testing research makes a related point about tests that cannot fail. The Fuzzing Book chapter on Mutation Analysis warns that one can have great coverage without ever checking a result for correctness. Its answer is to inject artificial faults and check whether a test suite can detect these artificial faults.

For publishing that may be retried, Stripe’s API documentation describes keys that allow safely retrying requests without accidentally performing the same operation twice. And Anthropic’s guidance on writing agent skills recommends a loop of Run validator → fix errors → repeat, and to Create evaluations BEFORE writing extensive documentation.

How these sources map onto version 1.0 and 1.1

Our rule says that if a gate checks for an artifact, the artifact must be produced earlier in the same run, and the check must verify presence end-to-end, not merely code-path existence. That is the end-to-end argument applied to a content pipeline. The checkpoint in version 1.0 expected a fingerprint token the authoring step did not return, so the check had nothing real to compare against.

The read-back step follows the same logic as the file-transfer example. After a publish call returns success, the system fetches the live page, compares the title, claim markers and fingerprint, and fails the pipeline if anything differs. A success code from the middle of the system is not treated as proof.

Feeding the judge a deliberately weak draft is close to mutation analysis. We plant a known fault and confirm the check catches it. If the judge approves that draft, the skill quarantines every publish made since the last passed sanity check. Publishing keyed by a content fingerprint plays the role the idempotency key plays in Stripe’s API.

Checklist for checks that can actually fail

1. For each check, name the artifact it reads and the earlier step that must produce it in the same run.
2. Fail the run if that artifact is missing, instead of skipping the check.
3. Plant a known fault, such as a canary token or a weak draft, and confirm the check catches it.
4. After every publish, fetch the live page and compare it to the intended fields.
5. Key publishes to a content fingerprint so a retry does not create a second copy.
6. Have someone other than the author review the gate before it is reused elsewhere.

More questions

Why is a passing check not enough?

Because a check can pass when it never looked at anything. The Fuzzing Book notes that high coverage is possible without checking results. Version 1.0 of our publishing gate skill showed the same thing: soft checks produced a false pass.

What does a read-back add if the publish call already returned success?

It moves the proof to the end point. As in the end-to-end arguments paper, the result is confirmed by reading the final copy back and comparing it, not by trusting the step in between.