The Skill File That Stopped Mid-Verification and Still Got a Perfect Score

We shipped a transcript skill that looked perfect — until it stopped mid‑sentence

We built a skill that turns a single long video or audio source into timestamped written pieces where every quote must provably match a locked reference transcript. It was supposed to be airtight. An independent reviewer gave the first version top marks for form. It had no spelling complaints, the metadata was correct, the style matched the brief. It still failed.

The file simply stopped. Mid-way through its own final verification section the output truncated. The reviewer missed that. A third checker did not.

What we tried, what happened, the click

First attempt: verify quotes against the working copy we produced. It felt efficient. The code normalized cues, collapsed spacing, adjusted punctuation and then checked quotes against that normalized copy. It passed every internal assertion.

Then a third checker ran a logic sweep. It flagged the truncated verification step and kept going. Nineteen logic faults were found in that single skill. Across the three skills we produced that day the checker found forty‑nine problems. Every single fault was closed before anything reached the shelf.

The worst practical failure: we were verifying quotes against our own working copy rather than the raw fetched transcript. That’s a circular check. The working copy can drift. We proved this by deliberately inserting a subtle normalization change — a punctuation rewrite — in the working copy and watching the verification still pass. The check had no independent anchor. It was fake verification.

The second substantive failure: the download step could not, by construction, fetch the human-created subtitle track that a later selection step preferred. We tried to prefer human captions at selection time. But the downloader had only ever asked for auto captions. The selector could never see human captions even when they existed. It was a decision made upstream that broke downstream logic.

The turning point came when we combined those two observations. We fetched the raw transcript and immediately hash-locked it. Then we added a losslessness check that proves the working copy equals the raw cues. We requested both human and automatic caption tracks up front, and we recorded the track type for each. We re-ran the same third-checker sweep. The truncation was gone. The circular verification disappeared. The checker reported the faults closed. That is the moment the process became real verification instead of theater.

The rules that fell out

Rule: verify against the source you fetched, not the copy you made. Hash-lock the raw fetch. Prove the working copy is lossless. Record caption types at download time so selection is decidable. Treat every quoted span as a claim to prove: exact substring match against the locked reference, timestamp within two seconds, or block labeling the deliverable verified.

One representative detail we left here as a taste: when we lock the transcript at fetch time we compute a digest and store it with the job. Every quote check compares the quoted substring to the locked transcript using an exact-match operation, and the timestamp must be within two seconds of the referenced cue. That combination turned soft assertions into hard gates.

The full, tested, runnable version of the skill lives in our members’ library.

A counter-example we ran across

We once thought automated normalization would simplify downstream checks. It didn’t. A practical counter-example: a normalization that converts smart quotes to straight quotes silently changed multiple quoted spans so the verification step passed against the normalized copy but would have failed against the raw captions. In that case, normalization hid errors instead of revealing them. We keep normalization for output polish now, but only after the losslessness gate and raw-to-working equality are proven.

How we know: numbers and closure. Nineteen logic faults on that first skill. Forty‑nine across the three skills that day. All nineteen closed in version 1.1 before anything shipped. Independent review thought the first output perfect for form; the third checker found what form missed. The change that fixed it was simple and observable: lock the source, prove equality, fetch both caption types up front, and require exact matches with a two‑second timestamp tolerance.

Questions we got

Why lock the raw transcript instead of trusting the working copy?

Because the working copy can drift. We demonstrated that by changing normalization and watching verification still pass. Locking provides an immutable anchor so checks are meaningful.

Does requiring exact substring matches break useful paraphrasing?

It blocks labeling a deliverable “verified” when a quote is paraphrased. We made that deliberate. A timestamped quote is a claim; if it isn’t an exact match within two seconds, the deliverable cannot be labeled verified until a human resolves it.

Won’t requesting both caption types slow things down or cost more?

It adds an extra fetch, yes. But the alternative is an undecidable preference later on. Recording the track type at download time makes selection deterministic and short-circuits errors that cost far more time to debug.

We kept the story short because the lesson is simple and stubborn: verify against the source you fetched, not the copy you made.

Sources: internal production testing and verification notes from our content pipeline dated 2026-07-24.