The Skill File That Stopped Mid-Verification and Still Got a Perfect Score

AI-generated. Written by an automated system, checked by independent automated gates, and published without human editorial review. How we label and verify.

We shipped a transcript skill that looked perfect — until transcript quote verification stopped it mid‑sentence

We built a skill that turns a single long video or audio source into timestamped written pieces where every quote must provably match a locked reference transcript. It was supposed to be airtight. An independent reviewer gave the first version top marks for form. It had no spelling complaints, the metadata was correct, the style matched the brief. It still failed.

The file simply stopped. Mid-way through its own final verification section the output truncated. The reviewer missed that. A third checker did not.

What we tried, what happened, the click

First attempt: verify quotes against the working copy we produced. It felt efficient. The code normalized cues, collapsed spacing, adjusted punctuation and then checked quotes against that normalized copy. It passed every internal assertion.

Then a third checker ran a logic sweep. It flagged the truncated verification step and kept going. Nineteen logic faults were found in that single skill. Across the three skills we produced that day the checker found forty‑nine problems. Every single fault was closed before anything reached the shelf.

The worst practical failure: we were verifying quotes against our own working copy rather than the raw fetched transcript. That’s a circular check. The working copy can drift. We proved this by deliberately inserting a subtle normalization change — a punctuation rewrite — in the working copy and watching the verification still pass. The check had no independent anchor. It was fake verification.

The second substantive failure: the download step could not, by construction, fetch the human-created subtitle track that a later selection step preferred. We tried to prefer human captions at selection time. But the downloader had only ever asked for auto captions. The selector could never see human captions even when they existed. It was a decision made upstream that broke downstream logic.

The turning point came when we combined those two observations. We fetched the raw transcript and immediately hash-locked it. Then we added a losslessness check that proves the working copy equals the raw cues. We requested both human and automatic caption tracks up front, and we recorded the track type for each. We re-ran the same third-checker sweep. The truncation was gone. The circular verification disappeared. The checker reported the faults closed. That is the moment the process became real verification instead of theater.

The rules that fell out

Rule: verify against the source you fetched, not the copy you made. That is the essence of transcript quote verification done right. Hash-lock the raw fetch. Prove the working copy is lossless. Record caption types at download time so selection is decidable. Treat every quoted span as a claim to prove: exact substring match against the locked reference, timestamp within two seconds, or block labeling the deliverable verified.

One representative detail we left here as a taste: when we lock the transcript at fetch time we compute a digest and store it with the job. Every quote check compares the quoted substring to the locked transcript using an exact-match operation, and the timestamp must be within two seconds of the referenced cue. That combination turned soft assertions into hard gates.

The full, tested, runnable version of the skill lives in our members’ library.

A counter-example we ran across

We once thought automated normalization would simplify downstream checks. It didn’t. A practical counter-example: a normalization that converts smart quotes to straight quotes silently changed multiple quoted spans so the verification step passed against the normalized copy but would have failed against the raw captions. In that case, normalization hid errors instead of revealing them. We keep normalization for output polish now, but only after the losslessness gate and raw-to-working equality are proven.

How we know: numbers and closure. Nineteen logic faults on that first skill. Forty‑nine across the three skills that day. All nineteen closed in version 1.1 before anything shipped. Independent review thought the first output perfect for form; the third checker found what form missed. The change that fixed it was simple and observable: lock the source, prove equality, fetch both caption types up front, and require exact matches with a two‑second timestamp tolerance.

Questions we got

Why lock the raw transcript instead of trusting the working copy?

Because the working copy can drift. We demonstrated that by changing normalization and watching verification still pass. Locking provides an immutable anchor so checks are meaningful.

Does requiring exact substring matches break useful paraphrasing?

It blocks labeling a deliverable “verified” when a quote is paraphrased. We made that deliberate. A timestamped quote is a claim; if it isn’t an exact match within two seconds, the deliverable cannot be labeled verified until a human resolves it.

Won’t requesting both caption types slow things down or cost more?

It adds an extra fetch, yes. But the alternative is an undecidable preference later on. Recording the track type at download time makes selection deterministic and short-circuits errors that cost far more time to debug.

We kept the story short because the lesson is simple and stubborn: verify against the source you fetched, not the copy you made.

Sources: internal production testing and verification notes from our content pipeline dated 2026-07-24.

What standards, vendor docs and research say about checking quotes against a source

Start with the format most caption tracks arrive in. The W3C WebVTT specification says WebVTT files provide captions or subtitles for video content, along with chapters and other time-aligned metadata. Each cue carries timing, and the spec requires that a cue’s start time must be greater than or equal to the start time offsets of all previous cues in the file. That ordered timeline is what makes a timestamp tolerance meaningful: a quote can be pinned to a cue, and the cue to a moment in the recording.

Text cleanup is where quotes quietly change. Unicode Standard Annex #15 explains why normalization exists: when strings are kept in one normalized form, equivalent strings have a unique binary representation. It also warns about the stronger forms. Normalization Forms KC and KD must not be blindly applied to arbitrary text, because they erase formatting distinctions. Even a standard, well-specified rewrite changes characters. A home-made cleanup step that swaps punctuation is less predictable still.

Locking a fetch needs a fixed fingerprint of its bytes. Python’s hashlib documentation lists SHA256 among the FIPS secure hash algorithms SHA224, SHA256, SHA384, SHA512, and it notes that Some algorithms have known hash collision weaknesses, naming MD5 and SHA1. A digest taken at fetch time lets every later step prove it is still looking at the same transcript.

Model vendors recommend the same discipline for generated text. Anthropic’s guide to reducing hallucinations suggests asking the model to extract word-for-word quotes before working on a long document. It also describes having the model verify each claim with a supporting quote: If it can’t find a quote, it must retract the claim.

Research on factual precision takes the same unit-by-unit view. The FActScore paper breaks a generation into a series of atomic facts and scores the share of them supported by a reliable knowledge source. The key words are “reliable knowledge source.” A check is only as good as the reference it compares against.

How those sources map onto the 19 faults

The circular check fails the FActScore test at the reference step. Version 1.0 compared quotes to our own working copy, not to the raw fetched transcript. When we inserted a punctuation rewrite into the working copy, verification still passed. The reference had drifted along with the output, so a match proved nothing. Hash-locking the raw fetch gives the check a reference that cannot move.

The smart-quote counter-example is the Unicode warning in practice. A conversion from smart quotes to straight quotes changed several quoted spans. Verification passed against the normalized copy but would have failed against the raw captions. That is exactly why UAX #15 tells implementers not to apply aggressive forms blindly. Our response keeps normalization for output polish, and only after the losslessness gate proves the working copy equals the raw cues.

The caption-track fault is a reference problem too. The downloader only ever asked for automatic captions, so the later preference for human captions could never be satisfied. Version 1.1 requests both tracks up front and records the type of each, which makes the choice decidable at selection time.

The quote rule itself lines up with the vendor guidance. Every quoted span must be an exact substring of the locked transcript, with a timestamp within two seconds of the referenced cue. If either test fails, the deliverable cannot be labeled verified. That is the “retract the claim” rule, enforced by code rather than by the model’s own judgment. All 19 faults in the transcript skill, and 49 across the three skills from that day, were closed in version 1.1 before anything shipped.

Checklist for quote checks you can trust

1. Fetch every available caption track, human and automatic, and record each track’s type.
2. Compute a digest of the raw fetch, for example with SHA-256, and store it with the job.
3. Before using a working copy, prove it is lossless against the raw cues.
4. Check each quote as an exact substring of the locked transcript, not of the working copy.
5. Require the quote’s timestamp to fall within the agreed tolerance of its cue, two seconds in our case.
6. Apply punctuation and spacing cleanup only after verification, and never to the reference.
7. Test the checker by planting a small change in the working copy. If verification still passes, the check is circular.
8. Block the “verified” label on any mismatch until a person resolves it.

More questions

Why not normalize both the quote and the transcript the same way and compare?

Because then a difference the cleanup erased can never be detected. If smart and straight quotes become identical on both sides, a quote that changed one into the other still matches. Comparing against the raw, locked transcript keeps those differences visible. Cleanup can still happen for readability once the check is done.

Is a two-second timestamp tolerance strict enough?

It is a tolerance for timing, not for wording. The words must match exactly, and the timestamp must fall within two seconds of the cue it points to. Because WebVTT cues are ordered in time, the tolerance stops a correct sentence from being attached to the wrong moment without forcing frame-level precision.