Our Image Skill Passed Review, Then We Found It Shipped the Wrong File

The surprise

We built a skill to produce one publication image and prove the delivered file met spec before it left the line. It passed a formal external review with top marks. We celebrated briefly. Then a mandatory third checker read the run outputs and flagged 18 logic errors the reviewer had missed.

That was the moment it clicked.

What happened, concretely

We fed the skill a curated brief. The brief came from a separate curator who distilled five third‑party image guides into a single document; the writer who coded the skill never saw those sources. That barrier was deliberate so copying couldn’t happen.

We ran the skill on a batch of sample briefs. The independent reviewer examined form and structure and gave a perfect score. The report was clean. The run logs looked clean too.

Then the third checker re‑played the same runs against the actual delivered artifacts and found 18 distinct logic issues. Many were subtle. One was catastrophic.

We reproduce the worst one here because it shows how an outwardly perfect verification can still lie. The run performed a crop correction, then executed verification steps. The readout said the corrected file matched the frozen spec. But the packager grabbed a different file — the pre‑crop image — and placed that into the delivery bundle. The verification step had checked one inode, the delivery step shipped another. The fix had been silently discarded.

Result: an image that failed the publisher’s spec went out with a clean proof bundle. The external reviewer did not catch it. The third checker did.

How we proved each issue

We stopped guessing and reproduced the full IO chain for each failing run. For a given sample we intentionally introduced a small, visible crop change. We watched four things.

First: which file the correction wrote to disk. We logged the exact filename and timestamp. Then we read the same path back and compared pixel dimensions against the frozen spec. We recorded a checksum at that moment.

Third: the packager step that selects the artifact to include in the delivery bundle. We let it run, then unzipped the bundle and read the packaged artifact’s checksum. They disagreed. That mismatch was the turning point.

It was obvious in logs once we forced the same readback and checksum comparison after every mutation. When we enforced that readback as the canonical truth, the bug could not repeat.

The rule that fell out

Make the post‑mutation file the canonical artifact for every subsequent step and prove it with an on‑disk readback plus checksum before packaging.

What changed in version 1.1

We closed all 18 issues the same day. The central changes were defensive and surgical.

One representative change: after any image correction the run now immediately reads the file it just wrote, checks the exact pixel dimensions against the frozen spec, computes a checksum and writes that checksum into the proof bundle. The packager then uses that checksum to select the artifact it will include, and refuses to continue if they do not match. That single detail removed the class of “verified one file, delivered another” errors.

We also added hard gates. If optical text detection finds three or more readable characters the run stops and writes a failure file. If a watermark detector scores 0.5 or higher, the run fails hard. If dimensions miss the spec by a single pixel, the system blocks instead of warning. Every run now outputs a proof bundle so any third party can re‑verify delivery without contacting us.

A fair counter‑example

One of the 18 issues came from a false positive: the watermark detector flagged a subtle texture as a potential mark on one sample when human reviewers saw nothing wrong. We kept the detector, but we changed how a failure is handled. It still blocks, which slowed that run. Later manual inspection showed the detector needed tuning, not removal. The system was harsher, but safer.

How we know

All of these findings are from our production line tests. We validated the fixes by re‑running every previously failing case. The new runs produced matching readbacks, matching checksums, and clean proof bundles. The failing artifacts no longer appeared in delivery packages. The most serious pre‑1.1 bug — the wrong file being delivered after a crop — no longer reproduces.

Questions we got

Why did the independent reviewer miss 18 logic errors?

The reviewer assessed form and spec compliance on the outputs it was shown, not the actual packaged artifact on disk. The gap was between “verified in one place” and “what was actually shipped.” The third checker audited the delivery bundle itself and therefore saw mismatches the reviewer could not.

Won’t hard blocking on minor things create too many failed runs?

Some runs will fail more often. We accepted that. The tradeoff is explicit failure modes and immediate remediation rather than noisy warnings that get ignored and bad artifacts slipping out. We tuned detectors to reduce false positives, but blocking remains the fallback.

Can you share the runnable skill file?

The tested, runnable version lives in the members’ library.

Rule, final: make the file on disk the single source of truth, prove it by reading it back and hashing it before you ever package or claim success.

Sources: internal production testing and change logs from our content pipeline tests, version 1.0 → 1.1, plus the curated brief provenance described above.