Four Ways We Checked the Same Work, Four Different Things We Caught

Checking AI-written work revealed a gap we didn’t expect

We were confident. An automated pass/fail check had cleared a set of AI-written reference documents. Then a fresh, systematic second read found 39 problems across 8 of the 10 files. That collapse of confidence is why I now start every project with a short sentence: checking AI-written work once is not checking it.

We ran the second read because the first pass felt too clean. Ten files. A different reviewer for each file who had never seen the originals. A fixed ten-point checklist. The numbers were concrete: 39 issues total — 1 critical, 18 high-priority, 20 minor. Every file had already passed the earlier automated quality check. Seeing the counts made the surprise real.

What we tried and what happened

First try: automated checks only. They pass. We move on. Quiet confidence.

Second try: close human-style read. We asked reviewers to look for logic, flow, and internal consistency. That stage caught most of the logic problems: unclear reasoning steps, inconsistent examples, and one critical sequencing bug. The sequencing bug was explicit and repeatable. A verification value was being calculated before the file had finished writing the value it was checking. In one document the file could “pass” its self-check while later edits changed the thing the check was supposed to guarantee. You could reproduce it by editing the document, running the automated check, then saving additional content and seeing the verification still show green. That single, reproducible mismatch changed how we think about ordering checks.

Third try: a systematic second pass against the ten-point checklist by fresh reviewers. That pass surfaced the 39 issues. We logged them, categorized urgency, and watched reviewers find both subtle rationale errors and small but consequential mismatches that the first automated run ignored.

Fourth try: live tests against real, messy input. We took two of the same files and ran them against production-like data rather than looking at them on paper. That run found eight more concrete problems that none of the previous checks had caught. Examples: one exact keyword choice flipped whether a piece of content passed or failed a check — swapping a single word changed the verdict. In another case we had planted three similar errors to test coverage; two were caught but a closely related third was missed. Those misses were not theoretical; they happened on live input and produced different outputs than expected.

Why the checks found different problems

Reading closely finds logic. Running live finds boundaries. The human read flagged internal consistency and sequencing. The automated check enforced surface constraints and formats. The live run hit edge cases: malformed inputs, ambiguous tokens, small keyword variations that change matching logic. Four different methods. Four different fault classes. No single method found everything.

That realization hardened into a rule for us: each way of checking AI-written work catches a different category of mistake, and none of them alone is sufficient.

One representative detail — a taste, not the recipe

Here’s one concrete taste of what changed our minds. In one document a conditional read used the keyword “complete” in a matcher that decided pass/fail. When we tested a near-identical input where the user wrote “completed,” the matcher treated it as a fail. The automated check had passed because the canonical test used “complete.” The human reader didn’t think to try “completed.” The live run threw multiple real examples and flipped the outcome. We fixed the matcher logic to use a small, context-aware normalization step and re-ran the test; the behavior aligned with expected outcomes. That one change removed a class of false negatives. I won’t publish the full normalization code here — the tested, runnable version lives in the members’ library — but this shows how one tiny token decision produces large downstream effects.

A counter-example we learned from

Not everything failed under live input. One file that looked fragile under the checklist behaved perfectly when run against messy data. The automated check and the fresh reviewer flagged several high-priority issues, but those issues didn’t manifest in production-style runs because the surrounding system normalized the inputs in the caller. That taught us two things: (1) overfitting checks to idealized failure modes wastes time, and (2) a downstream normalization layer can mask problems upstream — which may be acceptable sometimes and dangerous other times. We now record where normalization hides problems so we can decide whether to harden the document or accept the masking.

How we know this worked

We measured friction and fixes. After adding the second-pass and the live tests the defect class distribution shifted: logic and sequencing bugs fell after targeted edits; edge-case misses dropped only after expanding live test cases. The sequencing bug reproduced until we changed the order of operations; the keyword mismatch reproduced until we adjusted matching. The planted error test showed the coverage holes clearly. Each fix was validated by rerunning the specific check that had failed and by confirming the downstream behavior with real input.

Practical conclusion without a how-to

Checking AI-written work means staging at least these perspectives: machine checks for surface constraints, close human reads for reasoning, independent second-pass reviews for fresh eyes, and live runs against messy inputs for boundary behavior. The rule is blunt and actionable: mix them. But I won’t paste an end-to-end script here; the complete, runnable procedure we used is available in the members’ library.

FAQ — checking AI-written work

How many review layers do you recommend?

Four. Automated checks, a close human read, a systematic second-pass by fresh reviewers, and a live run against real inputs each find different errors.

Which layer caught the most critical bug?

The close human read found the critical sequencing bug where verification ran too early. That kind of temporal error showed up when someone followed the logic step by step rather than scanning for format compliance.

Can live testing replace the other checks?

No. Live testing exposes boundary and input variations but misses internal logic errors that a close read finds; also some problems only appear as a mismatch between test inputs and exact token choices.

We stopped trusting a single pass the day the files that “passed” still failed in the wild. It cost time. It also taught us what to watch for.

Sources: internal systematic reviews and live production tests performed July 24–25, 2026.