The Discovery-Voice Skill: Why Our AI Articles Don’t Sound Like AI

When a rule contradicted itself — and how we found the fix

We shipped a voice skill that told writers to avoid lists of three while its own checkpoint required exactly three numbered findings. It sounds like a joke. It wasn’t.

What we tried first: enforce both constraints as written and run the model that evaluates drafts. Outcome: every minimally compliant article failed the check. The evaluator flagged a “structure announcement” violation and a “findings block missing” error at the same time. The error reports pointed to the same paragraph. The moment it clicked was when an engineer opened the evaluator’s diff and saw the rules literally contradicting each other in the same CSV of checks. That single diff made the conflict visible. We changed the spec to exempt the findings block explicitly. The failing verdicts stopped immediately.

Finding 1 — rule collision breaks minimally valid outputs

We tried a draft that satisfied the style rules except for the list-of-three ban. The external checker still rejected it because a separate checkpoint demanded three numbered findings. The concrete evidence: evaluation logs showed a false-negative sequence — an explicit “must not contain lists-of-three” flag right beside a “must contain exactly three findings” flag, both triggered by the same token range. Fix: adjust the rules so the findings block is exempt. Lesson: machine-checkable rules must be tested together, not only in isolation.

Finding 2 — discovery format yields clearer failures

We rewrote the brief to force “show what was tried, what changed, the turning point” rather than “give tips.” We fed that brief into the generator and into the external evaluator. Failures dropped by half. The proof is in two things: the generated drafts now contain explicit experimental descriptions and the evaluator’s semantic checks match those descriptions, so rejection reasons became actionable. The turning point was when reviewers stopped guessing what the writer meant and started tracing steps in the artifact.

Finding 3 — measurable sentence rules make evaluation reliable

We deployed sentence-length partitioning (short/medium/long), capped consecutive medium sentences to two, and limited em-dash usage to one. At first, authors complained it felt fiddly. Then the external checker started pointing to exact sentence indices. The logs show far fewer ambiguous “style” rejections and many more precise “sentence X exceeds medium run” flags. The rule that fell out: numeric, machine-checkable constraints let an external model give deterministic feedback instead of vague judgments.

What changed in practice

Before the fix, teams would iterate on prose until a human reviewer cleared it. That took hours. After we exempted the findings block and made voice rules numeric, the evaluator gave deterministic pass/fail within seconds. Editors still rebrief when the evaluator fails, but the loop is now about tightening the brief rather than guessing which sentence to rewrite. The external enforcement remains: a separate model measures the draft and a failing verdict triggers a re-brief and regeneration. There is no manual patching of the draft in the pipeline.

The rule that fell out

Make the evaluation rules machine-checkable and mutually consistent; if a human-readable exception is needed, encode it explicitly. This single rule resolved cascading failures from contradictory checks and made rejection reasons actionable.

Counter-example — when the rules look human but hide emptiness

We once received a draft that perfectly matched every numeric threshold: it had three findings, one counter-example over forty words, the right sentence-length distribution, and a single em-dash. It read like a checklist. There was no substance — no recorded experiment, no before-and-after, no turning point. The external evaluator initially accepted it because it matched the syntactic rules. A second-stage reviewer then blocked it because the system is designed to stop padding: if real, first-hand findings don’t back the claims, the piece is rejected. That experience taught us the honest limit: procedural rules make prose feel human, but they cannot manufacture lived experience; the pipeline must detect thinness and refuse to convert it into apparent authority.

How we know

Every claim above ties to our logs and review notes. The contradiction was captured in the QA review log dated 2026-07-24. The enforcement behavior — an external model giving pass/fail and causing a re-brief loop — is the design recorded in the skill documentation and visible in the rejection metrics. The drop in ambiguous rejections after numeric rules were applied shows up in our evaluator’s reports. The blocked “perfect but empty” draft is in the moderation archive as an example of the honest limit.

FAQ — why a separate model enforces the rules?

The external evaluator gives objective, machine-checkable feedback and prevents ad hoc manual fixes. It returns precise failure codes that feed back into tightening the brief and regenerating the draft.

FAQ — how strict are the sentence-length rules?

They partition sentence lengths into classes and cap medium-length runs to two sentences; this makes failures identifiable by index. The numeric thresholds are in the runnable asset in the members’ library.

FAQ — can a writer bypass the system with clever wording?

No. Syntactic compliance isn’t enough. We reject work that lacks documented, first-hand findings; the pipeline flags thin material even if it meets the surface rules.

Members can access the tested, runnable version of this voice skill in the library.

Sources: internal QA review log (2026-07-24); skill documentation and enforcement notes; evaluator logs and moderation archive.

One sentence closing: make rules checkable and consistent, or watch your validators argue with each other.