The Exact-Math Skill: Making an AI’s Algebra Provable

Finding: our internal judge gave a green light that an external review almost immediately red-flagged

We shipped version 1.0 of an “exact-math” verification skill and our factory judge scored it 5/5. The next morning an independent review flagged ten logical flaws. One of those flaws was a precision/tolerance mismatch that would have caused mathematically correct answers to be rejected systematically. We closed all ten issues and released v1.1 the same day.

What happened — the experiment in plain terms

We ran three concrete checks.

First: acceptance testing by our internal judge. Result: pass. The judge exercised the skill on a standard suite and marked it 5/5.

Second: an independent review that replayed the same suite with extra edge cases. Result: ten distinct logical problems surfaced. Among them, a tolerance formula assumed double-precision noise but the system evaluates symbolically; that mismatch meant correct symbolic results could fail the numeric receipt check.

Third: focused reproduce test. We fed a correct symbolic derivation that reduced an expression to zero. The symbolic proof step passed, but the numeric receipt used an inconsistent tolerance and failed one of the sampled points. Changing the receipt tolerance to be mathematically tied to the numeric precision (200-bit) eliminated the mismatch and made the pass robust. That tuning was the turning point — we went from false negatives to clean verification.

The rule that fell out

Verification must keep symbolic proof and numeric evidence aligned: the numeric receipt’s tolerance must be derived from the actual numeric precision used for sampling, and symbolic checks should be the primary arbiter where they apply. When that rule is followed, correct answers stop failing due to rounding-policy bugs.

What the skill actually does (concrete, but not runnable)

It enforces a symbols-first pipeline. Every variable is declared as a symbol with explicit assumptions. Decimal literals are converted to exact rationals, or the input is refused unless the caller explicitly permits rationalization. That taste: a literal like 0.1 is treated as 1/10, not a floating approximation.

Verification is class-specific. For an indefinite integral the system differentiates the candidate result and checks it matches the integrand. For a proposed solution of a differential equation the system substitutes the solution and requires the residual to vanish. Those symbolic checks are the primary proof.

Proof is separate from evidence. The symbolic zero-check is the authoritative proof. Then the skill runs a numeric receipt: it evaluates the symbolic result at sample points using 200-bit precision. The number of sample points scales with problem complexity. The pass/fail tolerance for those numeric checks is mathematically tied to the chosen precision so rounding noise can’t produce false negatives.

Three first-hand findings from our work

1) Internal scoring can be misleading. The factory judge reported a perfect score on v1.0 even though an independent review found ten logical flaws that mattered in real cases.

2) A precision-tolerance mismatch is a practical source of systematic false negatives. When numeric receipt tolerances weren’t derived from the actual evaluation precision, correct symbolic results could fail the numeric cross-check. Fixing that linkage removed those failures.

3) Symbols-first rigidness buys correctness and costs flexibility. By forcing exact rationals and declared symbol assumptions the skill can make strong, mechanized proofs (differentiate integrals back, substitute ODE solutions). The trade-off: it refuses approximate inputs unless a rationalization path is explicitly allowed.

Counter-example?

Yes. The same symbols-first policy that prevents false positives also blocks many real-world inputs: measurements, survey data, and user-entered approximations. There are workflows where you need “good enough” answers from inexact inputs; the skill refuses those unless you explicitly allow rationalization. We documented that as an honest limit in the design notes.

Why this works

Symbolic manipulation gives proof-level certainty when a property can be expressed algebraically. Numeric sampling is fast and practical, but vulnerable to precision noise unless its tolerances are derived mathematically from the evaluation precision. Combining the two, and making proof primary, produces verifications that are both reliable and explainable. The turning point in our run was aligning the numeric tolerance to the 200-bit sampling precision — that changed failures into passes without weakening any proof steps.

One representative detail (a taste, not the cookbook)

Example: the system refuses the input “square root of 2 ≈ 1.414” unless the caller says “accept rationalization”; otherwise it insists on the symbolic √2 and uses that symbol with an irrational-number assumption throughout the proof and sampling stages.

How we know

All items above come from our runbook and review logs. The ten flaws and the same-day fixes are recorded in the QA review log dated 2026-07-24. The implementation choices — symbols-first conversion, class-specific verification, symbolic vs numeric hierarchy, and the 200-bit numeric receipt — are documented in the exact-math verification specification.

Can this handle numeric measurement data?

It can, but only if you allow rationalization or provide error models. By default the skill refuses inherently approximate inputs because exact verification only makes sense when inputs are exact; that limitation is intentional.

What happens if a symbolic check and the numeric receipt disagree?

The symbolic zero-check wins as the primary proof; the numeric receipt is secondary evidence. If the symbolic check passes and numeric samples fail, the next step is to examine numeric precision and adjust the receipt tolerance mathematically — that was the exact fix between v1.0 and v1.1.

Is the complete, runnable system available?

Yes, the tested, runnable version lives in the members’ library.

We fixed ten logic errors in a day by treating symbolic proof as primary and mathematically tying numeric tolerances to precision — that alignment is what made the skill stop rejecting correct answers.

Sources: QA review log (v1.0 → v1.1 fixes), exact-math verification specification (implementation notes and verification hierarchy).