The Exact-Math Skill: Making an AI’s Algebra Provable

AI-generated. Written by an automated system, checked by independent automated gates, and published without human editorial review. How we label and verify.

Finding: our internal judge gave a green light that an external review almost immediately red-flagged in our exact math skill

We shipped version 1.0 of an “exact-math” verification skill and our factory judge scored it 5/5. The next morning an independent review flagged ten logical flaws. One of those flaws was a precision/tolerance mismatch that would have caused mathematically correct answers to be rejected systematically. We closed all ten issues and released v1.1 the same day.

What happened — the experiment in plain terms

We ran three concrete checks.

First: acceptance testing by our internal judge. Result: pass. The judge exercised the skill on a standard suite and marked it 5/5.

Second: an independent review that replayed the same suite with extra edge cases. Result: ten distinct logical problems surfaced. Among them, a tolerance formula assumed double-precision noise but the system evaluates symbolically; that mismatch meant correct symbolic results could fail the numeric receipt check.

Third: focused reproduce test. We fed a correct symbolic derivation that reduced an expression to zero. The symbolic proof step passed, but the numeric receipt used an inconsistent tolerance and failed one of the sampled points. Changing the receipt tolerance to be mathematically tied to the numeric precision (200-bit) eliminated the mismatch and made the pass robust. That tuning was the turning point — we went from false negatives to clean verification.

The rule that fell out

Verification must keep symbolic proof and numeric evidence aligned: the numeric receipt’s tolerance must be derived from the actual numeric precision used for sampling, and symbolic checks should be the primary arbiter where they apply. When that rule is followed, correct answers stop failing due to rounding-policy bugs. That alignment is now built into every run of our exact math skill.

What the skill actually does (concrete, but not runnable)

It enforces a symbols-first pipeline. Every variable is declared as a symbol with explicit assumptions. Decimal literals are converted to exact rationals, or the input is refused unless the caller explicitly permits rationalization. That taste: a literal like 0.1 is treated as 1/10, not a floating approximation.

Verification is class-specific. For an indefinite integral the system differentiates the candidate result and checks it matches the integrand. For a proposed solution of a differential equation the system substitutes the solution and requires the residual to vanish. Those symbolic checks are the primary proof.

Proof is separate from evidence. The symbolic zero-check is the authoritative proof. Then the skill runs a numeric receipt: it evaluates the symbolic result at sample points using 200-bit precision. The number of sample points scales with problem complexity. The pass/fail tolerance for those numeric checks is mathematically tied to the chosen precision so rounding noise can’t produce false negatives.

Three first-hand findings from our work

1) Internal scoring can be misleading. The factory judge reported a perfect score on v1.0 even though an independent review found ten logical flaws that mattered in real cases.

2) A precision-tolerance mismatch is a practical source of systematic false negatives. When numeric receipt tolerances weren’t derived from the actual evaluation precision, correct symbolic results could fail the numeric cross-check. Fixing that linkage removed those failures.

3) Symbols-first rigidness buys correctness and costs flexibility. By forcing exact rationals and declared symbol assumptions the skill can make strong, mechanized proofs (differentiate integrals back, substitute ODE solutions). The trade-off: it refuses approximate inputs unless a rationalization path is explicitly allowed.

Counter-example?

Yes. The same symbols-first policy that prevents false positives also blocks many real-world inputs: measurements, survey data, and user-entered approximations. There are workflows where you need “good enough” answers from inexact inputs; the skill refuses those unless you explicitly allow rationalization. We documented that as an honest limit in the design notes.

Why this works

Symbolic manipulation gives proof-level certainty when a property can be expressed algebraically. Numeric sampling is fast and practical, but vulnerable to precision noise unless its tolerances are derived mathematically from the evaluation precision. Combining the two, and making proof primary, produces verifications that are both reliable and explainable. The turning point in our run was aligning the numeric tolerance to the 200-bit sampling precision — that changed failures into passes without weakening any proof steps.

One representative detail (a taste, not the cookbook)

Example: the system refuses the input “square root of 2 ≈ 1.414” unless the caller says “accept rationalization”; otherwise it insists on the symbolic √2 and uses that symbol with an irrational-number assumption throughout the proof and sampling stages.

How we know

All items above come from our runbook and review logs. The ten flaws and the same-day fixes are recorded in the QA review log dated 2026-07-24. The implementation choices — symbols-first conversion, class-specific verification, symbolic vs numeric hierarchy, and the 200-bit numeric receipt — are documented in the exact-math verification specification.

Can this handle numeric measurement data?

It can, but only if you allow rationalization or provide error models. By default the skill refuses inherently approximate inputs because exact verification only makes sense when inputs are exact; that limitation is intentional.

What happens if a symbolic check and the numeric receipt disagree?

The symbolic zero-check wins as the primary proof; the numeric receipt is secondary evidence. If the symbolic check passes and numeric samples fail, the next step is to examine numeric precision and adjust the receipt tolerance mathematically — that was the exact fix between v1.0 and v1.1.

Is the complete, runnable system available?

Yes, the tested, runnable version lives in the members’ library.

We fixed ten logic errors in a day by treating symbolic proof as primary and mathematically tying numeric tolerances to precision — that alignment is what made the skill stop rejecting correct answers.

Sources: QA review log (v1.0 → v1.1 fixes), exact-math verification specification (implementation notes and verification hierarchy).

What the research and the SymPy documentation say

The research on language models and math agrees on one point: the model is a weak place to do the arithmetic. The PAL paper starts from the observation that models often make logical and arithmetic mistakes in the solution part, even when the problem is broken down correctly. Its answer is to have the model write a program and let something else run it. The model reads the problem and generates code, but offloads the solution step to a runtime such as a Python interpreter.

“Let’s Verify Step by Step” looks at checking rather than solving. Its abstract notes that even state-of-the-art models still regularly produce logical mistakes, and compares feedback on the final result with feedback on each step. For training reward models on the MATH dataset, it reports that process supervision significantly outperforms outcome supervision. The authors also released PRM800K, described as 800,000 step-level human feedback labels.

GSM-Symbolic tests how stable model reasoning is when a problem changes in ways that should not matter. The paper reports that the performance of all models declines when only the numerical values in a question are altered. Adding one clause that seems relevant but is not causes significant performance drops (up to 65%) across the models it tested.

SymPy’s own documentation covers the exactness traps. Its gotchas page explains that with plain division inside an expression, Python will evaluate the two numbers before SymPy sees them, which is how a float slips into what should be exact. The tutorial describes an equals method that tests two expressions by evaluating them numerically at random points. And the basic operations tutorial warns that numerical evaluation can leave roundoff errors smaller than the desired precision, which the chop flag can remove.

How this lines up with the exact-math skill

The skill’s design follows the PAL idea further than PAL does. The model does not produce the answer. It sets up symbols with assumptions, and the symbolic engine does the work. Converting decimal literals to exact rationals, or refusing them, is the documented fix for the float trap in SymPy’s gotchas, applied as a rule instead of left to habit.

The v1.0 failure sits exactly where SymPy’s documentation says it will. A numeric check at finite precision leaves small residues. Our receipt compared those residues against a tolerance that was not derived from the 200-bit sampling precision, so correct results failed. Tying the tolerance to that precision was the change between v1.0 and v1.1.

The verification hierarchy also echoes the research. Differentiating an integral back, or substituting a solution into its differential equation, checks the steps’ result against the problem itself. A random-point check is useful evidence, which is how the skill uses it, but the symbolic zero-check stays primary. Let’s Verify is about training reward models, not about this skill, so the parallel is limited: both treat a final verdict as weaker than checking the work.

The judge that scored v1.0 at 5/5 is the uncomfortable part. An independent review with extra edge cases found ten flaws. GSM-Symbolic’s result, that small changes to a problem move model performance, is a reason not to let a model’s verdict stand in for a mechanical check.

A checklist for verifying model math

1. Never accept a number the model computed in prose. Make it produce something that can be executed.
2. Declare every variable as a symbol with its assumptions before computing.
3. Convert decimal literals to exact rationals, or refuse them unless rationalization is allowed.
4. Check by class: differentiate integrals back, substitute solutions into their equations.
5. Run a numeric receipt at a stated precision, and derive the tolerance from that precision.
6. When symbolic proof and numeric evidence disagree, inspect precision first.
7. Have the suite reviewed by something other than the system that built it, with changed numbers and edge cases.

More questions

Why not let the model check its own arithmetic?

Because the same weakness applies to the check. The PAL paper’s premise is that models often make logical and arithmetic mistakes in the solution part. Moving the computation into a runtime removes that step from the model entirely.

Is a numeric spot check at random points enough?

It is good evidence and a weak proof. SymPy’s tutorial describes equals as a test made by evaluating them numerically at random points, and finite precision leaves residues. Use it as a receipt beside a symbolic check, with a tolerance tied to the precision you evaluate at.