The report arrived the way it always does. The provider’s dashboard said the customer had paid; our database said they had not. Nobody had shipped anything that week. The payment webhook delivery log showed every event sent and acknowledged, no failures, nothing queued for retry. From the provider’s side the integration looked healthy. From ours, an entitlement was missing and someone was locked out of what they had paid for.
What we thought we were checking
We opened our own application logs first, which is the common wrong instinct: they said the handler ran and returned success. Our confidence rested on three things — test mode passed, every call returned 2xx, the delivery log was clean. Each is a real signal; none is a statement about durable state.
A delivery status answers was it sent. A write response answers what did I decide. A test-mode pass answers does the shape work. None answers the question that matters when money is involved: what exists right now in the system that owns the money. The house rule applies here with more force than anywhere else — 200 is not proof, read back the actual state.
What a payment webhook acknowledgement actually proves
An integration that looks fine has four surfaces that can return success while the state they imply does not exist.
The first is delivery. The provider records that it sent an event and that your endpoint answered; it has no visibility into whether your handler wrote anything. Frameworks make it easy to acknowledge on arrival and process after — and the moment you return 2xx, the retry schedule stops. A recoverable failure becomes a permanently lost event, and the log reads as delivered forever. So the count that matters is distinct event ids in the provider’s log against distinct event ids in your own append-only record of arrivals, over one window: more in theirs than in yours is acknowledgement running ahead of durable work.
Two idempotency problems that look like one
Outbound, your writes carry an idempotency key so a network retry does not create a second charge. Inbound, the provider’s events arrive more than once and must not be applied twice. Solving one does nothing for the other, and teams routinely believe they have done both.
The subtler failure is a key derived from the attempt rather than the business fact: a fresh value per attempt is a nonce sitting in a key header, and the retry charges again. The inverse is quieter — one key reused across operations that are not the same operation. A replayed key returns the first request’s stored response, so the object you fetch back carries the earlier amount or currency while its status looks healthy. A status check passes; only a field-by-field comparison against what you intended catches it.
Backwards edges nobody wrote down
Between intent and settlement there is a state machine; most integrations have one only in the sense that the code implies it. The provider emits its own vocabulary; your app stores its own. Every provider state must map to exactly one of yours, and an unmapped state defaults silently — in practice, to “has access”. Then the transitions: which moves into each state are legal, and which states are terminal.
Payment webhook delivery is not ordered, and every payload is a snapshot from its own moment. Without a written transition table, a late event carrying an old snapshot walks a dead record backwards into life, and nothing treats it as an error. Reading an intent status as settlement is the same family of mistake: entitlement granted against something that never completed.
Test mode is not coverage
The fourth surface is the test/live split. Test mode exercises the shape and hides the production behaviour: real authentication challenges, issuer decline codes, disputes, live-only configuration with its own signing secret, anything a test clock compresses. Asking for the written list of what test mode cannot exercise is itself the check: if the list does not exist its length is zero, and zero is not a plausible count of behaviours that differ between test and live.
What follows for anyone wiring up money
The principle is narrow and it does the work: the only source that can confirm a payment state is the one that owns the money. Your logs are your own account of your intent; the delivery log is the provider’s account of its sending. Neither is the state.
So this is a read-back task, not a logging task. Reconciliation runs both ways, because each direction finds a different disaster: your paid records against the provider’s settled objects finds access granted without money, the reverse walk finds money taken without access. The output is numbers, not a verdict — two drift counts and the ids behind them. Zero and zero passes; anything else is an open item with an owner.
The runnable procedure — the surfaces in order, each with what to detect and how to recover, and the checkpoint that lets someone outside the room re-derive the verdict — lives in the member library.
How we know
Grounded in: our own hardening pass over a live payment integration on 2026-09-04, opened by a report that the provider dashboard and our database disagreed about who had paid. Verified: distinct event ids in the provider’s delivery log counted against distinct event ids in our own arrival ledger for one window; the same logical charge issued twice in test mode and the object count read back from the provider; every created object re-fetched by id and compared field by field against what we sent. No success response was accepted as evidence. The runnable procedure lives in the member library.
Leave a Reply