AI-generated. Written by an automated system, checked by independent automated gates, and published without human editorial review. How we label and verify.
The report arrived the way it always does. The provider’s dashboard said the customer had paid; our database said they had not. Nobody had shipped anything that week. The payment webhook delivery log showed every event sent and acknowledged, no failures, nothing queued for retry. From the provider’s side the integration looked healthy. From ours, an entitlement was missing and someone was locked out of what they had paid for.
What we thought we were checking
We opened our own application logs first, which is the common wrong instinct: they said the handler ran and returned success. Our confidence rested on three things — test mode passed, every call returned 2xx, the delivery log was clean. Each is a real signal; none is a statement about durable state.
A delivery status answers was it sent. A write response answers what did I decide. A test-mode pass answers does the shape work. None answers the question that matters when money is involved: what exists right now in the system that owns the money. The house rule applies here with more force than anywhere else — 200 is not proof, read back the actual state.
What a payment webhook acknowledgement actually proves
An integration that looks fine has four surfaces that can return success while the state they imply does not exist.
The first is delivery. The provider records that it sent an event and that your endpoint answered; it has no visibility into whether your handler wrote anything. Frameworks make it easy to acknowledge on arrival and process after — and the moment you return 2xx, the retry schedule stops. A recoverable failure becomes a permanently lost event, and the log reads as delivered forever. So the count that matters is distinct event ids in the provider’s log against distinct event ids in your own append-only record of arrivals, over one window: more in theirs than in yours is acknowledgement running ahead of durable work.
Two idempotency problems that look like one
Outbound, your writes carry an idempotency key so a network retry does not create a second charge. Inbound, the provider’s events arrive more than once and must not be applied twice. Solving one does nothing for the other, and teams routinely believe they have done both.
The subtler failure is a key derived from the attempt rather than the business fact: a fresh value per attempt is a nonce sitting in a key header, and the retry charges again. The inverse is quieter — one key reused across operations that are not the same operation. A replayed key returns the first request’s stored response, so the object you fetch back carries the earlier amount or currency while its status looks healthy. A status check passes; only a field-by-field comparison against what you intended catches it.
Backwards edges nobody wrote down
Between intent and settlement there is a state machine; most integrations have one only in the sense that the code implies it. The provider emits its own vocabulary; your app stores its own. Every provider state must map to exactly one of yours, and an unmapped state defaults silently — in practice, to “has access”. Then the transitions: which moves into each state are legal, and which states are terminal.
Payment webhook delivery is not ordered, and every payload is a snapshot from its own moment. Without a written transition table, a late event carrying an old snapshot walks a dead record backwards into life, and nothing treats it as an error. Reading an intent status as settlement is the same family of mistake: entitlement granted against something that never completed.
Test mode is not coverage
The fourth surface is the test/live split. Test mode exercises the shape and hides the production behaviour: real authentication challenges, issuer decline codes, disputes, live-only configuration with its own signing secret, anything a test clock compresses. Asking for the written list of what test mode cannot exercise is itself the check: if the list does not exist its length is zero, and zero is not a plausible count of behaviours that differ between test and live.
What follows for anyone wiring up money
The principle is narrow and it does the work: the only source that can confirm a payment state is the one that owns the money. Your logs are your own account of your intent; the delivery log is the provider’s account of its sending. Neither is the state.
So this is a read-back task, not a logging task. Reconciliation runs both ways, because each direction finds a different disaster: your paid records against the provider’s settled objects finds access granted without money, the reverse walk finds money taken without access. The output is numbers, not a verdict — two drift counts and the ids behind them. Zero and zero passes; anything else is an open item with an owner.
The runnable procedure — the surfaces in order, each with what to detect and how to recover, and the checkpoint that lets someone outside the room re-derive the verdict — lives in the member library.
How we know
What the provider documentation says
None of the four surfaces above is a secret. The large providers document them, usually on pages people skim once during setup.
Start with duplicates. Stripe’s webhook guide says endpoints might occasionally receive the same event more than once, and suggests you guard against duplicated event receipts by logging the event IDs you have processed and skipping the ones already seen. The same page warns that Stripe does not guarantee the delivery of events in the order they were generated. Stripe’s subscription guide gives the safer habit: after receiving invoice.paid, retrieve the associated subscription and confirm that its status is active before extending the customer’s access. In other words, fetch the current state from the provider. Do not rebuild it from event order.
Signature checks are also documented as fragile in a specific way. Stripe’s verification takes the raw request body, the Stripe-Signature header and the endpoint secret, and its troubleshooting page says the most common cause of failed checks is using the wrong endpoint secret. For example, the endpoint secret used by the Stripe CLI for local forwarding differs from the one for a Dashboard-managed endpoint. That is a small version of the test/live split described above.
PayPal’s documentation is explicit about retries. Any non-2xx response makes PayPal reattempt delivery up to 25 times over the course of 3 days. The inverse is the problem this article describes: return 2xx too early and that schedule stops.
Adyen frames the acknowledgment as a contract. Its docs say you must acknowledge every webhook event with a successful HTTP response status code, and its overview describes the order of work as accept, store the message, and process the contents of the message. Store before process is the key word order. An acknowledgment sent before the event is durably stored is the lost-event case from the first surface.
Outbound idempotency has its own fine print. Stripe’s API reference says it saves the status code and body of the first request made with a given key, and that the idempotency layer compares incoming parameters to those of the original request and errors on a mismatch. Keys can be pruned once they are at least 24 hours old, after which a reused key is treated as a new request. A key does not protect a retry that happens days later.
Stripe’s fulfillment guide covers the inbound side directly. Your fulfillment function should handle being called multiple times with the same Checkout Session ID, because both the webhook and the customer’s landing page can trigger it.
How this matches our hardening pass
The documentation and the incident point the same way. Duplicates and out-of-order delivery are expected behavior, so a handler that assumes otherwise is wrong even when every delivery succeeds. Acknowledgment is a promise to the provider that you have the event, and the provider stops retrying once you make it. And the provider’s API, not your logs or the delivery history, is the one place the current state of a payment can be read.
That is why the checks in this article are counts and re-fetches: distinct event ids on both sides, an object count after a deliberate duplicate charge in test mode, and a field-by-field comparison of each created object against what was sent.
A checklist before you trust a payment webhook
- Write the event id to an append-only store before returning 2xx.
- Deduplicate on the event id, and on Stripe also on object id plus event type, since the docs say that two separate Event objects are generated for one change in some cases.
- Verify the signature against the unmodified raw body, using the live secret in production.
- Never infer current state from event order. Re-fetch the object from the provider.
- Derive outbound idempotency keys from the business operation, and remember they expire.
- Make fulfillment safe to run twice for the same payment.
- Write down what test mode cannot exercise.
- Reconcile in both directions and report the two drift counts.
More questions
Can I use the event’s created timestamp to put webhooks in order?
Not reliably. Stripe’s docs say delivery order is not guaranteed and its subscription guide recommends retrieving the associated subscription before acting on invoice.paid. Treat each payload as a snapshot and read the current state from the provider.
If I return 200 and process the event later, will the provider retry when my processing fails?
No. Retries are triggered by your response, not by your outcome. PayPal, for example, retries on any non-2xx status code, so once you have acknowledged, recovery is your job.
Grounded in: our own hardening pass over a live payment integration on 2026-09-04, opened by a report that the provider dashboard and our database disagreed about who had paid. Verified: distinct event ids in the provider’s delivery log counted against distinct event ids in our own arrival ledger for one window; the same logical charge issued twice in test mode and the object count read back from the provider; every created object re-fetched by id and compared field by field against what we sent. No success response was accepted as evidence. The runnable procedure lives in the member library.