Never Trust the 200: A Verification Habit We Built the Hard Way

AI-generated. Written by an automated system, checked by independent automated gates, and published without human editorial review. How we label and verify.

Verifying AI automation actually worked — why the green check misleads

We discovered that verifying AI automation actually worked only when we stopped trusting the system’s own “success” signal and read the end state ourselves. In one day we hit three unrelated failures: an API that returned HTTP 200 but saved a blank page; a webhook queue that showed as connected while failing to deliver for 18 minutes; and an AI judge that gave a fabricated result a passing verdict. Each dashboard said everything was fine. The actual outputs told a different story.

What happened — the three failures, concretely

1) The API call that lied. We pushed content to a content-management API. The call returned HTTP 200. The integration logs recorded success. The page created was empty. The turning point: we independently fetched the page by its ID and saw the body was missing. The moment it clicked was when we compared the request we sent with what the system stored. The request had a JSON body; the stored page had no body. Tracing headers showed the Content-Type header had been omitted in one failing client. The API still returned 200. Visible success, invisible failure.

2) The webhook that didn’t deliver. Our order system flagged the outgoing webhook queue as healthy. A synthetic test to the same endpoint succeeded immediately. But two real order events went undelivered for over 18 minutes. We only noticed when we read the downstream system’s delivery logs (separate from the queue dashboard) and discovered empty receive timestamps for the two triggers. Forcing synchronous delivery for a brief test reproduced immediate delivery. The dashboard had been reassuring while the live path silently failed.

3) The AI judge that approved fiction. An internal content-approval model ran and returned PASS on multiple generated items. The judge’s own log said “approved.” But when we fetched the actual published content from the public endpoint, we found invented citations and placeholder text the judge had not actually validated. We stopped trusting the judge when a verification step — a read of the published page — flagged the fabricated material. The judge’s PASS was self-referential; independent reading exposed the gap.

How we proved these weren’t flukes

We repeated each check with an independent read that did not rely on the originating system’s logs. For the API case we performed a GET on the resource immediately after the POST and compared the response body to the payload we sent. For the webhook we wrote the webhook payload to a separate durable storage upon arrival and inspected timestamps there. For the AI judge we fetched the published content and compared it to what the judge reported. In every case, the independent read disagreed with the system’s success signal.

The real turning point came with one small decision: stop assuming a success code equals success, and add a short independent read. The first time that read failed, the immediate difference was obvious — blank page, missing delivery, fabricated content — and it was impossible to ignore.

The rule that fell out

If the same system emits the success signal and is also the only source you inspect, you have no independent evidence the work completed correctly. Always validate by reading the end state from an independent perspective: fetch the created resource, check the downstream receive log, or pull the published output. Make that read a habit, not a one-off test. That habit is what verifying AI automation actually worked looks like, day to day.

One representative detail (a taste, not a recipe)

In the API case the symptom mapped to a single missing header: a POST without an explicit Content-Type produced 200 but the server ignored the body. That concrete mismatch — header absent, saved content absent — is illustrative. I’ll show that example to explain what we checked, but not publish the full integration code here. The tested, runnable version lives in the members’ library.

Why this works

System-level success signals are produced by the very components that can fail in subtle ways (parsing, queuing, local caches, model hallucination). An independent read exercises the actual outward-facing result and bypasses those internal blind spots. It catches cases where the error mode is “looks connected but does nothing” — the exact failure that dashboards and unit success flags miss.

A fair counter-example

We once ran an automation where the platform’s execution log, the integration’s own success flag, and an independent read all agreed immediately. That was a straightforward sync call that returned the created resource in the response, so the response itself acted as an independent read. The lesson: independent verification can sometimes be cheap when the system returns the end state directly; when it doesn’t, you must fetch it yourself.

How we know this is general

These three failures occurred across different technologies and vendors in the same working day. The common pattern — trusted success signals masking real failures — repeated with different symptoms. We validated by changing only the verification step (not the core systems) and saw consistent detection of wrongness. That repeatability is why we turned the check into a standing habit.

Practical guardrails (conceptual, not step-by-step)

Make independent reads small and frequent. Prefer fetching the created resource or checking the downstream receiver rather than relying on the originating system’s logs. Log the independent read outcome separately. When the independent read disagrees with the success signal, escalate and record both states for debugging. The tested, runnable guide with templates and checks lives in the members’ library.

FAQ

How often should I do the independent read?

Make it part of every end-to-end validation: immediately after an action completes in production-like flows, not just in periodic audits. Frequency depends on risk — higher-risk automations need the read every run.

Won’t independent reads create more load and latency?

Yes, but the cost is usually small compared with the business cost of silent failures. For high-throughput paths, sample reads or asynchronous verification with alerting on mismatches can reduce load while keeping coverage.

Can I trust third-party dashboards at all?

They’re useful for trends and alerts. Don’t rely on them as sole proof. Treat them as one input; always validate the end state independently for correctness.

Read the end state. Make it routine. That single habit turned three invisible failures into visible problems in one day.

Sources: internal debugging sessions and incidents in our content pipeline and webhook integrations documented at ebizapple.com.

What reliability engineering and the HTTP spec say

Google’s SRE book splits monitoring into two kinds. White-box monitoring reads a system’s internals, and the book credits it with catching things like failures masked by retries. Black-box monitoring is defined as testing externally visible behavior as a user would see it. A system’s own success flag is white-box evidence. An independent read of the end state is black-box evidence.

The book’s chapter on alerting describes how Google closes that gap with a tool called Prober, which runs a protocol check against a target and reports success or failure. It also notes that Prober can validate the response payload of the protocol, not just the status. That second detail is the important one. A status code alone would not have caught a blank page.

Synthetic checks have a known weakness too. The SRE Workbook warns that when an issue affects real users but not artificial traffic, the successful artificial requests hide the real user signal.

The HTTP specification is narrower than many integrations assume. RFC 9110 says a 200 response indicates that the request has succeeded. It does not promise the server understood the body the way you meant it. On missing headers, the same RFC says that without a Content-Type the recipient may either assume a media type of application/octet-stream or examine the data to guess. A server is allowed to accept a request and still not treat its body as JSON.

Retries are where verification meets safety. RFC 9110 calls a method idempotent when the effect of multiple identical requests with that method is the same as one. Stripe’s API documentation applies that to writes with idempotency keys, so that after a connection error you can safely repeat the request without risk of creating a second object.

How this maps to our three failures

Each of our three cases was white-box evidence saying yes while the outside view said no. The API returned 200 and saved a blank page; the request had a JSON body and one failing client had omitted the Content-Type header. The webhook queue showed healthy while two real order events went undelivered for over 18 minutes, even though a synthetic test to the same endpoint succeeded immediately. That is the Workbook’s warning about artificial traffic, in miniature. The AI judge returned PASS on content that turned out to contain invented citations and placeholder text.

What caught all three was the same move the SRE book describes: a GET after the POST, a separate durable record of webhook arrivals, and a fetch of the published page. None of them relied on the originating system’s logs.

Checklist for an independent read

1. After every write, fetch the resource by its ID and compare the stored body to the payload you sent.
2. Set Content-Type explicitly on every request with a body.
3. Record webhook arrivals in storage you control, with timestamps, and check them against what was sent.
4. Keep synthetic probes, but never let them stand in for real-traffic checks.
5. Read published output from the public endpoint, not from the judge’s log.
6. Use idempotency keys or upserts so a retry after a failed check cannot create duplicates.
7. When the read and the success signal disagree, store both and alert.

More questions

Isn’t a 200 response enough if the API is well built?

Per RFC 9110, a 200 indicates that the request has succeeded, as the server sees it. Whether the server stored what you meant is a separate question, and only reading the resource back answers it.

Is it safe to retry a write when the check fails?

Only if the write is idempotent. Use a method that is idempotent by definition, or send an idempotency key so the server can recognize the repeat. Otherwise a failed verification followed by a blind retry can turn one missing record into two.