Why a “success” checkmark was a lie we believed
We had a pipeline step that returned a clean green success. The dashboard showed the checkmark. We kept moving. For a while I told myself everything was fine.
Then someone added a very small read-back: fetch the page that step had supposedly changed and look at the field. The field was empty. Not null. Empty. The API call said success. The outcome was not there.
That moment is sharp. The request was accepted by the system — the network returned 200, the job showed complete — but the outcome we cared about hadn’t happened. The turning point: we made every step that mattered read its own result back and compare it to what it intended, and fail loudly if they did not match. After that, the green checkmark stopped being a trust fall and started being a hypothesis to test.
What we actually saw
Finding 1 — Success meant “request accepted,” not “work done.” A step flipped to success and the UI moved on. When we read the page back, the metadata field that step said it had set was empty. The success was real; the outcome was not.
Finding 2 — Silent failures hide in automation. We had several automated steps that reported success and moved on. Only when we started re-reading the actual result did we find that sometimes the thing they claimed to do had not happened. Those failures were quiet until the read-back called them out.
Finding 3 — A green checkmark you have never audited is a guess wearing a uniform. We noticed this as a pattern. If you never verify what the system produced, you are trusting a status indicator rather than the result itself. That trust can be wrong without visible alarms.
The experiment that changed how we work
We instrumented one failing path. For every step in that path we added a small verification call: after the step returned success, we queried the source of truth and asserted the expected change existed. If the assertion failed, the step raised an error and bubbled up.
The first run failed immediately. The pipeline reported two successes and then a blockage. The read-back showed exactly which success had lied. We fixed a race condition and reran. The verification passed. The pipeline finished. We could watch the exact change happen.
Two things changed for us in that run. First, silent errors surfaced at the point of failure, not three screens later when customers complained. Second, our incident mean-time-to-detect plummeted because the system itself made the mismatch obvious.
The rule that fell out
Every step that matters must read its own result back, compare it to what it intended, and fail loudly if they do not match. That is the rule. Make the system prove its work.
One fair counter-example
We learned we cannot re-read everything. Verification costs an extra call and adds latency. For trivial, idempotent actions that affect nothing critical, the cost outweighs the benefit. So we reserve read-backs for the steps where a silent failure would actually hurt outcomes: publishing content, changing billing metadata, granting access. For low-risk increments we keep the normal status checks.
How we decided which steps to verify
We asked two questions for each step: if this step silently failed, would anyone notice quickly? And would the failure cause real harm or cost to fix later? If the answers were no and yes respectively, we used a read-back. If the step only touched ephemeral logging or non-user-facing telemetry, we let the green check stand.
How we know this works
We measured two things. Errors that used to surface as customer tickets now produce an internal alert at the time of the failed verification. And the fixes that followed were narrowly scoped because the verification pinpointed the exact mismatch. Those are direct observations from our pipeline after we rolled the change into production.
FAQ
Why not just trust the status code?
Status codes and success flags tell you the system accepted the request. They don’t prove the desired change occurred. We’ve seen clean successes where the underlying field was empty. Trust the result, not the flag.
Won’t extra read-backs slow everything down?
Yes, they add calls and latency. That’s why we don’t verify everything. We pick the steps where a silent failure would cause real harm or cost. For those, the extra call is cheaper than chasing down bugs later.
How do you implement a read-back without doubling complexity?
Keep the verification small and focused. After the step completes, query the minimal source of truth and assert the specific change. If it fails, stop the pipeline and return a clear error. The pattern is simple: do, check, fail loudly.
We learned the hard way that a green checkmark can be an elegant lie. Watching the system prove its own work changed our failures from mysteries into reproducible bugs we could fix the moment they happened.
Sources: ebizapple first-hand experience and product testing observations.
Leave a Reply