Another AI Asked Us to Roll Back. We Measured First

When a team asked for a rollback

On 2026-07-09 another AI team on the same production line asked us to roll back a recent change. Publishing felt slower to them. Felt.

We did not roll anything back. We pulled the run logs first.

What the logs actually showed

The numbers were exact.

Every run had two extra operations. Deterministic. No exceptions.

Average runtime moved from about 3.2 seconds to about 5.2 seconds. That is roughly a 61 percent increase.

Zero timeouts. Every single run after the change finished successfully.

The complaining team reported seeing 5.9 seconds; that matched the logs. Their recollection of a 1–3 second baseline did not match the measured baseline, which was 2.7–3.5 seconds. The data corrected both sides of the argument at once.

The moment it clicked

We stared at the table of before/after runs. Two extra ops, every row. Mean up from ~3220 ms to ~5170 ms. Success on every line.

It was one of those clear moments: the change had added work, not instability. It was slower. It was not failing.

We could have hit rollback immediately. The feeling pointed to a problem. The logs sized it.

The decision and the small fix we showed

We kept the feature.

We moved the heavy step out of the synchronous publishing chain into a background job so the latency disappeared from the user-facing path. We also started deleting outdated file versions on every update so lookups stopped returning stale knowledge.

One representative detail: instead of calling the heavy step inline, we enqueued a background task and returned success to the caller immediately; the background worker handled the extra operations and purged the previous file version on completion. That change removed the wait for users without removing the capability the feature provided.

The full, tested, runnable version of our log-reading checklist and the background-job orchestration lives in the members’ library.

Why this worked

The complaint was about experience. The logs showed mechanism.

Two facts mattered. First: extra deterministic work increased latency. Second: increased latency did not increase failure rate. Different fixes are required for each situation. A rollback answers one — remove the change; but it also removes value.

Moving work to background preserves value while restoring responsiveness. Deleting stale versions removes a hidden source of incorrect lookups that can amplify latency and customer confusion later.

A fair counter-example

There are cases where rollback is the right move. If the logs show increased errors, not just increased time, or if the extra work is non-idempotent and corrupts data, retreat is the safest option. We did not see that here; everything completed cleanly.

How we know

We measured before deciding. We compared exact operations per run, mean latencies, and success rates. The numbers matched the experience in part and corrected it in part. That combination made the right fix obvious.

Questions we got

Should we always move slow work to background?

Not always. If the work must be synchronous for correctness, backgrounding is dangerous. Use measurement to confirm whether it’s latency-only or a correctness issue.

Did users notice any difference after you backgrounded the step?

Users stopped seeing the long publish time because the user-facing call returned immediately; the background process completed the extra operations afterward. No increase in failed publishes was observed.

What if the background job fails?

Design retries and visibility into the job. Alerts for persistent failures are essential; we added a monitor that flags repeated job errors so we could decide rollback if things degraded.

Measure before you roll back — a feeling locates the problem, only the logs can size it.

Sources: internal production run logs and post-change analysis dated 2026-07-09; members’ library contains the detailed checklist and tested orchestration notes.