AI-generated. Written by an automated system, checked by independent automated gates, and published without human editorial review. How we label and verify.
Scene: the job that died
A scheduled job in our pipeline failed partway through — the kind of failure that normally means the work is just gone. Being able to recover it is what saved us.
The job stopped. Logs showed a timeout. The dashboard marked it as failed. That normally means rerun, rebuild, explain to stakeholders. This time it was different.
Instead of disappearing, the failed item dropped into a separate list where failed work waits with its inputs kept, so it can be tried again. The system recorded the exact inputs, the step that failed, a timestamp and the error text. Nothing was deleted. Nothing was silently swallowed.
What we tried and what happened
We had already built one small mechanism: any item that errored out was written to a separate list with its inputs kept intact. We call that the holding list. When the job failed this time, the item landed there.
The next morning we replayed the held item. It completed cleanly. The original failure had been a temporary timeout, not a real problem with the work.
That replay was the turning point. We watched the job finish. The same inputs, the same steps. Success.
Before that moment our posture had been to treat failure as final. We assumed a failure meant the work was lost. We assumed there had been a bug, or a corrupt file, or some unrecoverable state. So we coded for alerting, rebuilds and blame. That wastes time. And work gets discarded.
How to recover from automation failure: make it a decision
After the replay we changed a rule. No automated job is allowed to fail silently. Every failure has to land somewhere a person or a retry can find it. That became our operational law.
We document three concrete things that showed us this mattered.
First: the failed item that landed in the holding list finished when replayed the next day. The problem had been a transient timeout. Recovery proven live.
Second: when we scanned the holding list over two weeks we found most entries healed themselves when retried. Patterns emerged quickly. Network hiccups. A downstream API rate-limit timeout. Brief infrastructure blips. These were temporary.
Third: we had also built the write-to-list step so the inputs were preserved exactly. That let us verify that the replay was running the same work, not a patched version. That was crucial for trust; we could attribute the difference to environmental conditions rather than silent changes to the task.
How the holding step behaved
Concrete detail: the holding list stores the original inputs as a serialized record, the step identifier, the error string, and a retry-count. That retry-count blocks infinite loops. We keep that detail here to explain how the system knows whether to try again. This is a taste, not a full build plan.
Operationally the holding list does two jobs. One, it acts as a buffer so the item is not lost. Two, it creates an auditable trail so someone can glance at it and decide: retry, escalate, or discard. That person is a human operator who looks only at the entries that show permanent failures.
A rule that fell out
From those moments a simple rule emerged. Every failure must be visible. If a system swallows a failure, real work disappears. If a system holds a failure, most of it comes back on replay.
We wrote that rule into our operations: hold anything that errors out; record inputs; let humans triage only the permanently broken entries; allow automatic replay for transient cases. It sounds obvious. But we had to see it several times before we trusted it.
When replay is wrong
Not every failure is transient. We found entries that would have failed forever if retried. Bad input. Invalid files. Misconfigured parameters provided by a client. These needed a human to glance at the holding list and throw the truly broken ones away.
We learned to put a short note with each held item describing what likely failed. That saved time. A human could delete or correct the ones that were genuinely garbage without trying endless retries.
One concrete counter-example: a batch from a partner repeatedly failed because a date field was misformatted. We retried it twice and it failed both times. It stayed on the holding list until someone inspected the inputs, fixed the date format, and resubmitted. Without the hold it would have been logged as ‘failed’ and the work lost.
Why this worked
Mostly, failures were environmental. We were hitting timeouts and temporary network errors. Those do not mean the work is wrong. They mean the environment is flaky. Holding and replaying gives the environment time to recover.
Second, preserving inputs lets us test whether the fault is with the data. When we replayed the item that had timed out, we confirmed the inputs were still valid. That distinction guided our next actions—investigate infra when many items time out, investigate inputs when the same item fails repeatedly.
Third, a visible holding list changes human behavior. Operators stop reflexively discarding work. They can triage quickly. That reduces waste.
We did not invent a magic fix. We changed how failures are treated. The system became resilient because we stopped assuming failure equals finality.
Questions we got
How often did replays fix the problem?
In our scan of two weeks most held items succeeded when retried. We did not measure a single percent in public notes here, but the pattern was clear: a majority. The concrete incident where a scheduled job completed after a morning replay is the clearest proof.
Won’t the holding list just become a junk pile?
It can, if nobody looks. That’s why entries include a short human-readable note and a retry-count. We also require a person to glance periodically and discard truly broken inputs. The holding list is an operating queue, not a long-term dump.
Can automatic retry cause harm?
Yes, if retries hide systemic issues or multiply side effects. We limit retries with a count and we record attempts. When the same item fails repeatedly a human intervenes. That mix of automated retry plus human triage prevented repeat damage in our pipeline.
Difference, plain: lost means gone. Held means recoverable. That is what a recoverable failure bought us.
Sources: our own content pipeline experiments and operational records from the team that ran the job and replayed the held item.
What AWS and Google Cloud documentation say about holding failed work
The holding list in this article has a standard name in messaging systems: a dead-letter queue. Both major cloud providers document the pattern, and their descriptions line up closely with what we saw.
The Amazon SQS developer guide describes dead-letter queues as a place for messages that are not processed successfully, and says they are useful because you can isolate unconsumed messages to determine why processing did not succeed. The guide also explains the retry limit. A redrive policy sets maxReceiveCount, which is the number of times a consumer can receive a message before it moves to the dead-letter queue. AWS warns against setting it too low and advises to set the maxReceiveCount high enough to allow for sufficient retries.
Google Cloud Pub/Sub documents the same idea as dead-letter topics. Its documentation names the two kinds of failure we ran into. There could be transient issues retrieving data needed to process a message, or a message could be in a format that the subscriber does not expect. When configuring a dead-letter topic, the console asks for a maximum number of delivery attempts as an integer between 5 and 100.
Retrying has a cost, and the AWS Builders’ Library is direct about it. The article on timeouts and retries describes exponential backoff with jitter to handle transient failures without overwhelming struggling services. A companion article on idempotent APIs points out that retrying is based on the assumption that an operation can be retried without any side effects, and that this has to be designed in, not assumed.
How the documentation lines up with our holding list
Our first finding was that the held item finished when it was replayed the next morning, because the original failure was a temporary timeout. That is the transient case the Pub/Sub documentation describes. The partner batch with the misformatted date field is the other case: a message in a format the receiver does not expect. Retrying it twice did not help, which is what you would predict for bad input.
We store a retry-count with each held entry to block infinite loops. That plays the same role as the SQS receive limit and the Pub/Sub delivery attempt limit. We also store the original inputs, the step identifier and the error string. The AWS guidance on isolating failed messages to find out why they failed is the reason those fields matter: without them, a person looking at the list has nothing to decide on.
Our answer about harm from automatic retry, that retries can multiply side effects, matches the AWS point about idempotency. A replay is only safe when running the same work twice does not do the work twice.
Checklist for a holding list that stays useful
- Write every failed item to a separate list with its exact inputs, the failing step, a timestamp and the error text.
- Add a retry-count and a hard limit, so no item loops forever.
- Space out automatic retries instead of firing them back to back.
- Before enabling replay, confirm that running the step twice has no extra side effects.
- Attach a short note on the likely cause so a person can triage fast.
- Have a person review entries that hit the limit and fix, resubmit or discard them.
More questions
Is a holding list the same as a dead-letter queue?
In practice, yes. Cloud platforms use the term for a separate place where messages that could not be processed are kept. Google Cloud calls its version a dead-letter topic, also known as a dead-letter queue.
How many retries should we allow before a person looks?
There is no single number. Google Cloud Pub/Sub accepts a value between 5 and 100 delivery attempts. AWS only advises setting the limit high enough to allow for sufficient retries, so pick a value that fits how long your typical transient faults last.
How we know
The factual claims in this article come from our verification store — each with a source type, a confidence label and a reference. The method is documented on How we know.
– When we replayed the held item the next morning it completed cleanly. The original failure had been a temporary timeout, not a real problem with the work. | source: product testing | conf: strong | ref: ebizapple first-hand — recovery proven live
– The turning point was realising most of our failures were temporary. Treat them as permanent and you lose real work; hold them and replay, and most heal themselves. | source: first-hand experience | conf: strong | ref: ebizapple first-hand — decision_rule
– We had built a dead-letter step: anything that errors out is written to a separate list with its inputs kept intact, so it can be retried later instead of silently lost. | source: product testing | conf: strong | ref: ebizapple first-hand — the mechanism
– Not every failure should be retried. A few were genuinely bad input that would fail forever, so the holding list also needed a person to glance at it and throw the truly broken ones away. | source: first-hand experience | conf: observed | ref: ebizapple first-hand — counter-case
– A scheduled job in our pipeline failed partway through, the kind of failure that normally means the work is just gone. This time the failed item dropped into a holding list instead of vanishing. | source: product testing | conf: strong | ref: ebizapple first-hand — the incident