Twelve parallel agents finished a harvest across twelve partitions of a 1 050-record corpus, and all twelve reported success. The data was wrong. Thirty-nine records belonging to one partition had been written into another partition’s output under the wrong label, five valid records had been discarded as malformed, and no worker’s report mentioned either. The count that came out of the run looked like a count that could be right. That is the entire difficulty: a plausible number does not announce itself as wrong.
What we assumed about parallel agents
We assumed the hazard of a fan-out was throughput — a worker dying mid-slice, a rate limit, a page that never returns. We built against that. Every worker was told to write its own output file named after its partition, and every worker did. The isolation we designed held.
What we had not thought about was the namespace nobody declared. Several agents independently decided they wanted a helper script, and independently chose the same obvious path for it. One rewrote another’s copy while that one was running. The output files never collided; the scratch space did, and it carried the collision straight into the labelled data. Shared mutable state does not need to be designed as a coordination channel to become one; it only needs two workers reaching for the same name at once.
We had also brought a number with us. The source published its own index, and that index said 1 048. The store actually held 1 050. Had we reconciled against the published figure we would have been off by two and confident, because 1 048 looks entirely reasonable.
Why a worker saying “done” means almost nothing
Each worker sees one slice. It has no view of the other eleven and no way to compare its slice against anything. When it reports success it is making one claim, and the claim is narrower than it sounds: I did not personally encounter an error. That is a statement about the inside of one context.
The errors that survive a harvest are precisely the ones that cannot be seen from inside one context. A collision needs two workers to be visible. A parser rule that is wrong is wrong identically in every partition, so no partition looks anomalous next to the others. Containers returned by a listing alongside real records look like records to whoever is holding only records. Twelve honest reports summed to a false one, and each individual report was accurate.
The parser failure is the sharper version. Five records came back flagged as invalid JSON. The data was fine; the line splitter was wrong, because Python’s str.splitlines() also breaks on vertical tab, form feed and a couple of Unicode separators that occur inside ordinary prose. The records were destroyed by the thing whose job was to read them, and the destruction was reported as a property of the source. Every verdict of “the data is malformed” is a hypothesis about your own code until somebody opens one and looks.
A count is the one answer with no symptom
When the deliverable of a job is a number — records indexed, fields extracted, pages covered — being wrong by a few produces nothing a person would notice. A crash is loud. A timeout is loud. A total that is 1 011 instead of 1 050 renders identically, sorts identically, and reads as a finished result. Wrong-by-a-few is the only failure mode that arrives looking exactly like success, and parallel agents are the machine that manufactures it.
This is why the check that matters compares two numbers produced by separate means. A count derived from the harvest, checked against the harvest, always agrees — that check is indistinguishable from not running one. What breaks the loop is a cheap listing pass taken before the fan-out, one that returns identity without payload: it costs almost nothing and yields a per-partition figure you did not generate. In our run the discrepancy surfaced from exactly there, and from nowhere else.
What follows for anyone fanning work out to parallel agents
Reconciliation is not a cleanup step that happens after the harvest. It is the part that converts a pile of files into something you can act on. Everything before it is raw material with an unknown error rate, and the run is not shorter without it — it is only unfinished in a way that will not be discovered.
Two consequences follow. Keep the model out of the data path: workers fetch and dump, and every decision with a right answer happens afterwards in code you can re-run, so a bug that would have cost a re-harvest costs seconds instead. And key everything on an identifier that belongs to the record rather than to where it was found, because under concurrency pages overlap, and a duplicate you can collapse is not a problem.
The machinery has a floor. Below roughly a hundred records it costs more than it saves, and reading the corpus in one pass is the correct answer.
The runnable procedure — the ordering, the guards and the checkpoint that makes a harvest verifiable by someone who was not there — lives in the member library.
How we know
Grounded in: our own harvest of a 1 050-record skill corpus on 2026-08-16/17, twelve concurrent agents across twelve partitions. Verified: 1 056 raw ids reduced to 1 050 indexed records after 6 containers were dropped and 39 label conflicts resolved by rule, with a two-way id diff on the largest corrected partition returning 164 in the source and 164 in the index and nothing in either direction. The runnable procedure lives in the member library.
Leave a Reply