AI-generated. Written by an automated system, checked by independent automated gates, and published without human editorial review. How we label and verify.
Twelve parallel agents finished a harvest across twelve partitions of a 1 050-record corpus, and all twelve reported success. The data was wrong. Thirty-nine records belonging to one partition had been written into another partition’s output under the wrong label, five valid records had been discarded as malformed, and no worker’s report mentioned either. The count that came out of the run looked like a count that could be right. That is the entire difficulty: a plausible number does not announce itself as wrong.
What we assumed about parallel agents
We assumed the hazard of a fan-out was throughput — a worker dying mid-slice, a rate limit, a page that never returns. We built against that. Every worker was told to write its own output file named after its partition, and every worker did. The isolation we designed held.
What we had not thought about was the namespace nobody declared. Several agents independently decided they wanted a helper script, and independently chose the same obvious path for it. One rewrote another’s copy while that one was running. The output files never collided; the scratch space did, and it carried the collision straight into the labelled data. Shared mutable state does not need to be designed as a coordination channel to become one; it only needs two workers reaching for the same name at once.
We had also brought a number with us. The source published its own index, and that index said 1 048. The store actually held 1 050. Had we reconciled against the published figure we would have been off by two and confident, because 1 048 looks entirely reasonable.
Why a worker saying “done” means almost nothing
Each worker sees one slice. It has no view of the other eleven and no way to compare its slice against anything. When it reports success it is making one claim, and the claim is narrower than it sounds: I did not personally encounter an error. That is a statement about the inside of one context.
The errors that survive a harvest are precisely the ones that cannot be seen from inside one context. A collision needs two workers to be visible. A parser rule that is wrong is wrong identically in every partition, so no partition looks anomalous next to the others. Containers returned by a listing alongside real records look like records to whoever is holding only records. Twelve honest reports summed to a false one, and each individual report was accurate.
The parser failure is the sharper version. Five records came back flagged as invalid JSON. The data was fine; the line splitter was wrong, because Python’s str.splitlines() also breaks on vertical tab, form feed and a couple of Unicode separators that occur inside ordinary prose. The records were destroyed by the thing whose job was to read them, and the destruction was reported as a property of the source. Every verdict of “the data is malformed” is a hypothesis about your own code until somebody opens one and looks.
A count is the one answer with no symptom
When the deliverable of a job is a number — records indexed, fields extracted, pages covered — being wrong by a few produces nothing a person would notice. A crash is loud. A timeout is loud. A total that is 1 011 instead of 1 050 renders identically, sorts identically, and reads as a finished result. Wrong-by-a-few is the only failure mode that arrives looking exactly like success, and parallel agents are the machine that manufactures it.
This is why the check that matters compares two numbers produced by separate means. A count derived from the harvest, checked against the harvest, always agrees — that check is indistinguishable from not running one. What breaks the loop is a cheap listing pass taken before the fan-out, one that returns identity without payload: it costs almost nothing and yields a per-partition figure you did not generate. In our run the discrepancy surfaced from exactly there, and from nowhere else.
What follows for anyone fanning work out to parallel agents
Reconciliation is not a cleanup step that happens after the harvest. It is the part that converts a pile of files into something you can act on. Everything before it is raw material with an unknown error rate, and the run is not shorter without it — it is only unfinished in a way that will not be discovered.
Two consequences follow. Keep the model out of the data path: workers fetch and dump, and every decision with a right answer happens afterwards in code you can re-run, so a bug that would have cost a re-harvest costs seconds instead. And key everything on an identifier that belongs to the record rather than to where it was found, because under concurrency pages overlap, and a duplicate you can collapse is not a problem.
The machinery has a floor. Below roughly a hundred records it costs more than it saves, and reading the corpus in one pass is the correct answer.
The runnable procedure — the ordering, the guards and the checkpoint that makes a harvest verifiable by someone who was not there — lives in the member library.
How we know
What the research and documentation say
Anthropic’s engineering write-up of its Research feature describes an orchestrator-worker design in which a lead agent coordinates the process while delegating to subagents that run in parallel. It is direct about what went wrong early on. Each subagent needs an objective, an output format, guidance on the tools and sources to use, and clear task boundaries. Without detailed task descriptions, the post says, agents duplicate work, leave gaps, or fail to find what they need.
The MAST paper looks across many systems instead of one. Its authors note that for multi-agent setups, performance gains on popular benchmarks are often minimal compared with single agents, and their analysis of what goes wrong identifies 14 unique modes of failure. The point for anyone running a fan-out is that the failures were common enough to need a taxonomy.
The duplicate problem has an older answer in distributed systems. Amazon’s Builders’ Library article on retries explains how a retried create call can produce a second resource, and describes using a unique client request identifier so a repeat of the same request is recognized as a repeat.
Shared scratch space has a documented fix too. Python’s tempfile documentation says of mkstemp() that There are no race conditions in the file’s creation, provided the platform implements exclusive open correctly. The older mktemp() carries the warning that Use of this function may introduce a security hole, because another process can claim the name first.
The parser fault is documented as well. For str.splitlines(), the Python reference says This method splits on the following line boundaries and that the boundaries are a superset of universal newlines.
How this lines up with our harvest
Our partitions did what Anthropic’s advice asks for. Every worker had its own output file and a clear slice. The collision came from a place no one had given boundaries to: a helper script that several agents placed at the same obvious path. A unique temporary name per worker, created exclusively, would have kept one agent from rewriting another’s copy mid-run.
The five records reported as invalid JSON were a splitter problem, and the reference describes that splitter’s behavior in plain terms. Splitting on \n only, or reading line by line with an explicit newline, keeps vertical tab and form feed inside the record where they belong.
The idempotency idea is why this article says to key everything on an identifier that belongs to the record. Overlapping pages under concurrency then produce duplicates that collapse by rule, which is how 39 label conflicts were resolved and 1 056 raw ids became 1 050 indexed records.
None of these sources removes the need for an independent count. A worker’s “done” is a claim about one context. The discrepancy in our run surfaced only from a cheap listing taken before the fan-out, and from nowhere else.
A checklist before fanning out to parallel agents
- Give each worker its own output path and its own scratch directory. Treat any shared path as a coordination channel.
- Create temporary files with exclusive creation, never by picking a name and writing to it.
- Take an identity-only listing of the source before the fan-out, with counts per partition.
- Key every record on its own identifier, not on where it was found.
- Keep the model out of the data path. Workers fetch and dump; code decides.
- Open a sample of anything flagged malformed before blaming the source.
- Diff ids both ways against the listing. Do not compare against a published index.
More questions
Do parallel agents need a shared workspace at all?
Often not. If each agent writes only to paths derived from its partition, the only shared state left is the source and the final merge. Anything else that two workers can both reach should be treated as a place where they will collide.
Why is a published index count not a good check?
Because it was produced by someone else at another time. In our run the published figure was 1 048 while the store held 1 050. Reconciling against it would have produced a confident answer that was off by two.
Grounded in: our own harvest of a 1 050-record skill corpus on 2026-08-16/17, twelve concurrent agents across twelve partitions. Verified: 1 056 raw ids reduced to 1 050 indexed records after 6 containers were dropped and 39 label conflicts resolved by rule, with a two-way id diff on the largest corrected partition returning 164 in the source and 164 in the index and nothing in either direction. The runnable procedure lives in the member library.