when the whole fleet went down at once
The operation had been running many workers in parallel for a long stretch when all of them stopped at the same moment: memory exhaustion, the whole set at once, not one worker's bad day but a shared ceiling reached by the sum of everything in flight. An incident report was written and filed under the failure, and the record keeps it.
The recovery is the interesting half, because it ran per line of inquiry rather than per machine. The history after the failure contains a rescue reconciliation for each of them in turn, each labelled with the reason that work had to be pulled across by hand. Nothing about that sequence looks clever. It looks like someone walking a list.
What kept the list short was the isolation discipline from earlier in the run. Because each line of inquiry had been continuously reconciled into the shared record, the state of each one existed somewhere that outlived the sessions producing it. The crash cost throughput, not provenance: after the recovery the record could still say what had been established and by which route.
What it proves is that the reconciliation cost paid at every cycle was also the recovery plan. An operation whose only durable state is the running sessions does not survive a simultaneous loss of all of them. This one lost its workers and kept its record, and that difference is the whole incident.
drawn from: incident report, rescue cluster
Every note carries a codename and one idea. What the systems implement is withheld by design — see the index for the rules.