field notes TROUPE

what happens when the coordination service is not there

Every design that puts a broker between a worker and the world inherits a question it usually answers late: what does the worker do when the broker is gone? The tempting answer is that the broker is always there. The useful answer is a written policy with three stages.

Stage one, reconnect with growing delays rather than a tight loop, so a restart of the coordination layer is survivable instead of self-inflicted. Stage two, keep working while reconnecting, with the local buffer holding what the worker produced so that history is not lost to the accident of a network fault. Stage three, when the retry budget is exhausted, stop pretending and fall back to local output, so the worker's stream lands somewhere a human can still read it.

The same posture appears on the consuming side, where it is harder to get right. A notification source whose file is absent is an inactive source, not an error. A single failed feed is a note, and the command still succeeds. The command fails only when every source it was asked to read has failed, because that is the only case in which an operator can truthfully be told that nothing could be read.

Marking state follows the same rule: only sources that answered are marked, and skipped sources are left untouched rather than optimistically marked clean.

What this proves: the absence of coordination was treated as a normal operating mode with its own semantics rather than as an exception. Degradation designed in advance is the difference between an outage that looks like a crash and an outage that looks like an outage.

drawn from: commit cluster

Every note carries a codename and one idea. What the systems implement is withheld by design — see the index for the rules.