field notes TROUPE

the binary replaced under the running process

Upgrading a long-running supervisor is two problems wearing one coat. The first is putting the new artefact on the machine without ever executing a half-written file. The second is re-pointing: getting the process that is already running to become the new thing.

The first problem has a good answer, and it was adopted after a worse one. Install by atomic rename into the final location, so the name always refers to a complete file. Between installs there was a window in which a partially written file occupied the name, which is only a small window if nothing looks through it.

The second problem is where the surprises live. A running process holds the file it was started from. Replacing that file by rename does not change what the process is executing; it leaves the process bound to a file that no longer has a name. When the process then tries to re-execute itself, the path it was launched from points either at the new artefact or, on one platform, at a name that resolves to a deleted file with a suffix appended.

That produced a relaunch that could not find itself. The fix belongs upstream, where the entry point is chosen, so it was prepared as a reproduction with a proposed change attached rather than as a local workaround that every operator would have to remember.

The operational lesson is that an install is not finished when the file is in place; it is finished when the running process reports the new artefact.

What this proves: we found where the seam between packaging and process identity actually is, and we pushed the fix to whatever owned the seam rather than pasting over it.

drawn from: commit cluster

Every note carries a codename and one idea. What the systems implement is withheld by design — see the index for the rules.