the retry that taught the pipeline to skip work

A scheduled step began failing intermittently on a directory listing during the window when another process was swapping that directory. The first response was the pragmatic one: catch the failure, log it, exit successfully. The alerting went quiet, and so did the work. The step was now skipping its output whenever the race landed on it, and nothing distinguished that from a step with nothing to do.

The real answer came later from forensics, and it was structural. The swap was two renames, so the path genuinely did not exist for an instant. The durable fix is a single atomic replacement with the name flipping between two prepared copies, which removes the window instead of surviving it.

Both halves were kept, and their order matters. The immediate mitigation was a bounded retry — a handful of attempts across a span comfortably longer than the window — verified live by watching the directory appear mid-retry. The structural fix was written down as the thing that makes the retry unnecessary, never as a replacement for it.

What it proves: they know the difference between a failure they have handled and a failure they have hidden, and a successful exit that used to be a failure is treated as a warning sign rather than a relief.