field notes
Short notes from real engineering work: one decision, one incident, one reversal at a time. Each one names the shape of the problem, the tension that made it hard, the decision, and what it cost. The internals stay out — the codename is the hint.
If you have five minutes rather than an afternoon: the three case studies condense this material to one problem, one decision and what the decision cost. The notes below are the record those studies cite — and the projects the codenames belong to are on the front page.
A small service here does two jobs on one timer. It restates, every cycle, that its node is still the one in charge, because the record it maintains expires if it goes quiet. It also tells a human when that status changes. For a long time…
A register of written decisions, checked automatically against the machinery meant to implement them, produces one of two useless results unless you are careful. Either it reports dozens of gaps that are not defects at all — decisions taken…
The pipeline's first step took the disks out of their array. Everything else depends on that step, and it had an effect nobody had written down: the running system's root lived on those disks, so removing them from the array left the root…
Two identical disks, one operating system, and a naming scheme assigned by the order in which the hardware answers at boot. This is the ordinary condition of a machine, and it means every script that names a disk is guessing. The guesses…
One component was removed from a machine's definition outright, along with the packages that came with it. The change was small, made sense at the time, and survived about a day. Two commits later the same component was back, with two…
Reproducible builds assume an input set larger than the thing being built. Here that input set — the entire closure of what a machine needs, kept so a rebuild does not fetch the world again — lives on one host, and the build runs on that…
The last stage of building a machine image is giving it to whoever will install it. Here the installer is the hosting provider's own automation, running against hardware that has not been commissioned, holding no credentials of ours, and…
A configuration repository here contained, inside itself, a second copy of a repository. Everything looked right: the directory had files in it, the machine ran, the definitions loaded. But the history did not record the files. It recorded…
A cluster has to be founded exactly once, and the founding step cannot tell whether it is first or merely early. Here the first node would bring up the group's own membership rules and access policy before the other nodes could be reached —…
Installation scripts are written on one machine and run on another, and the difference is rarely written down. On this hardware two identical disks present themselves in an order that differs between installations, so a script that names a…
Replacing a machine's operating system means writing over the only thing that knows how to start it. The boot configuration is not part of the image being installed; it belongs to the machine and to the firmware that reads it. In this work…
An installation image is a snapshot of a system taken at a moment nobody records, and the machine that downloads it inherits that moment. The first thing the bootstrap does here is move the installed package set forward to current, which is…
A clean-up was scheduled to retire a set of working copies and the branches behind them. Somewhere in that set were result artifacts that had never been merged. They existed only on a branch, inside a working copy due for disposal, and…
A scheduled step began failing intermittently on a directory listing during the window when another process was swapping that directory. The first response was the pragmatic one: catch the failure, log it, exit successfully. The alerting…
A daily step selected its working day by taking the calendar date before today. On most days that is right. On the day after the weekly close it is wrong by construction: the chosen date has no data and never will. The step ran, found…
An independent reviewer was set to attack a finished body of work rather than describe it. The output was graded: findings of the highest severity, findings below that, each with a precise location, each either fixed or left visibly open.…
An optimisation over a bounded grid returns the best point it visited. If that point sits at the boundary of the search, the result is not an optimum; it is a statement that the search ran out of room, and the two are indistinguishable…
Rebuilding the whole record from scratch on every cycle is honest and expensive, and everything downstream waits on it. The alternative — extend the tail by reading only the new rows — has one property to recommend it and one large risk:…
A published feed carried a column meant to describe the protective distance for each row. The column existed, was typed, was documented, and for every row carried a zero. One consumer read the zero and correctly attached no protection;…
Several guards in this system exist only to refuse: a configuration whose fingerprint does not match the signed one, a start request that did not come from the schedule, a pinned specification older than the binary that would act on it.…
method
Not a changelog entry and not a repository summary. One decision, one incident, one reversal, or one hard-won invariant — the thing an engineer would tell you at a whiteboard.
Prefer records with dates and root causes? Those are on incidents. Live numbers are on stats.