field notes
Short notes from real engineering work: one decision, one incident, one reversal at a time. Each one names the shape of the problem, the tension that made it hard, the decision, and what it cost. The internals stay out — the codename is the hint.
If you have five minutes rather than an afternoon: the three case studies condense this material to one problem, one decision and what the decision cost. The notes below are the record those studies cite — and the projects the codenames belong to are on the front page.
A clean-up was scheduled to retire a set of working copies and the branches behind them. Somewhere in that set were result artifacts that had never been merged. They existed only on a branch, inside a working copy due for disposal, and…
A scheduled step began failing intermittently on a directory listing during the window when another process was swapping that directory. The first response was the pragmatic one: catch the failure, log it, exit successfully. The alerting…
A daily step selected its working day by taking the calendar date before today. On most days that is right. On the day after the weekly close it is wrong by construction: the chosen date has no data and never will. The step ran, found…
An independent reviewer was set to attack a finished body of work rather than describe it. The output was graded: findings of the highest severity, findings below that, each with a precise location, each either fixed or left visibly open.…
An optimisation over a bounded grid returns the best point it visited. If that point sits at the boundary of the search, the result is not an optimum; it is a statement that the search ran out of room, and the two are indistinguishable…
Rebuilding the whole record from scratch on every cycle is honest and expensive, and everything downstream waits on it. The alternative — extend the tail by reading only the new rows — has one property to recommend it and one large risk:…
A published feed carried a column meant to describe the protective distance for each row. The column existed, was typed, was documented, and for every row carried a zero. One consumer read the zero and correctly attached no protection;…
Several guards in this system exist only to refuse: a configuration whose fingerprint does not match the signed one, a start request that did not come from the schedule, a pinned specification older than the binary that would act on it.…
One component in the live path answered operations with a code that meant nothing. The counterparty accepted the request, carried it out, and returned a failure. The truth was visible in that counterparty's own journal, where an accepted…
A replay was run to confirm that a change behaved as intended. The replay invoked the ordinary daily routine, and the routine began with a sweep: every entry not opened by the current pass was, by that definition, a day old and due to be…
Opening a position and attaching its protection were two separate requests. The gap between them was real, of seconds order, and entirely unmeasured: nothing in the system described it, so nothing bounded it. When the question was finally…
A protection level was attached to an open position and then read back to confirm it had landed. The read-back compared the level the system had sent against the level the counterparty reported, and demanded that the two be equal. The…
A wave of work was split across parallel workers so that several implementation tasks would land at once. Both of the slots doing the primary work died on their first steps, having exhausted their working context before producing a change.…
Large generated tables had been committed alongside the code that produced them, repeatedly, over months. Nothing about any single commit was wrong. The aggregate was: the repository had grown to gigabytes, most of it result artifacts that…
Before a decision that could be taken back was taken, the taking-back was built. Two manifests were prepared, each pinning a previously committed configuration byte for byte, each applied as a single command that ran a fixed sequence: back…
A filter that decides which candidates reach the live path had been fitted offline, in one language, against historical data. It had to run in the live path, in another. Between the fit and the decision sits a translation, and a translation…
A rule was proposed that sounds like discipline: keep an execution only if its measured cost is within a small multiple of the cost the model assumed for it, and drop the rest. The premise is that costlier-than-modelled work is work nobody…
Fitting a configuration to history and then reporting what it did is a test of memory, not of method. The programme's answer is a blocked holdout: the span is cut into contiguous blocks, the tuning sees only part of it, and the remainder is…
Months of results carried a caveat that had not been checked. A stretch of the input in one season looked thinner than the rest, and thin input is a comfortable explanation for anything measured over that stretch: it explains a bad result…
A holdout run produced a headline that split cleanly by era: strong in one year, weak in the other. The split was written up as a finding about time, because time was the axis the study had been built to test.
method
Not a changelog entry and not a repository summary. One decision, one incident, one reversal, or one hard-won invariant — the thing an engineer would tell you at a whiteboard.
Prefer records with dates and root causes? Those are on incidents. Live numbers are on stats.