field notes
Short notes from real engineering work: one decision, one incident, one reversal at a time. Each one names the shape of the problem, the tension that made it hard, the decision, and what it cost. The internals stay out — the codename is the hint.
If you have five minutes rather than an afternoon: the three case studies condense this material to one problem, one decision and what the decision cost. The notes below are the record those studies cite — and the projects the codenames belong to are on the front page.
The requirement was absolute: a replayed stream had to be byte-identical to what the live system emitted, or nothing built on the replay could be believed. It nearly was. A full-scale comparison across millions of records showed a small…
A live path produced nothing for a stretch of hours. The obvious readings were that there were no qualifying candidates, or that the feed was broken. Both were wrong, and the way the truth was found is the useful part: rather than guessing,…
Every guard added to a live path is itself a risk. A guard that misfires under load takes the system down more reliably than the fault it was built for. So a new live cost gate was added the other way round from the obvious one: it ships…
The archive had to be pulled incrementally — a day at a time, resumable, verifiable — and the only description of the wire encoding was that it existed. Published notes covered the container and the field order. They did not cover how the…
The archive is the ground truth for every later measurement, so the way it fails matters more than the way it works. The failure found was quiet. The upstream service would occasionally answer a data request with a successful, well-formed,…
An independent reviewer went through a family of published result tables looking for internal inconsistency rather than wrong arithmetic. They found one that mattered. The worst-case column carried no sign. A minimum had been formatted…
Any interesting question in a long research programme can be asked a dozen ways. Ask it a dozen ways and one will look good; that is arithmetic, not discovery. The programme's answer costs a little time every single day: a measurement is…
The system had a number it liked. It came from a cost model assembled out of a small table of assumed per-transaction costs — reasonable when written, never checked against what the system actually paid. Every optimisation downstream…
Work was done on parallel branches and collected at a central point. The collection step reported success, and it was not lying: the merge it performed completed without conflict. It also carried none of the work, and nothing in the exit…
A helper computed the moment of a daily boundary by converting between two clocks. The offset was applied in the wrong direction, which does not produce a small error: it moves the boundary by twice the offset. Every simulated cutoff landed…
Allocation and protection were written by different hands at different times, and they had drifted apart. A decision was given a size computed from one distance, and its protective exit was attached at several times that distance — so a…
The platform had already ended three items. The local state still believed all three were open. Every pass therefore re-issued a close request for them, the platform refused each one because the item no longer existed, and the pipeline…
Every restart used to cost most of a working day during which the system was running and not allowed to decide, because its rolling state had to be rebuilt from live data before the first decision was trusted. The fix seeds that state from…
The frozen specification said one thing and the deployed code computed the other. The sign of the score was inverted in the branch that acted on it, so every decision taken went the opposite way to the one specified. In the same path the…
A producer published fresh data by swapping the name that points at the current generation, and its consumers held a handle open to it. A staleness check inspected the identity of that handle in order to detect the swap. But a handle's…
Two constraints define reproducibility here, and they arrived years apart. The first is that the executable which ran must be regenerable from a pinned state of the record. So a manifest binds the built artifact's content hash to the exact…
Before a measurement is allowed to mean anything, two questions have to be answered, and both are about the input rather than the result. Does the decision set reproduce exactly as it was recorded, and what does the platform not give you.
The expensive way to close a family of mechanisms is to try variants until none of them works. It never quite finishes: there is always one more setting, one more combination, and whoever wants to keep going can always find a reason to.…
A sweep over the entire enumeration — every configuration the grid admitted, none skipped — returned nothing. Of the thousands of runs, not one survived the check against realized outcomes. On its own that is a routine negative, the kind of…
A programme that produces nothing but negative results eventually produces something that clears every gate. This one did: after a long run of closures, a candidate passed the engine check, the realized check, the mirror comparison and…
method
Not a changelog entry and not a repository summary. One decision, one incident, one reversal, or one hard-won invariant — the thing an engineer would tell you at a whiteboard.
Prefer records with dates and root causes? Those are on incidents. Live numbers are on stats.