field notes
Short notes from real engineering work: one decision, one incident, one reversal at a time. Each one names the shape of the problem, the tension that made it hard, the decision, and what it cost. The internals stay out — the codename is the hint.
If you have five minutes rather than an afternoon: the three case studies condense this material to one problem, one decision and what the decision cost. The notes below are the record those studies cite — and the projects the codenames belong to are on the front page.
The last thing one line of inquiry produced was not an answer. It was a record: a written statement of what had been established, what had been ruled out and why, and what remained open. The cycle that files it labels the document a…
The operation had been running many workers in parallel for a long stretch when all of them stopped at the same moment: memory exhaustion, the whole set at once, not one worker's bad day but a shared ceiling reached by the sum of everything…
For one derivation that a great deal depended on, the record contains a second worker named as the backup executor. Not a reviewer, not a checker: an executor, able to produce the same object independently if the first owner stopped. The…
A parallel operation has one chronic failure: it cannot tell a problem that is hard from a worker that is stuck. Both look identical from outside, with no output, no error and no signal. The record here shows the coordinator resolving that…
Several points in this history are marked by a small object landing in the record: something computed from a handful of instances, together with the rule that produced it. Before that, the corresponding argument existed as reasoning in…
Most of the statements this operation could make were true on the cases it had looked at. That is not a claim, it is a coincidence waiting to be reported as a result. The discipline that kept the difference visible was a convention about…
At some point a whole line of inquiry was taken off producing new results and pointed at other workers' outputs instead, re-checking them on sampled cases. The record shows the reassignment in a single clause, and it reads like logistics.…
When one route establishes something, the obvious next question is whether to believe it. The answer here was to establish it again from a direction that shares no machinery with the first: a separate derivation, run separately, recorded as…
One route was pushed to the end of its useful life and produced a failure. That sentence would be the end of the story in most operations. Here the failure arrived with something attached: a precise statement of what blocks the route, tight…
A worker spent part of the run pursuing a suspected correspondence between two separately defined features of the objects under study, on the grounds that if it held it would fold two hard problems into one easier one. It did not hold. The…
The operation had many workers and one shared picture, and the shared picture had to be rebuilt continuously. The mechanism was a numbered cycle. Work was reconciled into the shared record under a counter that only ever increases, and most…
An open question is the easiest thing in a research operation to get wrong in a way nobody notices. The failure is rarely a wrong answer on the day. It is contamination: a dead end in one line of inquiry quietly becomes an assumption in the…
Incident review is where a team's honesty is easiest to perform and hardest to prove. A good write-up is not the one that reads well. It is the one whose follow-up items are still recognisable a month later, each with an owner, with…
The most expensive failure in a payment path is not an outage. It is a system that believes it has been paid. A test gateway left reachable in a live configuration, a placeholder credential that authenticates nothing, a fallback that reads…
A snapshot policy is a promise about the future. It is also configuration that runs unattended, which means its failure mode is silence: nothing errors, nothing alerts, and storage grows.
Every retention policy is a promise to delete, and deletion is the one operation you cannot review by reading it. A predicate with the wrong comparison either keeps everything forever or removes a row that mattered, and both look identical…
Dependency risk rarely announces itself. It arrives as a default: a client library with its optional features left on, dragging in a platform-native cryptography library and a second encrypted-transport stack through bindings that need…
Most systems have an audit trail in the sense that logs exist somewhere. Very few can answer, for one specific record, which person changed it and what the value was beforehand. That question decides disputes, and it is nearly always…
Account recovery is usually the weakest path in an authentication system, because it is the one place where a stranger holding a link can obtain an account. The reset flow that landed treats that link as a credential, with every property a…
The cheapest way to hold a login is a signed token the client carries. It is also the hardest to take back: revocation becomes a blocklist, and a blocklist is a second store with its own lifetime problem. The platform had token plumbing…
method
Not a changelog entry and not a repository summary. One decision, one incident, one reversal, or one hard-won invariant — the thing an engineer would tell you at a whiteboard.
Prefer records with dates and root causes? Those are on incidents. Live numbers are on stats.