field notes
Short notes from real engineering work: one decision, one incident, one reversal at a time. Each one names the shape of the problem, the tension that made it hard, the decision, and what it cost. The internals stay out — the codename is the hint.
If you have five minutes rather than an afternoon: the three case studies condense this material to one problem, one decision and what the decision cost. The notes below are the record those studies cite — and the projects the codenames belong to are on the front page.
A handshake can complete while carrying nothing. Our relay had been built in stages, and one stage — the branch that fetches a peer's pre-published one-time handshake key and forwards it to the requester — was still a stub. The stub…
This is the note about the day we caught ourselves testing the wrong party. The claim under test was that two peers behind different kinds of address translation could find each other and hold a direct session. The test passed, repeatedly,…
Every roadmap wants to ship the interesting thing first. Ours could not, because the interesting thing was a cryptographic upgrade that invalidates every existing message shape at once: new key material, a new handshake, new frame variants.…
Two nodes that must talk to each other cannot both be upgraded at the same instant, and a peer-to-peer fabric has no maintenance window in which the whole network is quiet. Every wire format we ship has to be readable by parties still…
The early position was build, and it was argued with conviction: wrap an existing agent runtime, then fork it, then accumulate local changes. The strategic document of that period insisted the thing was a different category rather than a…
A command failed with a timeout. Not an error and not a rejection: an immediate, universal timeout. The obvious reading is the network, and the obvious reading was wrong. The settings loader had a default value that only applied while…
Running a long verification suite inside the session that is being verified is a race with a known winner. The session restarts, the background job dies with its parent, and the only record is a log that stops mid-sentence. It looks like a…
The first architecture was a relay. Every message travelled from a worker, through a coordination service, out to whoever was listening. It is the obvious shape, and it has a property that is easy to miss until it is expensive: the…
Every design that puts a broker between a worker and the world inherits a question it usually answers late: what does the worker do when the broker is gone? The tempting answer is that the broker is always there. The useful answer is a…
The most dangerous capability in an orchestration platform is a command that reaches production, and the most dangerous version of that capability is one that is easy to run. Cluster submission tools are exactly that: a single verb away…
Multi-agent rooms are usually modelled as chat: a place to broadcast text to subscribers. This design made the room the unit of authority instead. A worker's address inside the coordination layer is its identity. Which rooms it can be heard…
Upgrading a long-running supervisor is two problems wearing one coat. The first is putting the new artefact on the machine without ever executing a half-written file. The second is re-pointing: getting the process that is already running to…
An agent lifecycle has a boring default: if a worker is broken, destroy it and build a new one. It is easy, it is total, and it throws away the only expensive thing in the system, which is the worker's accumulated context.
A supervisor that can only learn what its workers choose to tell it is not supervising. The requirement was observation from the host, outside the isolation boundary, without the worker's cooperation and without the power to change what was…
An autonomous worker reports what it did. That report is the most convenient source of truth available and the least trustworthy one, because it is written by the party being judged. The platform's answer was to stop treating the worker's…
The decision that shaped everything else was to give each autonomous worker its own system container rather than a process inside a shared host. A process boundary is cheap, and it is also porous. A crew of workers that can see each other's…
The tree is assumed to be the truth about what is running, and occasionally it is not. Somebody changes a running system by hand, the change works, no failure forces the tree to catch up, and from that moment the source and the machine…
One repository held the record of decisions, the sequence that acts on them, the bootstrap that prepares a blank machine, and the definitions of the machines themselves. Nothing was wrong with it. It was organised by ownership, which put…
An operations document is read by an actor with no memory and full permission. That reader is not looking for orientation; it is about to change something, and the document is the only thing standing between its competence and the system.
An interrogation of a system's own documentation returned eighty-seven questions, every one answered, and that is where most exercises of the kind end: a document of findings, correct, complete, unread. Findings are not work. A list of them…
method
Not a changelog entry and not a repository summary. One decision, one incident, one reversal, or one hard-won invariant — the thing an engineer would tell you at a whiteboard.
Prefer records with dates and root causes? Those are on incidents. Live numbers are on stats.