field notes
Short notes from real engineering work: one decision, one incident, one reversal at a time. Each one names the shape of the problem, the tension that made it hard, the decision, and what it cost. The internals stay out — the codename is the hint.
If you have five minutes rather than an afternoon: the three case studies condense this material to one problem, one decision and what the decision cost. The notes below are the record those studies cite — and the projects the codenames belong to are on the front page.
The early position was build, and it was argued with conviction: wrap an existing agent runtime, then fork it, then accumulate local changes. The strategic document of that period insisted the thing was a different category rather than a…
A command failed with a timeout. Not an error and not a rejection: an immediate, universal timeout. The obvious reading is the network, and the obvious reading was wrong. The settings loader had a default value that only applied while…
Running a long verification suite inside the session that is being verified is a race with a known winner. The session restarts, the background job dies with its parent, and the only record is a log that stops mid-sentence. It looks like a…
The first architecture was a relay. Every message travelled from a worker, through a coordination service, out to whoever was listening. It is the obvious shape, and it has a property that is easy to miss until it is expensive: the…
Every design that puts a broker between a worker and the world inherits a question it usually answers late: what does the worker do when the broker is gone? The tempting answer is that the broker is always there. The useful answer is a…
The most dangerous capability in an orchestration platform is a command that reaches production, and the most dangerous version of that capability is one that is easy to run. Cluster submission tools are exactly that: a single verb away…
Multi-agent rooms are usually modelled as chat: a place to broadcast text to subscribers. This design made the room the unit of authority instead. A worker's address inside the coordination layer is its identity. Which rooms it can be heard…
Upgrading a long-running supervisor is two problems wearing one coat. The first is putting the new artefact on the machine without ever executing a half-written file. The second is re-pointing: getting the process that is already running to…
An agent lifecycle has a boring default: if a worker is broken, destroy it and build a new one. It is easy, it is total, and it throws away the only expensive thing in the system, which is the worker's accumulated context.
A supervisor that can only learn what its workers choose to tell it is not supervising. The requirement was observation from the host, outside the isolation boundary, without the worker's cooperation and without the power to change what was…
An autonomous worker reports what it did. That report is the most convenient source of truth available and the least trustworthy one, because it is written by the party being judged. The platform's answer was to stop treating the worker's…
The decision that shaped everything else was to give each autonomous worker its own system container rather than a process inside a shared host. A process boundary is cheap, and it is also porous. A crew of workers that can see each other's…
method
Not a changelog entry and not a repository summary. One decision, one incident, one reversal, or one hard-won invariant — the thing an engineer would tell you at a whiteboard.
Prefer records with dates and root causes? Those are on incidents. Live numbers are on stats.