field notes
Short notes from real engineering work: one decision, one incident, one reversal at a time. Each one names the shape of the problem, the tension that made it hard, the decision, and what it cost. The internals stay out — the codename is the hint.
If you have five minutes rather than an afternoon: the three case studies condense this material to one problem, one decision and what the decision cost. The notes below are the record those studies cite — and the projects the codenames belong to are on the front page.
A rebuild you have never performed is a hypothesis. The proof is a reset cycle: stop every workload, remove it, resubmit from the repository, and check that the services come back and answer. Running that deliberately, on a quiet day,…
Building a machine from an existing image is the cheapest way to add capacity, and it silently copies identity along with the operating system. One member joined the group carrying the identifier of a machine that was already a member. The…
Hardening runs once, unwatched, on a machine whose only access path it is about to change. That makes it the most dangerous program in the estate: it is the change that removes the ability to make changes. A machine that closes its…
Every provisioning pipeline is destructive on purpose: it partitions, formats and overwrites, then asks the machine to become something else. The trap is ordering. The bootstrap the pipeline needs has to be in memory before the first…
method
Not a changelog entry and not a repository summary. One decision, one incident, one reversal, or one hard-won invariant — the thing an engineer would tell you at a whiteboard.
Prefer records with dates and root causes? Those are on incidents. Live numbers are on stats.