field notes
Short notes from real engineering work: one decision, one incident, one reversal at a time. Each one names the shape of the problem, the tension that made it hard, the decision, and what it cost. The internals stay out — the codename is the hint.
If you have five minutes rather than an afternoon: the three case studies condense this material to one problem, one decision and what the decision cost. The notes below are the record those studies cite — and the projects the codenames belong to are on the front page.
The root volume was filling, and the obvious reading was that the machine needed more disk. The correct reading was that the machine had been arranged badly. The root filesystem was copy-on-write with duplication enabled, which buys…
Everything else trusts the identity service, which makes it the worst place to be uncertain. An audit found two kinds of uncertainty. The service was not recording sign-in events or administrative events, so the question of who changed what…
Every other service in the estate is used through software in this estate. Mail is used through a stranger's client, which means a defect arrives as "it doesn't work" from someone who cannot explain why, while the server's own logs say…
An alerting system has one job and several dependencies: something evaluates a rule, something delivers the result, something transports it, something makes a noise. Every link can be empty while the previous link reports success, and…
The convenient shape for a secret store is one credential that reads everything, held by every machine. The cost is that any machine becomes a copy of the whole keychain, and one mistaken read becomes indistinguishable from a legitimate…
The ordinary way to obtain a certificate for a name is to let the outside world fetch a token from the address that name resolves to. It is the path with the fewest moving parts, and it assumed something that stopped being true: that the…
Certificate renewal is a periodic job that writes a new file and exits. The running service keeps the old certificate in memory and keeps serving it, so the write is not the delivery. The first version of the automation finished by…
Routing used to be a file per host, with the address of every backend written into it. That works until a service moves, and then the file is wrong in a way nothing reports. The migration replaced those entries with a directory that…
A proposal arrived to make the database highly available: two copies, automatic failover, the standard shape that appears in every design document. The review before building it asked a narrower question than the proposal did. What failure…
The declarative path existed and lagged the imperative one, in the ordinary way: a change typed by hand at the moment of need, a repository updated later, or not at all. Most changes were still typed by hand, and the written record said…
There are three ways a check earns silence without earning confidence. The first is that it does not exist. Alerting covered machine health — processor, disk, memory — and nobody had asked whether the service people actually used was up, so…
A backup job is a producer with no consumer. Nothing in the system notices when the archive is empty. The job either exits zero, and the emptiness is invisible, or it fails into a log nobody reads, and the schedule keeps firing regardless.
Per-port rules look like security and behave like a promise to remember every port. Workloads get new ports when they are rescheduled, and the rule list does not grow with them, so the estate accumulates silent holes that nobody wrote down.
Many reporters, one writer. Each machine is allowed to say what it believes about itself; exactly one process is allowed to turn those statements into records. With several writers, two orderings of the same announcements produce two…
Leadership moves, and anything that hard-codes where leadership was becomes a fact somebody has to remember at the worst possible moment. A name that always resolves is not a reliable name; it is a name that can be confidently wrong, and…
For a long time the honest name for this estate was staging, and the word licensed sloppiness: try it, leave it running, fix it later. Calling it permanent changed nothing about the hardware and a great deal about the decisions. Permanent…
A rebuild you have never performed is a hypothesis. The proof is a reset cycle: stop every workload, remove it, resubmit from the repository, and check that the services come back and answer. Running that deliberately, on a quiet day,…
Building a machine from an existing image is the cheapest way to add capacity, and it silently copies identity along with the operating system. One member joined the group carrying the identifier of a machine that was already a member. The…
Hardening runs once, unwatched, on a machine whose only access path it is about to change. That makes it the most dangerous program in the estate: it is the change that removes the ability to make changes. A machine that closes its…
Every provisioning pipeline is destructive on purpose: it partitions, formats and overwrites, then asks the machine to become something else. The trap is ordering. The bootstrap the pipeline needs has to be in memory before the first…
method
Not a changelog entry and not a repository summary. One decision, one incident, one reversal, or one hard-won invariant — the thing an engineer would tell you at a whiteboard.
Prefer records with dates and root causes? Those are on incidents. Live numbers are on stats.