field notes KEEL

the redundancy we decided not to buy

A proposal arrived to make the database highly available: two copies, automatic failover, the standard shape that appears in every design document. The review before building it asked a narrower question than the proposal did. What failure does this remove?

The answer was uncomfortable. The database dying on its own was already the cheap case — the scheduler restarts it in seconds, and the application reconnects. The failure that actually happens takes the whole machine with it, and redundancy inside one machine survives neither. The proposed work would have bought availability against the one failure already handled, and nothing against the one that mattered.

So the decision was to buy copies instead of replicas: backup generations held by different companies, in different buildings, plus the restore drill that proves they can be read. That is where the effort went, and the record states the reasoning, including that the cheaper answer was the better one.

The habit worth keeping is asking what a redundancy removes before paying for it. It is a harder conversation than approving a diagram, because it requires saying out loud which failure you are choosing to live with.

What this proves is that judgement about what not to build is as much a capability as the things that got built, and that a small estate can afford to be explicit about the failures it has accepted.

drawn from: high-availability review commits

Every note carries a codename and one idea. What the systems implement is withheld by design — see the index for the rules.