field notes KEEL

capacity was a layout problem wearing a disk problem's clothes

The root volume was filling, and the obvious reading was that the machine needed more disk. The correct reading was that the machine had been arranged badly. The root filesystem was copy-on-write with duplication enabled, which buys redundancy against a failing device on a volume that is a single device: no protection, half the capacity. The image store had been placed on that same root volume, so pulling and pruning both moved large amounts of data through the smallest filesystem on the machine. And snapshots were being retained on a limit nobody had chosen, generated faster than anyone pruned them.

The remediation was a relocation rather than a purchase: duplication off on root, image store off root, snapshot retention cut to a fraction of its previous limit, bulk data moved to a separate volume on a plain filesystem where the trade-offs are boring.

The result was roughly double the usable root capacity at a much lower fullness, from the same hardware. The tell that this was a policy problem and not an incident: the same pressure returned within two months.

What it proves is that capacity work is mostly deciding what lives where and how long it is permitted to stay. The tooling is the easy half. The habit that holds the line is a retention number somebody owns, because an unowned retention policy defaults to forever, and nobody notices until the disk is full.

drawn from: root volume and pool commits

Every note carries a codename and one idea. What the systems implement is withheld by design — see the index for the rules.