field notes LARDER

the backup is a hypothesis until a restore runs

A scheduled backup job ran green for as long as anyone cared to look. Green meant the snapshot was written and the tool's own integrity check passed. It did not mean the data could come back, because nobody had tried, and a snapshot that cannot be restored is only a claim about disk space.

The drill assembled a real artifact set — a compressed dump of one venue's database, the rendered deployment definitions, and a full repository bundle — wrote it through the backup path, then restored into a fresh empty location and compared. The comparison is the whole point: digests across every restored file matched their manifests, the restore-plan listing matched the source line for line, and the bundle both verified and cloned. The artifact set was small enough that the restore completed in about a second, which only matters because it was measured rather than assumed.

Six failure modes were written down before the drill counted as finished. The follow-on run against the off-site copy is recorded as blocked on credentials rather than marked done — which is the plainest illustration of why the drill exists: the write path had been proven, and the recovery path had not.

The posture is the unusual part. A restore drill produces no feature, no demonstration, and no visible reliability improvement, and it is the only work that converts a recovery plan into a known-good procedure.

What it proves: the team measures recovery instead of asserting it, and treats "we have backups" as an untested statement until a byte comparison says otherwise.

drawn from: commit

Every note carries a codename and one idea. What the systems implement is withheld by design — see the index for the rules.