field notes KEEL

the last hop is the alert

An alerting system has one job and several dependencies: something evaluates a rule, something delivers the result, something transports it, something makes a noise. Every link can be empty while the previous link reports success, and nothing in the chain is watching the chain.

The delivery was rebuilt around a single hub on a machine deliberately chosen not to be one of the machines being monitored, with every producer posting straight to it, device delivery unchanged, and the public route to that hub closed behind credentials. The plan had also assumed the notification tool forwarded messages between its own nodes. Reading its source showed the forwarding was only a polling path for one mobile client, so the assumption was a fiction: there was no node-to-node delivery to configure, which is precisely why the hub had to be built.

The other failure was quieter. A recipient definition written in an older configuration shape was accepted without complaint and produced zero receivers. The rules loaded, the evaluation ran, the alerts fired, and nobody was told. Schema drift that fails silently is worse than a crash, because a crash is a signal.

The cost is that the alert path now has to be exercised as a path: an alert generated on purpose, followed through the hub, observed at the far end. What it proves is that monitoring which ends at a dashboard measures the estate, not the operator's ability to act, and that silence is the default state of every alerting system until the last hop has been tested.

drawn from: alert path commits

Every note carries a codename and one idea. What the systems implement is withheld by design — see the index for the rules.